[VLM101] Building a Multimodal Chatbot with Fine-tuning (feat.MCP / RunPod)

💡 A course for students who need this

😤 "I'm frustrated with just using APIs"

• Those who built a service with ChatGPT API but feel frustrated due to cost burden and many limitations

• Those curious about the inner workings of black-box AI models and want to get hands-on experience

💸 "AI service operating costs are too expensive"

• Startup developers considering building their own model due to the burden of OpenAI Vision API costs

• Those planning a service that requires processing large volumes of images

🚀 "I want to become a multimodal AI expert"

• Those who want to advance their career as an AI developer but have only worked with text-based LLMs

• Job seekers who want to add differentiated projects to their portfolio

🤔 "I'm not sure exactly what VLM is"

• Those who want to keep up with AI trends but don't fully understand what multimodal or VLM is

• Those curious about the principles of AI that processes images and text simultaneously

💡 Concrete Changes You'll Gain After Taking the Course

🎯 Immediately Applicable Practical Skills

After completing the course, you'll be able to independently work on the following real-world projects:

• Your Own VLM Service: Image analysis chatbot specialized for specific domains (medical, education, shopping, etc.)

• Local AI Workflow: An automated system where multiple AI tools collaborate using MCP

• Cost-Effective AI Services: Services that reduce API dependency and operate with proprietary models

📈 Portfolio for Career Development

• GitHub Repository: A well-organized repository with complete practice code and trained models

• Technical blog material: Can write technical posts documenting the VLM fine-tuning process and results

• Interview Experience: A differentiated interview story with "hands-on experience fine-tuning VLM"

🧠 Deep Understanding and Application Skills

Beyond simple usage:

• Fully understand the internal workings of VLM to quickly learn new models as well

• Apply model optimization techniques such as Quantization, GGUF conversion to other projects as well

• The ability to design AI workflows utilizing the MCP ecosystem

Practice Environment

• The course is explained based on MacOS. If you're using a Windows machine, you should be able to follow along as long as Docker is installed.

• The course uses Cursor. I believe you can follow along without any issues using the VSCode version as well.

• Cloud Environment

  • RunPod: GPU instance rental service, using H100 or A100

  • Estimated Cost: $10 for the entire hands-on practice

  • Advantages: Can practice immediately without complex environment setup

  • ⚠ Important Notice

    • RunPod account creation and payment card registration required

Learning Materials

• You can check the attached PDF and source code

Prerequisites and Important Notes

• LLM-related knowledge (refer to previous LLM 101 lecture)

• Basic Python syntax (classes, functions, module usage)

• Basic concepts of deep learning/machine learning (neural networks, training, inference, etc.)

• Experience with model training in GPU environments is helpful (but not required)

• Familiarity with using terminal/command line will be helpful