[VLM101] Creating a Multimodal Chatbot with Fine-tuning (feat.MCP)

💡 Lectures required for these students

😤 "It's frustrating to only use API"

• If you have created a service using ChatGPT API, but are frustrated because it is expensive and has many restrictions,

• For those who are curious about the inside of an AI model like a black box and want to touch it directly

💸 "AI service operating costs are too expensive"

• Startup developers who are considering building their own models due to the cost of calling the OpenAI Vision API

• Anyone planning a service that requires large-scale image processing

🚀 "I want to become a multimodal AI expert"

• Anyone who wants to advance their career as an AI developer but has only taken a text-based LLM

• Job seekers who want to add a differentiated project to their portfolio

🤔 "I don't know exactly what VLM is"

• People who want to follow AI trends but don't exactly understand what multimodal is or what VLM is

• For those who are curious about the principles of AI that processes images and text simultaneously

💡 Specific changes you can achieve after taking the course

🎯 Immediately actionable practical skills

After completing the course, you will be able to work on the following hands-on projects on your own:

• My own VLM service : Image analysis chatbot specialized for specific domains (medical, education, shopping, etc.)

• Local AI Workflow : An automated system where multiple AI tools collaborate using MCP

• Cost-effective AI service : Service that reduces API dependency and operates with its own model

📈 Portfolio for career advancement

• GitHub repository : A complete repository containing the entire practice code and trained models.

• Technical Blog Material : Technical postings summarizing the VLM fine-tuning process and results can be written.

• Interview Experience : A Differentiated Interview Story with “Experience Fine-tuning VLM Directly”

🧠 Deep understanding and application

Beyond simple usage:

• Fully understand the internal workings of VLM, enabling rapid learning of new models

• Apply model optimization techniques such as Quantization and GGUF transformation to other projects

• Ability to design AI workflows using the MCP ecosystem

Practice environment

• The lecture will be based on MacOS. If you have a Windows machine and Docker is installed, you can mostly follow along.

• In this lecture, we will use cursor. I think you can follow the vscode version without any problems.

• Cloud environment

  • RunPod : GPU instance rental service, using H100 or A100

  • Estimated Cost : $10 for the entire practice

  • Pros : You can start practicing right away without any complicated environment settings.

  • ⚠ Note

    • You need to create a RunPod account and register a payment card.

Learning Materials

• Please check the attached PDF and source code

Player Knowledge and Notes

• LLM related knowledge (refer to previous LLM 101 lecture)

• Basic Python syntax (using classes, functions, modules)

• Deep learning/machine learning basic concepts (neural networks, training, inference, etc.)

• Experience training models in a GPU environment is preferred (but not required)

• Familiarity with terminal/command usage would be helpful