Modern VLMs: Qwen, Gemma & Kimi Training in Auckland, New Zealand

This program introduces participants to Vision-Language Models (VLMs) using fully open-source, modern models like Qwen 3 VL, Gemma 3 Vision, and Kimi-VL.
Learners will understand how multimodal AI works, prepare simple image/video datasets, run open-source VLMs, perform fine-tuning using LoRA, and deploy multimodal applications.

The course focuses on intuitive explanations, hands-on guided exercises, and easy-to-use tools (Google Colab, Hugging Face Spaces, Gradio).

By the end, learners will be able to build and deploy their own simple VLM applications.

Introduction to VLMs & Core Foundations

Module 1: Understanding Vision-Language Models

• What are VLMs?
• How VLMs use images, text, video
• Real-world applications (captioning, Q&A, document understanding)

Module 2: Modern Open-Source VLMs

• Overview of Qwen 3 VL
• Overview of Gemma 3 Vision
• Overview of Kimi-VL
• Capabilities comparison

Module 3: Architecture

• Vision encoder (“eyes”)
• Language model (“brain”)
• Fusion mechanism (“how eyes talk to brain”)
• Why modern VLMs support video & long-context

Module 4: Data Fundamentals

• What is an image–text pair?
• What is a video–text pair?
• Basics of captions and annotations
• Introduction to publicly available datasets (COCO, WebVid, small open sets)

Module 5: Simple Dataset Preparation

• Preparing a small image dataset
• Writing simple captions
• Basic image preprocessing (resize, normalize)
• Basic video preprocessing (extract frames, understanding frame sampling)

Module 6: Running Your First VLM (Hands-on)

• Loading Qwen 3 VL / Gemma 3 Vision / Kimi-VL from Hugging Face
• Performing:
• image captioning
• image Q&A
• simple video understanding
• Understanding outputs

Fine-Tuning (Images + Simple Video)

Module 7: Introduction to Fine-Tuning

• What is fine-tuning?
• Why beginners use small datasets
• Full fine-tuning vs LoRA (simple explanation)
• Why LoRA is preferred for beginners

Module 8: Fine-Tuning on Images

• Creating a small training set
• Using PEFT/LoRA with open-source VLMs
• Training loop (simple walkthrough)
• Observing model improvements visually
• Avoiding overfitting on tiny datasets (basic tips)

Module 9: Evaluating Fine-Tuned Models

• Understanding good vs bad captions
• Simple evaluation:
• manual inspection
• intuition behind BLEU, CIDEr (no math)
• Saving and loading fine-tuned models

Module 10: Intro to Video Fine-Tuning

• Why video = “many images”
• Extracting short clips
• Simple frame sampling
• Preparing video–text pairs

Module 11: Fine-Tuning Kimi-VL or Qwen 3 VL on Small Video Tasks

• Using 3–10 video clips for demonstration
• Simple LoRA-based video fine-tuning
• Testing video Q&A or video captioning
• Discussing common mistakes

Evaluation, Deployment & Responsible AI

Module 12: Model Evaluation

• What is a benchmark?
• Why we evaluate models
• High-level introductions:
• SEED-Bench
• MMMU
• VideoMME
• Comparing outputs from Qwen, Gemma, Kimi

Module 13: Error Analysis & Debugging Basics

• Identifying hallucinations
• Identifying misinterpretations in images
• Identifying temporal errors in video
• Dataset quality issues

Module 14: Deployment Basics

• Running models locally (CPU/GPU)
• Running models in Google Colab
• Exporting models
• Creating simple demos with Hugging Face Spaces
• Gradio UI for:
• image captioning
• image Q&A
• short video Q&A

Module 15: Building a Simple End-to-End Demo

• Upload an image or short video
• Run inference using a VLM
• Display outputs in a web UI
• Optional: Add a text box for user questions

Module 16: Responsible AI Essentials

• Image bias & fairness issues
• Video privacy considerations
• Avoiding misleading outputs
• Best practices for safe usage of VLMs

Module 17: Wrap-Up & Learning Path