Phase 12: Multimodal Ai

Phase 12: Multimodal AI

> Models that see, hear, read, and reason across modalities.

Start this phase on GitHub

Prerequisites: Phase 7 Transformers and Phase 4 Computer Vision.

First lesson: Vision Transformer Patch Tokens

Run this command from the repository root:

python3 phases/12-multimodal-ai/01-vision-transformer-patch-tokens/code/main.py

Keep the command, exit code, patch grid and sequence lengths, parameter counts,
and one sentence explaining why higher resolution creates more visual tokens.

Next action: Change one image or patch size, predict the sequence length,
then continue to CLIP and Contrastive Pretraining.

Browse the full Phase 12 lesson list or the
cross-phase roadmap.

Lessons in this phase

• Vision Transformers and the Patch-Token Primitive
• CLIP and Contrastive Vision-Language Pretraining
• From CLIP to BLIP-2 — Q-Former as Modality Bridge
• Flamingo and Gated Cross-Attention for Few-Shot VLMs
• LLaVA and Visual Instruction Tuning
• Any-Resolution Vision: Patch-n’-Pack and NaFlex
• Open-Weight VLM Recipes: What Actually Matters
• LLaVA-OneVision: Single-Image, Multi-Image, Video in One Model
• Qwen-VL Family and Dynamic-FPS Video
• InternVL3: Native Multimodal Pretraining
• Chameleon and Early-Fusion Token-Only Multimodal Models
• Emu3: Next-Token Prediction for Image and Video Generation
• Transfusion: Autoregressive Text + Diffusion Image in One Transformer
• Show-o and Discrete-Diffusion Unified Models
• Janus-Pro: Decoupled Encoders for Unified Multimodal Models
• MIO and Any-to-Any Streaming Multimodal Models
• Video-Language Models: Temporal Tokens and Grounding
• Long-Video Understanding at Million-Token Context
• Audio-Language Models: the Whisper to Audio Flamingo 3 Arc
• Omni Models: Qwen2.5-Omni and the Thinker-Talker Split
• Embodied VLAs: RT-2, OpenVLA, π0, GR00T
• Document and Diagram Understanding
• ColPali and Vision-Native Document RAG
• Multimodal RAG and Cross-Modal Retrieval
• Multimodal Agents and Computer-Use (Capstone)