Phase 10: Llms From Scratch

Phase 10: LLMs from Scratch

> Build, train, and understand large language models.

Start this phase on GitHub

Prerequisites: Phase 5 NLP Foundations. Phase 7 Transformers is strongly
recommended before the model-building lessons.

First lesson: Tokenizers

Run this command from the repository root:

python3 phases/10-llms-from-scratch/01-tokenizers/code/main.py

Keep the command, exit code, encode/decode round-trip results, learned merge
count, and compression ratios. tiktoken is an optional comparison.

Next action: Add a word the tiny corpus has not seen, inspect its pieces,
then continue to Building a Tokenizer from Scratch.

Browse the full Phase 10 lesson list or the
cross-phase roadmap.

Lessons in this phase

• Tokenizers: BPE, WordPiece, SentencePiece
• Building a Tokenizer from Scratch
• Data Pipelines for Pre-Training
• Pre-Training a Mini GPT (124M Parameters)
• Scaling: Distributed Training, FSDP, DeepSpeed
• Instruction Tuning (SFT)
• RLHF: Reward Model + PPO
• DPO: Direct Preference Optimization
• Constitutional AI and Self-Improvement
• Evaluation: Benchmarks, Evals, LM Harness
• Quantization: Making Models Fit
• Inference Optimization
• Building a Complete LLM Pipeline
• Open Models: Architecture Walkthroughs
• Speculative Decoding and EAGLE-3
• Differential Attention (V2)
• Native Sparse Attention (DeepSeek NSA)
• Multi-Token Prediction (MTP)
• DualPipe Parallelism
• DeepSeek-V3 Architecture Walkthrough
• Jamba — Hybrid SSM-Transformer
• Async and Hogwild! Inference
• Speculative Decoding and EAGLE
• Gradient Checkpointing and Activation Recomputation