AI Engineering from Scratch Phase 17 Introduction
AI Engineering from Scratch
0% Completed

Phase 17: Infrastructure And Production

Phase 17: Infrastructure & Production

> Ship AI to the real world. Scale, monitor, optimize.

Start this phase on GitHub

Prerequisites: Phase 11 LLM Engineering and Phase 13 Tools and Protocols.

First lesson: Managed LLM Platforms

Run this command from the repository root:

python3 phases/17-infrastructure-and-production/01-managed-llm-platforms/code/main.py

Keep the command, exit code, cost and latency comparison, redundancy uplift,
and a short decision record naming the workload assumptions behind your choice.

Next action: Change one workload or SLA input, explain the new choice,
then continue to Inference Platform Economics.

Browse the full Phase 17 lesson list or the
cross-phase roadmap.

Lessons in this phase

  • Managed LLM Platforms — Bedrock, Vertex AI, Azure OpenAI
  • Inference Platform Economics — Fireworks, Together, Baseten, Modal, Replicate, Anyscale
  • GPU Autoscaling on Kubernetes — Karpenter, KAI Scheduler, Gang Scheduling
  • Serving Engine Internals — PagedAttention, Continuous Batching, Chunked Prefill
  • EAGLE-3 Speculative Decoding in Production
  • Prefix-Cache Serving — RadixAttention and KV Reuse
  • Hardware-Specialized Inference Compilation — FP8 and NVFP4 on Blackwell
  • Inference Metrics — TTFT, TPOT, ITL, Goodput, P99
  • Production Quantization — AWQ, GPTQ, GGUF K-quants, FP8, MXFP4/NVFP4
  • Cold Start Mitigation for Serverless LLMs
  • Multi-Region LLM Serving and KV Cache Locality
  • Edge Inference — Apple Neural Engine, Qualcomm Hexagon, WebGPU/WebLLM, Jetson
  • LLM Observability Stack Selection
  • Prompt Caching and Semantic Caching Economics
  • Batch APIs — the 50% Discount as Industry Standard
  • Model Routing as a Cost-Reduction Primitive
  • Disaggregated Prefill/Decode — NVIDIA Dynamo and llm-d
  • Production Serving Stack — KV Offloading and Cache-Aware Routing
  • AI Gateways — LiteLLM, Portkey, Kong AI Gateway, Bifrost
  • Shadow Traffic, Canary Rollout, and Progressive Deployment for LLMs
  • A/B Testing LLM Features — GrowthBook, Statsig, and the Vibes Problem
  • Load Testing LLM APIs — Why k6 and Locust Lie
  • SRE for AI — Multi-Agent Incident Response, Runbooks, Predictive Detection
  • Chaos Engineering for LLM Production
  • Security — Secrets, API Key Rotation, Audit Logs, Guardrails
  • Compliance — SOC 2, HIPAA, GDPR, PCI-DSS, EU AI Act, ISO 42001
  • FinOps for LLMs — Unit Economics and Multi-Tenant Attribution
  • Self-Hosted Serving Selection — Matching Engine to Hardware and Scale
  • Model Demo Interfaces — From a Function to a Shareable App
PreviousNext