The LLM StackFrom Silicon to Agents
⌕
An open, ground-up field guide

The LLM StackFrom Silicon to Agents

A ground-up field guide to building, training, aligning, serving, and reasoning with large language models.

14
Parts
146
Chapters
~3,773
Pages
1132k
Words

Start reading →Learning mapGlossaryTools & calculatorsInterview companionPrint / PDF edition

Front Matter
  1. Preface: How to Read This Book & The Map of the Stack
IPart I — Mathematical & Systems Foundations
  1. Linear Algebra for Deep Learning
  2. Probability, Statistics & Information Theory
  3. Calculus, Optimization & Convexity
  4. Numerical Computing, Floating Point & Precision
  5. Machine Learning Fundamentals
  6. Neural Networks From Scratch: MLPs & Backprop
  7. Automatic Differentiation & PyTorch Internals
  8. GPU Architecture & The Memory Hierarchy
  9. Parallel Computing & Collective Communication
  10. The Accelerator Landscape: TPUs, Trainium, AMD/ROCm & Gaudi
IIPart II — The Transformer Architecture
  1. Tokenization: BPE, WordPiece, Unigram & Byte-Level
  2. Embeddings & The Input Pipeline
  3. The Attention Mechanism From Scratch
  4. Multi-Head Attention, MQA, GQA & MLA
  5. Positional Encodings: Sinusoidal, Learned, RoPE & ALiBi
  6. The Transformer Block: Norms, Residuals, MLPs & Activations
  7. Building a GPT From Scratch (nanoGPT-style)
  8. Architecture Variants: Encoder-Decoder, Decoder-Only & Prefix-LM
  9. Mixture-of-Experts (MoE) Architectures
  10. Modern Architecture Improvements & Design Choices
  11. Beyond Attention: SSMs, Mamba, RWKV & Linear Attention
  12. Diffusion & Non-Autoregressive Language Models
IIIPart III — Pretraining at Scale
  1. Pretraining Data: Sources, Crawling & The Data Pipeline
  2. Data Cleaning, Deduplication & Quality Filtering
  3. The Pretraining Objective & Loss
  4. Scaling Laws: Kaplan, Chinchilla & Beyond
  5. Distributed Training I: Data Parallelism, DDP, ZeRO & FSDP
  6. Distributed Training II: Tensor, Pipeline, Sequence & Expert Parallelism
  7. Megatron-LM, DeepSpeed & Parallelism in Practice
  8. Mixed Precision, bf16 & FP8 Training
  9. Optimizers: SGD, Adam, Adafactor, Lion, Muon & Shampoo
  10. Learning Rate Schedules, Warmup, Batch Size & Hyperparameters
  11. Training Stability, Loss Spikes & Debugging Large Runs
  12. Checkpointing, Fault Tolerance & Long-Running Jobs
  13. Long-Context Pretraining & Context Extension
  14. Data Mixing, Domain Weighting & Curriculum
  15. Synthetic Data for Pre- and Post-Training
  16. Continual & Domain-Adaptive Pretraining
  17. Training an LLM From Scratch: The End-to-End Recipe
IVPart IV — Kernels, Efficiency & Quantization
  1. The Roofline Model & Performance Engineering
  2. FlashAttention I: IO-Awareness & The Online Softmax
  3. FlashAttention 2 & 3: Work Partitioning, Warp Specialization & FP8
  4. Writing GPU Kernels with Triton
  5. CUDA Programming Essentials for ML Engineers
  6. PagedAttention & KV-Cache Memory Management
  7. Quantization I: Post-Training Quantization (GPTQ, AWQ, SmoothQuant)
  8. Quantization II: INT4/INT8/FP8, GGUF, bitsandbytes & QAT
  9. Kernel Fusion, torch.compile, CUDA Graphs & Compilers
  10. Memory-Efficient Training: Checkpointing, Offloading & LoRA Math
VPart V — Post-Training & Alignment
  1. Supervised Fine-Tuning & Instruction Tuning
  2. Chat Templates, Data Formatting & Sequence Packing
  3. PEFT I: LoRA, QLoRA, DoRA & The Adapter Family
  4. PEFT II: Prompt/Prefix Tuning, IA3, Model Merging & Soups
  5. The RLHF Pipeline & Reward Modeling
  6. Policy Gradients & PPO for Language Models
  7. Direct Preference Optimization & Its Variants
  8. GRPO, RLOO & Critic-Free RL
  9. RL with Verifiable Rewards (RLVR) & The Reasoning Recipe
  10. Reasoning, Chain-of-Thought & Test-Time Compute
  11. Constitutional AI, RLAIF & Self-Improvement
  12. Distillation, Model Compression & Knowledge Transfer
  13. Reward Hacking, Over-Optimization & Alignment Failures
VIPart VI — RL Infrastructure (Deep Dive)
  1. The Anatomy of an RL-for-LLM System
  2. The Generation–Training Loop & Rollout Engines
  3. TRL: HuggingFace's RL Library
  4. veRL: HybridFlow & The Single-Controller Architecture
  5. OpenRLHF, NeMo-Aligner & Ray-Based Systems
  6. Prime-RL, Async RL & Decentralized Training
  7. Colocated vs Disaggregated RL & Weight Synchronization
  8. Reward Engineering, Verifiers & Sandboxes
  9. Advantage Estimation, KL Control & Stability Tricks
  10. Agentic & Multi-Turn RL
  11. Scaling RL: Throughput, Load Balancing & The Latest Tricks
  12. RL Data, Curriculum & Replay Management
VIIPart VII — Inference & Serving
  1. The Anatomy of LLM Inference: Prefill, Decode & The KV Cache
  2. Continuous Batching & Request Scheduling
  3. vLLM: Architecture, PagedAttention & Internals
  4. SGLang: RadixAttention & Structured Programs
  5. TensorRT-LLM, TGI & Other Serving Stacks
  6. Speculative Decoding: Draft Models, Medusa, EAGLE & Lookahead
  7. Prefix Caching & KV-Cache Reuse
  8. Disaggregated Prefill/Decode & Chunked Prefill
  9. Sampling Strategies & Decoding Algorithms
  10. Structured & Constrained Generation
  11. Multi-GPU & Multi-Node Inference
  12. Inference Economics: Latency, Throughput & Cost
  13. Serving Mixture-of-Experts: Expert Parallelism & All-to-All Inference
  14. Multi-Tenant LoRA & Adapter Serving at Scale
VIIIPart VIII — Agents & Harness Engineering
  1. Tool Use & Function Calling
  2. The Agentic Loop: ReAct, Plan-Execute & Reflection
  3. Harness Engineering: Building a Coding Agent
  4. Context Engineering & Management
  5. Memory Systems for Agents
  6. The Model Context Protocol (MCP)
  7. Multi-Agent Systems & Orchestration
  8. Agent Evaluation & Benchmarks
  9. Prompt Engineering as Engineering
IXPart IX — Retrieval & RAG
  1. Embeddings & Representation Learning
  2. Vector Databases & Approximate Nearest Neighbor Search
  3. Retrieval-Augmented Generation Architectures
  4. Chunking, Reranking & Hybrid Search
  5. Advanced RAG: GraphRAG, Agentic RAG & Long-Context vs RAG
  6. Multimodal & Visual-Document Retrieval: ColPali & Late Interaction
XPart X — Multimodal & Generative Frontiers
  1. Vision Transformers & Image Encoders
  2. Vision-Language Models
  3. Audio, Speech & Multimodal Fusion
  4. Diffusion Models & Generative Modeling (Breadth)
  5. Unified & Any-to-Any Models
XIPart XI — Evaluation
  1. The Evaluation Problem & Benchmark Landscape
  2. LLM-as-a-Judge & Automated Evaluation
  3. Building Eval Harnesses
  4. Reasoning, Coding & Agentic Evals
  5. Red-Teaming, Safety & Robustness Evaluation
  6. Statistical Rigor in Evaluation: Confidence Intervals & Significance
XIIPart XII — Production, Systems & MLOps
  1. Designing an LLM Serving System
  2. Observability, Logging & LLMOps
  3. Caching, Routing & Cost Control in Production
  4. Safety, Guardrails & Content Moderation
  5. Data Flywheels & Continuous Improvement
  6. Security: Prompt Injection, Jailbreaks & Defenses
  7. Online Evaluation: A/B Testing, Canaries & Guardrail Metrics
  8. Reliability Engineering for LLM Systems: SLOs & Incident Response
XIIIPart XIII — Interpretability, Safety & Governance
  1. Mechanistic Interpretability & Model Internals
  2. Knowledge Editing & Machine Unlearning
  3. Privacy, Memorization & Differential Privacy for LLMs
  4. Watermarking, Provenance & AI-Content Detection
  5. AI Safety: Scalable Oversight, Dangerous-Capability Evals & Frontier Safety
  6. AI Governance, Compliance & Regulation
XIVPart XIV — Capstone: Build a 100M LLM End-to-End
  1. The Capstone: Building Stack-100M, and the 2026 Small-Model Landscape
  2. Data: Sourcing, Filtering, Dedup, Tokenize & Pack ~20B Tokens
  3. A Byte-Level BPE Tokenizer From Scratch (and Why Vocab Size Is a Design Lever at 100M)
  4. The Stack-100M Architecture: SOTA Components, Cited and Assembled
  5. Mini Scaling Laws: Fit Your Own Law Before Spending the Budget
  6. Optimizer & Schedule: Muon + MuonClip and Warmup-Stable-Decay
  7. The Pretraining Run: A Complete Single-GPU Training Loop
  8. Mid-Training: Quality Annealing, Long-Context Extension & Capability Injection
  9. Post-Training: SFT, DPO, and Narrow RLVR (GRPO) That Works at 100M
  10. A Narrow Auto-Research Agent: ReAct, Tool-Use & Retrieval by Distillation
  11. Evaluation & Serving: Honest Benchmarks, int4 Quantization, and Running on a Laptop
  12. Retrospective: Cost Accounting, Reproducibility, and the Path to 1B
Appendix
  1. Glossary of Terms
  2. The Math Reference Sheet
  3. Key Papers: An Annotated Reading List
  4. Tooling & Environment Setup Cheatsheet
  5. From-Scratch Code Index

Cite this book

@book{kagitha_llm_stack_2026,
  title  = {The LLM Stack: From Silicon to Agents},
  author = {Kagitha, Prakash},
  year   = {2026},
  url    = {https://prakashkagitha.github.io/llm-stack-book/},
  note   = {Open web textbook, CC BY 4.0}
}