The atlas

The same map, as a list

Grouped by field. The dot on each concept shows your status; chapters marked “later” are not written yet.

Artificial Intelligence — Building systems that perceive, reason, learn and act.

• • Symbolic AIChapter 1

    Intelligence as explicit symbols, rules and search.

    • Symbolic AI
    • Logic & rules

      • Logic and Rules
    • Search & planning

      • Planning
      • Search
    • Knowledge representation

      • Knowledge Representation
    • Expert systems

      • Expert Systems
      • The Knowledge-Acquisition Bottleneck
• • Mathematical toolkitChapter 2

    The language every learning system is written in.

    • Linear algebra

      • Dot Product
      • Matrix Multiplication
      • Tensors and Shapes
      • Vectors
    • Probability

      • Conditional Probability and Bayes' Theorem
      • Probability of Sequences
      • Expected Value and Variance
      • Probability and Distributions
      • Softmax
    • Calculus

      • The Chain Rule
      • Derivatives and Gradients
    • Optimization

      • Gradient Descent
      • Loss Functions
      • Momentum and Adam
      • Stochastic Gradient Descent (SGD)
    • Statistics

      • Sampling and Uncertainty
    • Information theory

      • Cross-Entropy Loss
      • Entropy
      • KL Divergence
      • Perplexity
• • Machine LearningChapter 3

    Systems that improve at a task from data.

    • Supervised, Unsupervised and Self-Supervised Learning
    • From Rules to Learning
    • Classical MLChapter 3

      Regression, trees, SVMs, clustering — hand-crafted features.

      • Data Leakage
      • Decision Trees and Random Forests
      • Distribution Shift
      • Evaluation Metrics for Classifiers
      • Hand-Crafted Features vs Learned Features
      • Features, Labels and Tasks
      • Generalization, Overfitting and Underfitting
      • k-Means Clustering
      • Linear Regression
      • Logistic Regression
      • Naive Bayes
      • Principal Component Analysis (PCA)
      • Regularization
      • Support Vector Machines
    • Reinforcement learningChapter 5

      Learning from consequences and rewards.

      • Actor–Critic Methods
      • Deep RL, from Atari to AlphaGo to RLHF
      • Exploration vs Exploitation
      • MDPs, Policies and Value
      • Policy Gradients
      • Q-Learning
      • Reinforcement Learning
    • Deep LearningChapter 4

      Neural networks that learn their own features.

      • From MLPs to Transformers: The Architecture Story
      • Representation Learning
      • Neural nets & trainingChapter 4

        • Activation Functions
        • The Artificial Neuron
        • Backpropagation
        • Batch Normalization
        • Computational Graphs and Autodiff
        • Dropout
        • The Forward Pass
        • Weight Initialization
        • Multilayer Perceptron (MLP)
        • The Perceptron
        • Vanishing and Exploding Gradients
      • CNNsChapter 5

        • Convolutional Neural Networks
        • Convolution
        • ImageNet, AlexNet and ResNet
        • Images as Tensors
        • Pooling and Downsampling
      • RNNs & LSTMsChapter 6

        • LSTMs and GRUs
        • Recurrent Neural Networks
        • Sequence-to-Sequence Models
      • Language (NLP)Chapter 6

        • Language Modeling
        • N-Gram Models
        • One-Hot Encoding
        • Text as Data
        • Word embeddingsChapter 6

          • Embeddings
          • GloVe
          • Neural Language Model
          • Word2Vec
        • AttentionChapter 6

          • Attention
          • The Fixed-Vector Bottleneck
          • Neural Machine Translation
      • TransformersChapter 7
        • Self-attentionChapter 7

          • Causal Masking
          • Multi-Head Attention
          • Self-Attention
        • Transformer blockChapter 7

          • Feed-Forward Sublayer (MLP)
          • Layer Normalization
          • Positional Encoding
          • Residual Connections
          • The Transformer Block
        • Encoder / decoder variantsChapter 7

          • Encoder, Decoder & Encoder–Decoder
• • Foundation ModelsChapter 8

    Large models pretrained once and adapted to many tasks.

    • Language models (LLMs)Chapter 8
      • PretrainingChapter 8

        • Autoregressive Next-Token Prediction
        • Decoding: Greedy, Temperature, Top-k, Top-p
        • Emergent Abilities and the Debate
        • GPT-1 → GPT-2 → GPT-3
        • In-Context Learning
        • Parameters, Tokens and Context Windows
        • Why Next-Token Prediction Goes So Far
        • Pretrain, Then Fine-Tune
        • Pretraining at Scale
        • Tokenization
      • Data, scale & trainingChapter 9

        • Benchmark Contamination
        • The Compute Budget (C ≈ 6ND)
        • Compute-Optimal Training (Kaplan vs Chinchilla)
        • Data Mixtures, Repetition and Synthetic Data
        • Data Parallelism and All-Reduce
        • Deduplication and MinHash
        • Learning-Rate Schedules, Warmup and Clipping
        • Mixed-Precision Training (FP16 and BF16)
        • Tensor and Pipeline Parallelism
        • Building a Pretraining Dataset
        • Scaling Laws (Preview)
        • Sharded Training (ZeRO and FSDP)
        • Designing the Tokenizer
        • The Pretraining Loop
        • Where Training Memory Goes
        • Loss Spikes, Failures and Checkpoints
      • Post-trainingChapter 10

        • From Base Model to ChatGPT
        • Chat Templates
        • Constitutional AI and AI Feedback
        • Direct Preference Optimization
        • Instruction and Demonstration Data
        • Evaluating Post-Training
        • PPO for Language Models
        • Learning from Comparisons
        • Generate, Rank, Then Train
        • Reward Models
        • Reinforcement Learning from Human Feedback
        • Safety Tuning and Refusal
        • Supervised Fine-Tuning
      • Efficient architectures & inferenceChapter 11

        • The Quadratic Cost of Attention
        • FlashAttention
        • Multi-Query and Grouped-Query Attention
        • Knowledge Distillation
        • The KV Cache
        • LoRA, QLoRA and Parameter-Efficient Fine-Tuning
        • Mixture of Experts (MoE)
        • The Modern Decoder Block: Pre-Norm, RMSNorm, SwiGLU
        • PagedAttention and Continuous Batching
        • Prefill, Decode and the Memory Wall
        • Pruning and Sparsity
        • Quantization
        • Rotary Position Embedding (RoPE)
        • Speculative Decoding
    • VisionChapter 5

      • Detection and Segmentation
      • Vision Transformer (ViT)
    • Audio & speechChapter 5

      • Audio and Spectrograms
      • Speech Recognition and Synthesis
    • Diffusion & generative mediaChapter 15

      • Classifier-Free Guidance
      • Diffusion Models
      • Generative Models: GANs, VAEs and Image Tokens
      • Latent Diffusion and Text-to-Image
      • Video Generation
    • MultimodalChapter 15

      • Audio Language Models and Voice
      • CLIP and Image–Text Embeddings
      • Reading Documents, Charts and Screens
      • Turning Signals into Tokens
      • Vision-Language-Action Models
      • Vision-Language Models
      • World Models
• • AI SystemsChapter 12

    Models embedded in larger loops with data and tools.

    • Embeddings & retrievalChapter 12

      • Approximate Nearest-Neighbour Search
      • Contrastive Learning
      • Keyword Search and BM25
      • Semantic and Hybrid Search
      • Text Embeddings
    • RAGChapter 12

      • Chunking
      • Context Engineering
      • Parametric vs Retrieved Knowledge
      • Evaluating Retrieval and RAG
      • Reranking
      • Retrieval-Augmented Generation (RAG)
    • Tools & structured outputChapter 12

      • Guardrails
      • Structured Outputs and Constrained Decoding
      • System Prompts and Instructions
      • Tool Calling
    • AgentsChapter 13

      • Agent Evaluation
      • Agent Memory
      • Agent Planning and ReAct
      • Agent Reflection
      • Agent Reliability and Budgets
      • Agent Runtime
      • Agent Trust Boundaries
      • Tool Execution in Agents
      • Coding Agents
      • Computer-Use Agents
      • Human-in-the-Loop Agents
      • LLM Agents
      • Model Context Protocol
      • Multi-Agent Systems
      • Workflows vs Agents
    • Reasoning & test-time computeChapter 14

      • Chain of Thought
      • Group Relative Policy Optimization (GRPO)
      • Inference-Time Scaling
      • Process vs Outcome Supervision
      • Synthetic Reasoning Data and Distillation
      • Faithfulness of Reasoning Traces
      • Search over Reasoning Steps
      • Verifiers and Best-of-N
      • Self-Consistency
      • Training vs Inference Compute
      • RL with Verifiable Rewards
• • Science of AIChapter 16

    Measuring, trusting, securing and understanding models.

    • EvaluationChapter 16

      • Error Bars on Evals
      • LLM-as-a-Judge
      • Benchmarks and What They Measure
      • Multimodal Hallucination and Evaluation
      • Human Preference Evaluation and Arenas
    • ReliabilityChapter 16

      • Abstention and Selective Prediction
      • Calibration
      • Hallucination
      • Robustness and Prompt Sensitivity
    • Safety & securityChapter 16

      • Jailbreaks and Adversarial Prompts
      • Memorization and Training-Data Extraction
      • Red Teaming
    • InterpretabilityChapter 16

      • Circuits and Mechanistic Interpretability
      • Probing Representations
      • Sparse Autoencoders and Dictionary Learning
      • Superposition and Polysemantic Neurons