Reference

Glossary

Short definitions — two sentences at most — each linking to the full explanation.

93 of 93 terms

Activation functionReLU, sigmoid, tanh, GELU, nonlinearity: The nonlinearity applied after each neuron's weighted sum; without it, stacked layers collapse into one linear map. Activation Functions →
AdamAdamW, momentum, optimizer: An optimizer combining momentum with per-parameter step sizes. AdamW, its weight-decay variant, is the default for Transformers. Momentum and Adam →
AI winter: A period of collapsed funding and interest after AI's promises outran its results — roughly 1974–1980 and from 1987. AI Winters →
Attention: A mechanism that lets a model compute a weighted average of other positions' information, with weights based on relevance. Attention →
Automatic differentiationautodiff, autograd, computational graph: Software that records a computation's operations and applies the chain rule to compute exact gradients automatically. Computational Graphs and Autodiff →

Backpropagationbackprop, backward pass: Computing the gradient of the loss for every weight by passing the error backward through the network with the chain rule. Backpropagation →
Batch normalizationbatch norm: Normalizing each layer's activations with mini-batch statistics to speed up and stabilize training. Batch Normalization →
Bayes' theoremprior, posterior, likelihood, base rate: P(H | E) = P(E | H)·P(H) / P(E). Turns 'how likely is the evidence given the cause' into 'how likely is the cause given the evidence'. Conditional Probability and Bayes' Theorem →

Causal maskcausal masking, look-ahead mask: Blocks each position from attending to later positions (their scores are set to −∞ before softmax), so next-token prediction can't cheat. Causal Masking →
Chain rule: The derivative of a composition is the product of the derivatives along the chain. Backpropagation applies it layer by layer. The Chain Rule →
Classification: Predicting a category (spam / not spam, digit, next token). Features, Labels and Tasks →
CNNconvolutional neural network, RNN, LSTM, inductive bias: Architectures that build in structure: CNNs share local filters across images; RNNs/LSTMs process sequences step by step. From MLPs to Transformers: The Architecture Story →
Conditional probabilityP(A | B): The probability of A given that B happened: restrict attention to the cases where B is true. Conditional Probability and Bayes' Theorem →
Confidence intervalstandard error, error bars: A range expressing the uncertainty of a measured quantity. For an accuracy p on n examples, roughly p ± 2·√(p(1−p)/n). Sampling and Uncertainty →
Cosine similarity: The dot product of two vectors divided by the product of their lengths: the cosine of the angle between them, from −1 to 1. Dot Product →
Cross-entropylog loss, negative log-likelihood, NLL: The loss −log p(correct answer). It is the training objective of classifiers and language models. Cross-Entropy Loss →

Data leakagetarget leakage, contamination: Information in training or evaluation data that won't be available at prediction time, inflating results. Data Leakage →
Derivativeslope: How fast a function's output changes as its input is nudged — the local slope. Derivatives and Gradients →
Distribution shiftcovariate shift, concept drift: A mismatch between the data a model was trained on and the data it meets in use. Distribution Shift →
Dot productinner product: Multiply two vectors element by element and sum the results. Large when the vectors point the same way; the basic similarity score in ML. Dot Product →
Dropout: Randomly switching off units during training so the network can't rely on any single one; a regularizer. Dropout →

Embeddingembedding vector, dense representation: A learned vector representing an item (token, word, document, image) so that similar items have nearby vectors. Embeddings →
Encoder-only / decoder-onlyBERT, GPT, encoder–decoder, T5: The three ways to wire Transformer blocks: bidirectional encoders (BERT), causal decoders (GPT) or both with cross-attention (T5). Encoder, Decoder & Encoder–Decoder →
Entropyinformation, surprise, bits, nats: The average surprise (−log p) of a distribution's outcomes — how uncertain it is. Entropy →
Expected valuemean, expectation: The probability-weighted average of a random quantity. Training minimizes the expected loss over the data. Expected Value and Variance →
Expert systemMYCIN, XCON: A program encoding a specialist's knowledge as if–then rules in a narrow domain; AI's first commercial success (1970s–80s). Expert Systems →

Featurefeature vector, input variable: A measurable input property of an example; the model's inputs. Features, Labels and Tasks →
Feature engineeringhand-crafted features: Designing input features by hand — the bottleneck that deep learning's learned features removed. Hand-Crafted Features vs Learned Features →
Feed-forward networkFFN, MLP: In a Transformer block, a two-layer network applied to each token separately after attention; holds most of the parameters. Feed-Forward Sublayer (MLP) →
Fixed-vector bottleneck: The constraint in early encoder-decoders where the whole source must be compressed into one fixed-size vector for the decoder. The Fixed-Vector Bottleneck →
Forward passinference: Computing a network's output from its input, layer by layer. The Forward Pass →

Generalization: How well a model performs on data it was not trained on — the actual goal of learning. Generalization, Overfitting and Underfitting →
Gradientpartial derivative, ∇: The vector of partial derivatives of a function with respect to all its inputs. It points in the direction of steepest increase. Derivatives and Gradients →
Gradient descentlearning rate, step size: Minimize a loss by repeatedly moving parameters a small step against the gradient: w ← w − η·∇L(w). Gradient Descent →

HeuristicA*, heuristic search: A rule of thumb that estimates how promising an option is, used to guide search toward a goal with less exploring. Search →

Inference engineforward chaining, backward chaining: The part of a rule-based system that applies rules to facts to derive conclusions. Logic and Rules →

k-meansclustering: Clustering by alternately assigning points to the nearest centre and moving centres to their points' mean. k-Means Clustering →
KL divergenceKullback–Leibler divergence, relative entropy: A non-negative, asymmetric measure of how different one probability distribution is from another. KL Divergence →
Knowledge graphontology, semantic network, frames: A structured representation of entities and the relations between them — a descendant of symbolic knowledge representation. Knowledge Representation →
Knowledge-acquisition bottleneckbrittleness, tacit knowledge: The difficulty of getting knowledge into rule-based systems: expertise is tacit, full of exceptions, and vast. The Knowledge-Acquisition Bottleneck →

Labeltarget, ground truth: The correct output for a training example — what a supervised model learns to predict. Features, Labels and Tasks →
Language modelnext-word prediction: A model that assigns a probability distribution to the next text unit given the preceding units. Language Modeling →
Layer normalizationLayerNorm, RMSNorm: Rescales each token's vector to zero mean and unit variance, with a learned scale and shift, keeping activations stable. Layer Normalization →
Linear regressionleast squares: Predicting a number as a weighted sum of features, fitted by minimizing squared error. Linear Regression →
Logistic regressionsigmoid, logit: A linear classifier: a weighted sum passed through a sigmoid to give a probability, trained with cross-entropy. Logistic Regression →
Logits: The raw, unnormalized scores a model outputs — one per class or vocabulary token — before softmax turns them into probabilities. Softmax →
Loss functionobjective, cost function, MSE: A single number measuring how wrong a model's predictions are. Training minimizes its average over the data. Loss Functions →
LSTMGRU, gated recurrent unit, long short-term memory: A gated recurrent architecture that learns when to retain and update information across sequence steps. GRUs are a more compact related design. LSTMs and GRUs →

Machine learninglearning from data: Building systems that learn their behaviour from examples rather than following hand-written rules. From Rules to Learning →
Matrix multiplicationmatmul, linear layer: Each output entry is the dot product of a row of one matrix with a column of the other. A neural-network layer is a matrix multiplication plus a nonlinearity. Matrix Multiplication →
Multi-head attentionMHA, attention head: Several attention operations run in parallel on smaller slices of the vectors, each with its own queries, keys and values; outputs are concatenated. Multi-Head Attention →
Multilayer perceptronMLP, hidden layer, fully connected: A network of layers of neurons — input, hidden layers, output — each building on the previous layer's outputs. Multilayer Perceptron (MLP) →

N-grambigram, trigram: A short sequence of n words. An n-gram language model predicts from the previous n−1 words using counts. N-Gram Models →
Naive Bayes: A classifier applying Bayes' theorem with the assumption that features are independent given the class. Naive Bayes →
Neuronunit, node, weights, bias: The basic unit of a neural network: a weighted sum of inputs plus a bias, passed through an activation function. The Artificial Neuron →

One-hot encoding: One coordinate per vocabulary item, with exactly one 1 and all other entries 0. It represents identity but gives no similarity between different words. One-Hot Encoding →
Overfittingunderfitting, bias–variance: Fitting the training data (including its noise) so closely that performance on new data suffers. Generalization, Overfitting and Underfitting →

PCAprincipal component analysis: Finding the directions of greatest variance in data to reduce its dimensions. Principal Component Analysis (PCA) →
Perceptron: The first learning neuron (1958): a thresholded weighted sum with a mistake-driven update rule; limited to linear boundaries. The Perceptron →
PerplexityPPL: exp(average cross-entropy per token): roughly how many tokens a language model is effectively choosing between. Perplexity →
PlanningSTRIPS: Finding a sequence of actions, each with preconditions and effects, that achieves a goal. Planning →
Positional encodingposition embedding, RoPE: Information about token position added to (or built into) attention, since attention alone ignores order. Positional Encoding →
Precision and recallF1, confusion matrix: Precision: share of flagged items that were correct. Recall: share of true items that were caught. Evaluation Metrics for Classifiers →
Probability distributionrandom variable, categorical distribution, Gaussian: An assignment of probabilities to every possible outcome, summing to 1. Classifiers and language models output one. Probability and Distributions →

Query, key, valueQ/K/V, QKV: Three learned projections of each token in attention: the query asks 'what am I looking for?', keys advertise 'what I contain', values carry the information passed on. Self-Attention →

Random forestdecision tree, gradient boosting: An ensemble of decision trees trained on random subsets of data and features, with predictions averaged. Decision Trees and Random Forests →
Regression: Predicting a number (price, demand, temperature). Features, Labels and Tasks →
Regularizationweight decay, L1, L2, ridge, lasso: Techniques that discourage overly complex models, such as penalizing large weights, to improve generalization. Regularization →
Representation learninglearned features, deep learning: Networks learning their own intermediate features from raw data instead of relying on hand-crafted ones. Representation Learning →
Residual connectionskip connection, residual stream: Adding a layer's input to its output (x + f(x)) so the layer learns a correction; makes deep networks trainable. Residual Connections →
RNNrecurrent neural network, hidden state, backpropagation through time: A network that reads a sequence step by step, repeatedly updating a hidden state that summarizes earlier inputs. Recurrent Neural Networks →
ROC / AUCROC curve: The trade-off between true- and false-positive rates across all thresholds; AUC summarizes it (0.5 = chance, 1 = perfect). Evaluation Metrics for Classifiers →

SearchBFS, DFS, state space: Solving a problem by exploring sequences of actions from a start state until one reaches the goal. Search →
Self-attention: Attention in which every token in a sequence attends to the tokens of the same sequence, producing context-aware representations. Self-Attention →
Self-supervised learningpretraining objective: Learning from unlabelled data by predicting a hidden or next part of it — e.g. the next word. How LLMs are pretrained. Supervised, Unsupervised and Self-Supervised Learning →
Sequence-to-sequenceseq2seq, encoder-decoder: An encoder reads a source sequence and a decoder produces a target sequence, often with a different length or order. Sequence-to-Sequence Models →
Smoothingadd-one smoothing, Laplace smoothing: Adjusting count-based probabilities so unseen continuations are not assigned exact zero probability. N-Gram Models →
Softmax: A function that turns a list of arbitrary scores into positive numbers that sum to 1 — a probability distribution. Larger scores get exponentially more weight. Softmax →
Stochastic gradient descentSGD, mini-batch, batch size, epoch: Gradient descent using the gradient of a small random batch of examples as a cheap, noisy estimate of the full gradient. Stochastic Gradient Descent (SGD) →
Supervised learninglabels: Learning a mapping from inputs to known correct outputs (labels). Supervised, Unsupervised and Self-Supervised Learning →
Support vector machineSVM, kernel trick: A classifier that picks the boundary with the widest margin; kernels let it draw non-linear boundaries. Support Vector Machines →
Symbolic AIGOFAI, classical AI: AI built from explicit symbols and hand-written rules, manipulated by logic and search. Dominant from the 1950s to the 1980s. Symbolic AI →

Tensorshape, broadcasting: An n-dimensional array of numbers, e.g. [batch, tokens, d_model]. Tracking tensor shapes is most of the bookkeeping in deep learning. Tensors and Shapes →
Token: The unit of text a language model reads and writes — often a word piece like 'trans' + 'formers'. Models see token IDs, not characters. explained in Chapter 8
Transformer: A neural-network architecture built from stacked self-attention and feed-forward layers, introduced in 2017; the basis of modern LLMs. The Transformer Block →
Turing testimitation game: Turing's 1950 proposal: if an interrogator can't tell a machine's typed conversation from a human's, treat the machine as intelligent. The Turing Test →

Unsupervised learningclustering, dimensionality reduction: Finding structure in data without labels, such as clusters or low-dimensional directions. Supervised, Unsupervised and Self-Supervised Learning →

Validation settest set, train/validation/test split, cross-validation: Held-out data used to choose models and hyperparameters; the test set is kept for one final evaluation. Generalization, Overfitting and Underfitting →
Vanishing gradientsexploding gradients: Gradients shrinking toward zero (or blowing up) as they pass back through many layers, stalling or destabilizing training. Vanishing and Exploding Gradients →
Variancestandard deviation: The average squared distance from the mean — how spread out a quantity is. Its square root is the standard deviation. Expected Value and Variance →
Vector: An ordered list of numbers, e.g. [0.2, −1.3, 4.0]. In ML, almost everything — a word, an image, a user — is represented as a vector. Vectors →

Weight initializationXavier, He initialization: Choosing the scale of random starting weights so signals and gradients stay stable across layers. Weight Initialization →
Word2vecskip-gram, CBOW, negative sampling: Efficient methods for learning word vectors from local context prediction tasks; negative sampling speeds up skip-gram training. Word2Vec →