Skip to content
Road to Intelligence

Concept · Chapter 7: Transformers

The Transformer Block

Must knowImplement40 minDifficulty

A Transformer block is attention followed by a feed-forward network, each wrapped in normalization and a residual connection — and a Transformer is just many identical blocks stacked.

The problem

Attention alone mixes information but can't compute much, and deep stacks of layers are hard to train.

The solution

Pair attention (communication between tokens) with an MLP (computation within tokens), and wrap both in layer norm and residual connections so dozens of blocks can be stacked stably.

The consequence

A simple, uniform, highly parallel unit that scales: the same block design, with small modifications, runs inside BERT, GPT, T5, vision Transformers and today's LLMs.

You should understand first

  1. Vectors
  2. Dot Product
  3. Embeddings
  4. Attention
  5. Probability and Distributions
  6. Softmax
  7. Self-Attention
  8. Multi-Head Attention
  9. Causal Masking
  10. Positional Encoding
  11. Residual Connections
  12. Layer Normalization
  13. Feed-Forward Sublayer (MLP)
  14. The Transformer Block

The recipe

A (decoder, pre-LN) Transformer block is two lines:

x←x+MultiHeadAttention(LN(x))x←x+FFN(LN(x))\begin{aligned} x &\leftarrow x + \text{MultiHeadAttention}(\text{LN}(x)) \\ x &\leftarrow x + \text{FFN}(\text{LN}(x)) \end{aligned}

Input shape [n×dmodel][n \times d_{\text{model}}], output shape [n×dmodel][n \times d_{\text{model}}]. Because the shape is preserved, you can stack as many as you can afford.

The whole model around it

  1. Tokens → embeddings

    Look up a vector for each token ID; add (or later apply) positional information.

  2. N × block

    Each block lets tokens exchange information (attention) and then processes each token (FFN), refining the residual stream.

  3. Final LayerNorm → LM head

    Map each position's vector to one logit per vocabulary entry.

  4. Softmax

    Turn the last position's logits into a next-token distribution.

Try it

Try it · toy model

Transformer Explorer

Step through a real forward pass of one GPT-style Transformer block — embeddings, positions, multi-head attention, residuals, the MLP and the output softmax — with the tensor shape at every stage.

Know well15 min

Can you draw it?

The goal of this chapter is that you can sketch this on a whiteboard: a tall box repeated N times containing LN → attention → add, LN → MLP → add; embeddings and positions at the bottom; LN, linear and softmax at the top; arrows showing the residual path bypassing each sublayer. If you can, you understand the architecture of every GPT-style model.

Why should I care?

As a researcher

Most architecture papers are edits to this block — normalization, attention variant, activation, positions. You need the baseline to see what changed.

As an engineer

Parameter count, memory, FLOPs and latency all follow from the block's shapes: d_model, heads, FFN width and number of layers.

Modern systems that depend on it

  • GPT-style LLMs
  • BERT-style encoders
  • Vision Transformers
  • Speech models like Whisper

Historical context

Before

Recurrent networks (LSTMs, GRUs) with attention bolted on; convolutional sequence models.

After

Stacked into encoder-only, decoder-only and encoder–decoder models; refined with pre-LN, RMSNorm, SwiGLU, RoPE and efficient attention.

Used today

Every modern large language model is a stack of (lightly modified) Transformer blocks.

What to remember

  • Block (pre-LN): x = x + MHA(LN(x)); then x = x + FFN(LN(x)).
  • Attention = communication between tokens; FFN = computation within each token.
  • Input and output have the same shape [n × d_model], so blocks stack.
  • Around the stack: embeddings + positions below, final LN + LM head + softmax above.
  • Scale knobs: layers (N), width (d_model), heads (h), FFN width (usually 4·d_model).

Key papers

Essential

Attention Is All You Need

Ashish Vaswani, Noam Shazeer et al. · 2017 · NeurIPS 2017

Introduced the Transformer — the architecture behind BERT, GPT and nearly every modern large language model, and later adapted to vision, audio and more.

How to read it: Section 3 is the architecture — read it with Figure 1 open. Sections 3.2.1–3.2.2 contain the attention equation. You can skim the training details on a first pass.

~1 h 15 min readarXiv:1706.03762✓ verified 2026-09-26
Essential

Deep Residual Learning for Image Recognition

Kaiming He, Xiangyu Zhang et al. · 2015 · CVPR 2016

Residual (skip) connections made very deep networks trainable. Every Transformer block relies on the same trick.

~45 min readarXiv:1512.03385✓ verified 2026-09-26
Important

Layer Normalization

Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton · 2016

The normalization used inside Transformers; it keeps activations at a stable scale regardless of batch size.

~30 min readarXiv:1607.06450✓ verified 2026-09-26

Watch