Skip to content
Road to Intelligence

Concept · Chapter 7: Transformers

Feed-Forward Sublayer (MLP)

Must knowKnow well15 minDifficulty

The feed-forward sublayer is a small two-layer neural network applied to each token separately, transforming the information that attention has gathered.

The problem

Attention only mixes existing vectors by weighted averaging; on its own it can't compute new nonlinear features of a token.

The solution

After attention, pass each token's vector through an MLP: expand to a wider hidden layer, apply a nonlinearity, project back.

The consequence

Attention moves information between tokens; the MLP processes it within each token. The MLPs hold most of a Transformer's parameters and are thought to store much of its learned knowledge.

You should understand first

  1. Vectors
  2. Feed-Forward Sublayer (MLP)

What it computes

FFN(x)=W2 ϕ(W1x+b1)+b2\text{FFN}(x) = W_2\, \phi(W_1 x + b_1) + b_2

W1W_1 expands xx from dmodeld_{\text{model}} to a wider hidden size (4× in the original: 512 → 2048), ϕ\phi is a nonlinearity (ReLU originally; GELU in GPT-2; gated variants like SwiGLU in many recent LLMs), and W2W_2 projects back.

"Position-wise" means the same weights are applied to every token, but each token separately — no information passes between tokens here. That's attention's job.

Where the parameters are

Per block, attention has four d×dd \times d matrices (WQ,WK,WV,WOW_Q, W_K, W_V, W_O): 4d24d^2 parameters. The FFN has two d×4dd \times 4d matrices: 8d28d^2. So roughly two-thirds of a standard block's parameters sit in the feed-forward sublayer Established. Interpretability work suggests these MLPs act partly like key–value memories that store factual associations Interpretation — the 3Blue1Brown video below explores this idea.

What to remember

  • FFN(x) = W₂ · activation(W₁ x + b₁) + b₂, applied to each position independently.
  • Hidden width is typically 4 × d_model.
  • About two-thirds of a standard block's parameters are in the FFN.

Key papers

Essential

Attention Is All You Need

Ashish Vaswani, Noam Shazeer et al. · 2017 · NeurIPS 2017

Introduced the Transformer — the architecture behind BERT, GPT and nearly every modern large language model, and later adapted to vision, audio and more.

How to read it: Section 3 is the architecture — read it with Figure 1 open. Sections 3.2.1–3.2.2 contain the attention equation. You can skim the training details on a first pass.

~1 h 15 min readarXiv:1706.03762✓ verified 2026-09-26

Watch