Skip to content
Road to Intelligence

Concept · Chapter 6: Language Before Transformers

Attention

Must knowKnow well25 minDifficulty

Attention lets a model build each output from a weighted mix of all the inputs, with the weights computed on the fly from how relevant each input is.

The problem

An encoder–decoder RNN squeezes a whole sentence into one fixed-size vector before translating it, so long sentences lose information.

The solution

Keep every input position's vector, and at each step score them all for relevance, turn the scores into weights with softmax, and take the weighted average.

The consequence

The model gets direct access to any position instead of relying on a memory that fades — the idea that the Transformer then made its core operation.

You should understand first

  1. Vectors
  2. Dot Product
  3. Embeddings
  4. Attention

The problem it solved

In 2014, neural translation used two recurrent networks: an encoder read the source sentence word by word and produced one vector; a decoder generated the translation from that vector alone. Everything the decoder knew about a 40-word sentence had to fit in one fixed-size vector. Quality dropped on long sentences.

The idea

Don't throw the encoder's per-word vectors away. When the decoder is about to produce the next word, let it look back at all of them and decide which matter right now. Translating "la maison bleue" → "the blue house", when producing "blue" it should focus on "bleue".

  1. Score

    Compare the decoder's current state with each input position to get a relevance score.

  2. Normalize

    Apply softmax so the scores become positive weights that sum to 1.

  3. Mix

    Take the weighted average of the input vectors. That average is the context for this step.

Why it matters

Attention improved translation of long sentences substantially over the fixed-vector encoder–decoder Established, and its learned weights often line up with sensible word alignments. Three years later, the Transformer asked: what if attention is the only mechanism — with every token attending to every other token of the same sequence? That is self-attention.

What to remember

  • Attention = score every position → softmax → weighted average.
  • Weights are computed from the data at run time, not fixed in advance.
  • Bahdanau et al. (2014) introduced this neural translation attention to address the fixed-vector bottleneck.

Key papers

Essential

Sequence to Sequence Learning with Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le · 2014 · NeurIPS 2014

Established the encoder–decoder pattern: read an input sequence into a vector, then generate an output sequence from it. Its central weakness motivated attention.

How to read it: Notice the trick of reversing the source sentence — a hint that long-range dependencies were the real problem.

~45 min readarXiv:1409.3215✓ verified 2026-09-26
Essential

Neural Machine Translation by Jointly Learning to Align and Translate

Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio · 2014 · ICLR 2015

Introduced attention in neural networks for language: instead of squeezing a sentence into one vector, the decoder looks back at every input word and decides which ones matter right now.

How to read it: Figure 3's alignment heat-maps are the best picture of 'attention' ever drawn — look at them first.

~1 h readarXiv:1409.0473✓ verified 2026-09-26
Essential

Attention Is All You Need

Ashish Vaswani, Noam Shazeer et al. · 2017 · NeurIPS 2017

Introduced the Transformer — the architecture behind BERT, GPT and nearly every modern large language model, and later adapted to vision, audio and more.

How to read it: Section 3 is the architecture — read it with Figure 1 open. Sections 3.2.1–3.2.2 contain the attention equation. You can skim the training details on a first pass.

~1 h 15 min readarXiv:1706.03762✓ verified 2026-09-26

Watch