Concept · Chapter 6: Language Before Transformers
LSTMs and GRUs
LSTMs and GRUs add learned gates to a recurrent network so it can keep, discard and update information more deliberately.
The problem
A plain RNN repeatedly transforms one hidden state, making old information and its training signal hard to preserve.
The solution
Use gates to control how much previous memory survives and how much new information is written at each step.
The consequence
Longer dependencies became more learnable, but the sequence still has to be processed step by step.
You should understand first
- Text as Data
- Probability and Distributions
- Conditional Probability and Bayes' Theorem
- Probability of Sequences
- Language Modeling
- Vectors
- Dot Product
- The Turing Test
- Symbolic AI
- Logic and Rules
- Expert Systems
- Knowledge Representation
- The Knowledge-Acquisition Bottleneck
- From Rules to Learning
- Supervised, Unsupervised and Self-Supervised Learning
- Features, Labels and Tasks
- Loss Functions
- Derivatives and Gradients
- Gradient Descent
- Linear Regression
- Entropy
- Softmax
- Cross-Entropy Loss
- Logistic Regression
- The Perceptron
- Activation Functions
- The Artificial Neuron
- Embeddings
- Recurrent Neural Networks
- The Chain Rule
- Matrix Multiplication
- Multilayer Perceptron (MLP)
- The Forward Pass
- Computational Graphs and Autodiff
- Backpropagation
- Vanishing and Exploding Gradients
- LSTMs and GRUs
A memory with controls
Imagine reading "The keys to the old cabinet are missing." To predict the verb, the model needs to carry the plural subject across several intervening words. A plain RNN repeatedly overwrites one state. An LSTM gives memory a more direct route through time and learns gates that decide what to retain, write and reveal. A GRU uses a smaller gating arrangement with a similar aim.
Each gate is computed from the current input and previous state, passed through a sigmoid, then multiplied by an information path. A gate near 0 blocks most of that path; one near 1 lets it through. The values are learned during training, not hand-coded rules such as "remember nouns".
LSTMs made long-range sequence tasks practical and powered important translation systems. They did not remove the dependency between step and step : a recurrent model still has to read in order.
What to remember
- A gate is a learned number between 0 and 1 that scales an information path.
- An LSTM keeps a separate cell state with gates for writing and reading; a GRU combines memory and output more compactly.
- Gating helps long-range learning but does not remove sequential computation.
Key papers
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Kyunghyun Cho, Bart van Merrienboer et al. · 2014 · EMNLP 2014
Proposed a jointly trained RNN encoder and decoder for mapping one sequence to another, with the gated recurrent unit in the architecture.
Long Short-Term Memory
Sepp Hochreiter, Jürgen Schmidhuber · 1997 · Neural Computation
LSTM added gated memory cells so recurrent networks could keep information over long sequences; it dominated sequence modelling until Transformers.