Skip to content
Road to Intelligence

Concept · Chapter 5: Vision, Speech & Reinforcement Learning

Audio and Spectrograms

12 minDifficulty

A spectrogram turns a waveform into a time × frequency picture by measuring the frequency content of short, overlapping windows of sound.

The problem

Raw audio is a very long one-dimensional list of pressure samples, and the patterns that matter (pitch, vowels, consonants) are about frequencies changing over time.

The solution

Slide a short window along the signal, take the Fourier transform of each window, and stack the magnitudes as columns.

The consequence

Speech becomes a 2-D input that CNNs and Transformers can process like an image or a sequence of frames.

You should understand first

  1. Vectors
  2. Audio and Spectrograms

From air pressure to numbers

A microphone measures air pressure many times per second. At 16 kHz, one second of speech is 16,000 numbers; a ten-second clip is 160,000. Plotted, it is the familiar wiggly waveform. Loud versus quiet is easy to see; which vowel was spoken is not.

Intuition: a piano roll for any sound

The Fourier transform answers "how much of each frequency is in this signal?" Applied to a whole clip it loses when things happened. So apply it to short slices instead: a window of about 25 ms, moved along in steps of about 10 ms. Each window yields one column of frequency strengths; side by side, the columns form a spectrogram: time left to right, pitch bottom to top, brightness for strength. A held note becomes a horizontal line, a rising whistle a diagonal, a click a vertical stripe, and a voiced vowel a stack of harmonics whose loud bands (formants) identify the vowel.

The trade-off you cannot escape

A window of N samples at sample rate fsf_s separates frequencies about fs/Nf_s/N Hz apart, but smears everything that happens within its N/fsN/f_s seconds. At 8 kHz, a 64-sample window lasts 8 ms and resolves 125 Hz; a 512-sample window lasts 64 ms and resolves about 16 Hz. Make one sharper and the other blurs: it is the same uncertainty trade-off that appears throughout signal processing.

Try it · toy model

Turn Sound into a Picture

Slice a sound into short windows, measure each window's frequencies, and discover why no window length is sharp in both time and frequency.

7 min

What speech models actually use

Two tweaks on the plain spectrogram: frequencies are grouped into mel bands, spaced roughly linearly at low frequencies and logarithmically at high ones (closer to how pitch is perceived), and magnitudes are log-compressed. Whisper, for example, resamples audio to 16 kHz and computes an 80-channel log mel spectrogram on 25 ms windows with a 10 ms stride . Some models skip the spectrogram and learn directly from raw samples with 1-D convolutions; WaveNet (2016) generated raw waveforms one sample at a time .

What to remember

  • Sample rate: 16,000 samples per second is common for speech; one second is 16,000 numbers.
  • Each spectrogram column is the frequency content of one short window (about 25 ms, stepped every 10 ms).
  • Short windows are sharp in time but blurry in frequency; long windows the reverse. No window is sharp in both.
  • Speech models usually use log mel spectrograms: frequency bands spaced like human pitch perception, on a log (decibel-like) scale.

Key papers

Optional

WaveNet: A Generative Model for Raw Audio

Aaron van den Oord, Sander Dieleman et al. · 2016

Generated audio one sample at a time with a stack of dilated causal convolutions, and produced much more natural-sounding synthetic speech.

~25 min readarXiv:1609.03499✓ verified 2026-09-26

Watch