Skip to content
Road to Intelligence

Concept · Chapter 4: Neural Networks

The Artificial Neuron

Must knowImplement20 minDifficulty

An artificial neuron computes a weighted sum of its inputs, adds a bias, and passes the result through a nonlinear activation function — a dot product plus a bend.

The problem

We need a simple, trainable building block that can be combined in large numbers to represent complicated functions.

The solution

Each unit computes z = w·x + b (how strongly the input matches the pattern stored in its weights) and outputs a = φ(z), where φ is a smooth nonlinearity so the unit can be trained by gradient descent.

The consequence

Millions to billions of these units, organized in layers, make up every neural network — and each one is just logistic regression with a different activation.

You should understand first

  1. Vectors
  2. Dot Product
  3. The Turing Test
  4. Symbolic AI
  5. Logic and Rules
  6. Expert Systems
  7. Knowledge Representation
  8. The Knowledge-Acquisition Bottleneck
  9. From Rules to Learning
  10. Supervised, Unsupervised and Self-Supervised Learning
  11. Features, Labels and Tasks
  12. Loss Functions
  13. Derivatives and Gradients
  14. Gradient Descent
  15. Linear Regression
  16. Probability and Distributions
  17. Entropy
  18. Softmax
  19. Cross-Entropy Loss
  20. Logistic Regression
  21. The Perceptron
  22. Activation Functions
  23. The Artificial Neuron

Tiny numeric example

  1. Inputs and weights

    x = [0.5, −1], w = [2, 1], b = 0.3.

  2. Weighted sum

    z = 2·0.5 + 1·(−1) + 0.3 = 0.3

  3. Activation

    ReLU: a = max(0, 0.3) = 0.3. Sigmoid: a = σ(0.3) ≈ 0.57.

The equation

a=ϕ(w⊤x+b)a=ϕ(Wx+b)    (a whole layer)a = \phi(\mathbf{w}^\top\mathbf{x} + b) \qquad\qquad \mathbf{a} = \phi(W\mathbf{x} + \mathbf{b}) \;\;\text{(a whole layer)}

Try it

Every circle in the lab's network diagram is one of these units. Click an edge to change a weight and watch the neuron's output — and the decision boundary — respond.

Try it · toy model

Neural Network Lab

A real two-layer network you can train, edit and dissect: watch activations flow forward, the decision boundary bend, and backpropagation send each example's error back to every weight.

Implement15 min

Why should I care?

As a researcher

Interpretability research asks what individual neurons (and directions made of many neurons) detect — the unit you'll analyse is this one.

As an engineer

Parameter counts, FLOPs and memory all follow from counting these weighted sums. A layer of n neurons over d inputs is a d×n matrix multiply.

Modern systems that depend on it

  • Every neural-network layer
  • Feed-forward sublayers in Transformers
  • Neuron-level interpretability

Historical context

Before

The McCulloch–Pitts neuron (1943) and Rosenblatt's perceptron (1958), with hard threshold outputs.

After

Smooth activations enabled gradient-based training; units are now organized into huge matrix operations rather than considered one by one.

Used today

Every layer of every neural network is a vector of these units computed at once as a matrix multiply plus an activation.

What to remember

  • z = w·x + b; a = φ(z).
  • Weights: what pattern the neuron responds to. Bias: how easily it fires.
  • The dot product measures how much the input matches the weights.
  • A neuron with sigmoid φ is exactly logistic regression.
  • A layer of neurons = one matrix multiplication + an activation.

Key papers

Watch