Neural Networks & Backpropagation
Perceptrons, layers, how a network learns step by step
Prerequisites
- math-intuition
Your brain has about 86 billion neurons. Each one is pretty simple on its own — it receives signals from other neurons, and if the combined signal is strong enough, it fires and sends its own signal forward. That's it. But 86 billion of these simple things, wired together? That's enough to write poetry, recognize faces, and argue about pizza toppings.
Artificial neural networks borrow this idea — but dramatically simplified. We're not building a brain. We're building something much dumber, but still surprisingly powerful: layers of tiny math functions, each one taking numbers in and passing numbers out, connected together in ways that let the whole system learn patterns no single piece could ever find on its own.
In this lesson, you'll see how a single artificial neuron works, how you stack them into layers to build a network, and how the network actually learns — the famous process called backpropagation. By the end, you'll have a working intuition for the architecture behind everything from image recognition to ChatGPT.
What is a neural network?
Let's start small — with a single artificial neuron (sometimes called a perceptron). It does four things:
- Takes in one or more inputs (just numbers).
- Multiplies each input by a weight — a number that controls how much that input matters.
- Adds everything up, plus a bias (a constant offset).
- Passes the result through an activation function that decides whether and how strongly the neuron "fires."
That's genuinely it. Multiply, add, squish through a function. Here's the flow:
A single neuron
One neuron by itself isn't very useful. It can only learn simple, straight-line patterns. The magic happens when you stack neurons into layers. An input layer receives the raw data. One or more hidden layers transform it step by step. And an output layer produces the final answer.
A sample neural network architecture
Each circle above is a neuron. Each line is a weighted connection. Data flows left to right: inputs come in, get transformed layer by layer, and a prediction comes out the other side. More layers means the network can learn more complex patterns — that's the "deep" in "deep learning." A network with two hidden layers might learn to distinguish cats from dogs. One with dozens of layers can generate photorealistic images or hold a conversation.
How a network learns
Building a network is one thing. Teaching it to do something useful is another. The learning process is surprisingly mechanical — it's the same five-step loop, repeated thousands or millions of times. Click through to see how it works.
Step 1: Forward Pass
Data flows left to right through the network. Each neuron takes its inputs, multiplies by weights, adds the bias, and applies the activation function. The result passes to the next layer until you get a final output.
At first, with random weights, the output is meaningless. That's fine — the network hasn't learned anything yet.
That five-step loop — forward pass, loss, backpropagation, weight update, repeat — is the core of how virtually every neural network learns. The same basic process powers image classifiers that detect cancer, recommendation engines that pick your next binge watch, and the language model generating text right now when you talk to ChatGPT. The architectures get fancier, the datasets get bigger, the hardware gets faster — but the loop stays the same.
Now try it yourself
Here's a tiny neural network you can play with. Drag the input sliders and watch the values flow through the network in real time. Notice how changing one input affects everything downstream — each hidden neuron combines both inputs using different weights, and the final output depends on all of them together. The numbers inside each node show its current activation value. Connection colors show weight polarity (blue = positive, red = negative) and thickness shows weight magnitude.
Key Takeaways
- A neural network is layers of simple math functions (neurons) connected together. Each neuron multiplies inputs by weights, adds a bias, and applies an activation function.
- Networks learn through a loop: forward pass (make a prediction), calculate loss (how wrong?), backpropagation (trace the error backwards), update weights (nudge toward better), and repeat.
- Backpropagation is the key insight — it efficiently computes how much each weight contributed to the error, so you know exactly which weights to adjust and by how much.
- "Deep" means more layers. More layers let the network learn increasingly abstract features, which is why deep networks can tackle complex tasks like image generation and language understanding.
- The same training loop powers everything from a spam filter to GPT. The differences are in architecture, data, and scale — not the fundamental learning algorithm.
Common Misconceptions
- "Neural networks are trying to replicate the brain." — They're inspired by biological neurons, but the resemblance is superficial. Real neurons are vastly more complex. Artificial neural networks are better understood as function approximators — mathematical tools for finding patterns in data.
- "More layers always means a better model." — Deeper networks can learn more complex patterns, but they also need more data, more compute, and careful architecture design. Too deep without the right techniques (like residual connections) and the network can't train at all. Sometimes a shallow network is all you need.
- "Neural networks understand what they're doing." — They find statistical patterns. A network that classifies cat photos has no concept of what a cat is — it has learned pixel patterns that correlate with the label "cat." Powerful, but not understanding.