Activation Functions, Explained
After reading this you will be able to write down sigmoid, tanh, ReLU and their smooth cousins with their derivatives, predict which ones let gradients survive many layers, and recognize a dying ReLU from its firing histogram.
What an activation function does
Stack two linear layers and you get one linear layer. Multiply matrices W_2 (W_1 x) and the result is just (W_2 W_1) x, still a straight line. A network built only from matrix multiplications can never bend a decision boundary. The activation function is the nonlinear step applied element by element between layers, and it is the reason a network can fit curves at all.
Here is the hook. Take a plain sigmoid, whose derivative peaks at 0.25. Backpropagation multiplies one derivative per layer as the gradient travels backward. Ten sigmoid layers scale the gradient by at most 0.25^{10} \approx 9.5 \times 10^{-7}. The first layer sees roughly a millionth of the gradient the last layer sees, so it barely moves during training. Swap in ReLU, whose derivative is exactly 1 for active units, and that same product stays at 1. Same architecture, same data, a factor of a million difference in how well the early layers learn. That is the whole story of why activation choice matters, and this tool draws it.
The functions and their derivatives
Every activation used here is applied to a single scalar, the pre-activation x. What matters for training is the pair: the function f(x) and its derivative f'(x).
The sigmoid squashes any input into (0, 1). Its derivative is largest at x = 0, where \sigma(0) = 0.5 and \sigma'(0) = 0.25. Far from zero the output saturates near 0 or 1 and the derivative collapses toward 0.
Tanh is a rescaled sigmoid mapping to (-1, 1). Its derivative peaks at \tanh'(0) = 1, four times steeper than sigmoid, which is why tanh layers preserve gradients better than sigmoid layers. It still saturates at the tails.
ReLU is a hinge. For positive inputs it is the identity with slope 1; for negative inputs it is flat zero. No saturation on the positive side means no gradient shrinkage there, which is why deep ReLU networks train where deep sigmoid networks stall.
Leaky ReLU replaces the flat negative side with a gentle slope \alpha, typically 0.01. The derivative is 1 for positive inputs and \alpha for negative ones, so a unit pushed negative still receives a small gradient and can recover.
ELU bends the negative side into a smooth curve that saturates at -\alpha. It is differentiable everywhere and keeps a nonzero negative-side gradient near the origin.
Here \Phi is the standard normal cumulative distribution. GELU and Swish are the smooth ReLU relatives used in transformers. Both look like ReLU for large positive x, dip slightly below zero for small negative x, and are differentiable everywhere, so their derivative curves have no corner.
The corner in ReLU at x = 0 means the derivative is undefined exactly there. In practice code returns 0 or 1 at that single point and it never matters, because the probability of landing on it is zero.
A worked example you can check by hand
Reading f and f' at x = 1
Load the tool with its defaults and drag the vertical marker to x = 1. Here are the values you should read off, computed by hand.
- Sigmoid: \sigma(1) = 1/(1 + e^{-1}) = 1/1.368 = 0.7311. Derivative 0.7311 \times (1 - 0.7311) = 0.1966.
- Tanh: \tanh(1) = 0.7616. Derivative 1 - 0.7616^2 = 0.4200.
- ReLU: \max(0, 1) = 1. Derivative
1, since x \gt 0. - Swish: 1 \times \sigma(1) = 0.7311. Its derivative is \sigma(1) + 1 \cdot \sigma'(1) = 0.7311 + 0.1966 = 0.9277.
Notice the ranking of the derivatives at this point: ReLU 1.000, Swish 0.928, tanh 0.420, sigmoid 0.197. That order is exactly why gradients survive best through ReLU and Swish layers and worst through sigmoid layers.
The vanishing-gradient race
The second panel of the tool models backpropagation with one number: the typical derivative of the chosen activation, raised to the power of the layer count. If d is that per-layer derivative, the surviving gradient magnitude after L layers is
Plot g(L) on a log scale and it becomes a straight line with slope \log d. A steeper downhill line means faster vanishing. This is an idealization: real derivatives vary per input and weights rescale the gradient too, so treat the panel as the best case for saturating functions and a floor for ReLU, not a production forecast.
| Function | Per-layer d | L = 5 | L = 10 | L = 20 |
|---|---|---|---|---|
| Sigmoid | 0.25 | 0.000977 | 9.54e-7 | 9.09e-13 |
| Tanh | 1.0 | 1.0 | 1.0 | 1.0 |
| ReLU (active) | 1.0 | 1.0 | 1.0 | 1.0 |
| Sigmoid (typical) | 0.2 | 0.00032 | 1.02e-7 | 1.05e-14 |
The peak tanh and ReLU derivatives are both 1, so their idealized line is flat. That flat line is the whole reason these functions replaced sigmoid in deep networks. In reality tanh still saturates in its tails, so its real product sits between the flat line and the sigmoid line.
Reading the firing histogram and dying ReLU
The third panel trains a small ReLU network and counts, over the training inputs, how often each hidden unit produces a positive output. A healthy unit fires on some fraction of inputs, say 30 to 70 percent. A dead unit fires on exactly 0 percent.
A ReLU unit dies when its pre-activation is negative for every input. Then its output is always 0, its derivative is always 0, and backpropagation sends it zero gradient forever. It cannot update its way back to life. A large learning rate causes this: one oversized weight update can push the bias so negative that no input ever brings the pre-activation above zero.
If the histogram shows a growing spike of units at 0 percent firing as you raise the learning rate, those units are dead, not merely quiet. Lower the learning rate, or switch to Leaky ReLU or ELU, which keep a nonzero negative-side slope so a stunned unit still receives gradient and can recover.
When to use which
Default to ReLU for hidden layers of feed-forward and convolutional networks: it is cheap and trains deep stacks well. If you see many dead units, try Leaky ReLU with \alpha = 0.01 or ELU. For transformers, GELU and Swish are the common choices and tend to give slightly smoother training because they are differentiable everywhere.
Reserve sigmoid and tanh for places where their bounded range is the point. Sigmoid is right for a single output that must be a probability in (0, 1). Tanh is fine for a bounded hidden state, as in some recurrent cells. Do not stack many sigmoid layers as hidden activations: the 0.25 derivative cap makes deep gradients vanish, as the table above shows.
Common mistakes
- Sigmoid in deep hidden layers
- Each layer multiplies the gradient by at most
0.25. Ten layers gives at most 9.5 \times 10^{-7}, so early layers freeze. Use ReLU-family activations for hidden layers instead. - Learning rate too high with ReLU
- A single large update can kill units permanently. Watch the firing histogram; if the zero-percent bar grows, cut the learning rate.
- Expecting a smooth activation to fix a bad initialization
- GELU and Swish remove the corner, not the need for sensible weight scaling. Poor init still starves or explodes gradients regardless of the activation.
- Reading the vanishing panel as a real training curve
- It multiplies one representative derivative per layer. Real weights and per-input variation shift the actual product, so use the panel for intuition about the slope, not exact predictions.
Related tools
To see activations at work inside a full training loop, open the Neural Network Playground and watch a decision boundary bend. To trace exactly how a derivative enters a weight update, step through the Backpropagation Step-by-Step tool. For the depth story with real per-layer gradients rather than an idealized product, use Vanishing & Exploding Gradients. ReLU also appears between convolution and pooling in the Convolution Filter Visualizer.
Frequently asked questions
Why is ReLU better than sigmoid for deep networks?
ReLU has derivative exactly 1 for positive inputs, so a chain of active ReLU layers multiplies the gradient by 1 each step and it does not shrink. Sigmoid caps its derivative at 0.25, so ten layers scale the gradient by at most 0.25^{10} \approx 9.5 \times 10^{-7} and early layers barely learn.
What is the dying ReLU problem?
A ReLU unit whose pre-activation is negative for every training input always outputs 0 and always has derivative 0. It receives no gradient and cannot update, so it stays dead. A high learning rate is the usual cause.
How do I fix dead ReLU units?
Lower the learning rate first. If units still die, switch to Leaky ReLU (negative slope 0.01) or ELU, which keep a nonzero gradient on the negative side so a stunned unit can recover.
What is the difference between GELU and Swish?
Both multiply the input by a soft gate. GELU uses the normal CDF, x \cdot \Phi(x); Swish uses the sigmoid, x \cdot \sigma(x). Their shapes are very close, both are smooth ReLU relatives, and both are common in transformers.
Does the activation choice matter for a shallow network?
Less so. With one or two hidden layers there are too few derivative multiplications for vanishing to bite, so sigmoid, tanh and ReLU all train. The gap opens as depth grows, which is exactly what the vanishing panel shows.
Related tools
Backpropagation Step-by-Step
Walk through one forward and backward pass of a tiny neural network, number by number.
Handwritten Digit Recognizer
Draw a digit and watch a neural network trained on MNIST recognize it live in your browser.
Neural Network Playground
Train a tiny neural network live in your browser and watch its decision boundary learn four classic datasets.
Convolution Filter Visualizer
Slide a 3×3 kernel over an image and watch convolution, ReLU and max-pooling build the output pixel by pixel.
Hopfield Network
Store pixel patterns in a Hopfield memory, corrupt them, and watch the network recall the original.
Self-Attention Visualizer
See how Transformer tokens attend to each other with arcs, heatmaps, heads and causal masks.