Activation Function Explorer
Activation functions give neural networks their nonlinearity, and their derivatives decide how well gradients survive the trip backwards through many layers. The top chart plots any selection of sigmoid, tanh, ReLU, Leaky ReLU (with an adjustable slope), ELU, GELU and Swish, each with its derivative dashed underneath; drag the vertical marker to read off f(x) and f′(x) at any input. The second panel multiplies the typical derivative of each chosen function across 1–50 layers and plots the surviving gradient magnitude on a log scale — the vanishing-gradient problem drawn as a straight race downhill. The third panel trains a tiny ReLU network and histograms how often each unit fires: raise the learning rate and watch units die, stuck at zero forever.
Runs 100% in your browser — models are trained and computed locally on your device.
Read the full guide to this tool
Notes
- Saturating activations squash: σ′(x) ≤ 0.25 everywhere, so ten sigmoid layers scale a gradient by at most 0.25¹⁰ ≈ 10⁻⁶ — early layers barely learn.
- ReLU(x) = max(0, x) has derivative exactly 1 for active units, which is why deep ReLU networks train where deep sigmoid networks stall.
- A ReLU unit whose pre-activation goes negative for every input receives zero gradient and can never recover — the “dying ReLU” problem; Leaky ReLU and ELU keep a small negative-side slope to avoid it.
- GELU(x) ≈ x·Φ(x) and Swish x·σ(x) are smooth ReLU relatives used in transformers; unlike ReLU they are differentiable everywhere.
- Runs 100% in your browser — models are trained and computed locally on your device.
Related tools
Backpropagation Step-by-Step
Walk through one forward and backward pass of a tiny neural network, number by number.
Handwritten Digit Recognizer
Draw a digit and watch a neural network trained on MNIST recognize it live in your browser.
Neural Network Playground
Train a tiny neural network live in your browser and watch its decision boundary learn four classic datasets.
Convolution Filter Visualizer
Slide a 3×3 kernel over an image and watch convolution, ReLU and max-pooling build the output pixel by pixel.
Hopfield Network
Store pixel patterns in a Hopfield memory, corrupt them, and watch the network recall the original.
Self-Attention Visualizer
See how Transformer tokens attend to each other with arcs, heatmaps, heads and causal masks.