Vanishing & Exploding Gradients
Stack enough layers and the gradient reaching the early ones is a long product of factors — if those factors are typically below one it vanishes, above one it explodes. This playground trains a genuinely deep multilayer perceptron (pick 2 to 30 hidden layers, 16 units wide) on a tiny two-blob classification task, and draws a horizontal bar per layer showing the mean absolute gradient on a log scale, updating live as SGD runs. Choose the weight initialization — naive N(0,1), small, Xavier or He — and the activation — sigmoid, tanh or ReLU — then press Play: with sigmoid and naive init the first layers starve while the loss flat-lines; switch to He initialization with ReLU and every layer gets a healthy signal. A loss curve and the first-to-last-layer gradient ratio track the difference numerically.
Runs 100% in your browser — models are trained and computed locally on your device.
Read the full guide to this tool
Notes
- Through L layers the gradient scales roughly like ∏ wℓ·f′(zℓ): sigmoid contributes factors ≤ 0.25, so depth 20 with naive init shrinks signals by many orders of magnitude.
- Xavier initialization draws weights with variance 1/n to keep activation variance constant across layers (suited to tanh); He initialization uses 2/n to compensate for ReLU zeroing half its inputs.
- Exploding gradients are the mirror problem: weight variance above the critical scale multiplies signals up until the loss turns NaN — in practice gradient clipping caps the update norm.
- Residual connections, batch normalization and LSTM gating were all invented largely to keep this product of factors near one in very deep networks.
- Runs 100% in your browser — models are trained and computed locally on your device.
Related tools
Backpropagation Step-by-Step
Walk through one forward and backward pass of a tiny neural network, number by number.
Handwritten Digit Recognizer
Draw a digit and watch a neural network trained on MNIST recognize it live in your browser.
Neural Network Playground
Train a tiny neural network live in your browser and watch its decision boundary learn four classic datasets.
Activation Function Explorer
Compare sigmoid, tanh, ReLU and friends with their derivatives, and see why deep gradients vanish.
Convolution Filter Visualizer
Slide a 3×3 kernel over an image and watch convolution, ReLU and max-pooling build the output pixel by pixel.
Hopfield Network
Store pixel patterns in a Hopfield memory, corrupt them, and watch the network recall the original.