Gradient Descent, Explained
After reading this you will be able to write down the update rule for SGD, Momentum, RMSProp and Adam, work each one by hand on a small quadratic, and read a contour-map race to see why one optimizer stalls while another overshoots.
What gradient descent does
Gradient descent minimises a function f(\theta) by repeatedly stepping in the direction that decreases it fastest. That direction is the negative gradient, -\nabla f. You pick a start point, compute the slope, take a small step downhill, and repeat.
The hook is the ravine. Take the loss f(x, y) = x^2 + 10y^2. The bowl is 10 times steeper along y than along x. Plain gradient descent picks a single step size for both directions, so the step that is safe for the steep y wall is tiny for the shallow x floor. The iterates zigzag across the ravine and crawl along it. That single picture explains why Momentum, RMSProp and Adam exist: each one changes how the raw gradient becomes an actual step.
The visualizer races the four methods from one shared start on five landscapes: a convex bowl, that ill-conditioned ravine, the banana-shaped Rosenbrock valley, a saddle point, and Himmelblau's function with its four separate minima.
When to use it, and when not
Gradient descent is the right tool when you can compute a gradient cheaply and the function is smooth enough that the local slope points somewhere useful. That covers most neural network training, logistic regression and least squares. All four variants here assume a differentiable objective.
It is the wrong tool in three common cases. First, when the objective is not differentiable or is riddled with flat plateaus and discrete choices, as in routing a travelling salesman tour. Second, when the landscape has many bad local minima and you have no gradient at all, where a population or temperature-based search fits better. Third, when the problem is small and convex enough to solve in closed form: for a quadratic you can just solve \nabla f = 0 directly.
Gradient descent finds a point where the gradient is zero. On a convex bowl that is the global minimum. On Himmelblau's four-minimum surface it finds whichever minimum is downhill from your start, and different start points reach different minima. The method never promises a global answer on a non-convex surface.
The four update rules
Write g_t = \nabla f(\theta_t) for the gradient at step t and \alpha for the learning rate. Plain stochastic gradient descent is the baseline:
Here \theta_t is the current position, g_t is the slope, and \alpha scales the step. Every coordinate gets the same \alpha. That is the whole problem on an ill-conditioned surface.
Momentum keeps a running velocity v that averages recent gradients, so consistent directions build up speed and oscillating ones cancel:
The decay \beta is usually 0.9. Along the ravine floor the gradient always points the same way, so v grows and the step lengthens. Across the ravine the sign flips each step, so the terms partly cancel and the zigzag shrinks.
RMSProp instead rescales each coordinate by a running root-mean-square of its own gradients:
The average of squared gradients s is large in steep directions and small in flat ones, so dividing by \sqrt{s} shrinks the big steps and grows the small ones. The decay \rho is typically 0.9 and \epsilon \approx 10^{-8} guards against division by zero.
Adam combines both ideas and adds bias correction because m and v start at zero:
Here m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t is the momentum term and v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2 is the RMSProp term, with defaults \beta_1 = 0.9 and \beta_2 = 0.999.
The stability limit you can see
On one quadratic direction with curvature \lambda, plain gradient descent multiplies the distance to the minimum by 1 - \alpha\lambda each step. Convergence needs that factor to have magnitude below 1, which gives the exact stability rule:
Take f(x) = 5x^2, so \lambda = 10 and the limit is \alpha \lt 0.2. At \alpha = 0.1 the factor is 1 - 1 = 0, so you reach the minimum in one step. At \alpha = 0.19 the factor is -0.9, so you overshoot and slowly converge. At \alpha = 0.21 the factor is -1.1, and every step grows by 10 percent. That is exactly the divergence the tool warns about when you drag the slider up.
A worked race on the ravine
Reproduce the demo defaults: the ravine f(x, y) = x^2 + 10y^2, start point (x, y) = (2, 1), learning rate \alpha = 0.05. The gradient is \nabla f = (2x, 20y). At the start it is (4, 20).
Plain SGD, first three steps
- At (2, 1) the gradient is (4, 20). Step: x = 2 - 0.05 \cdot 4 = 1.8, y = 1 - 0.05 \cdot 20 = 0.
- At (1.8, 0) the gradient is (3.6, 0). Step: x = 1.8 - 0.18 = 1.62, y = 0.
- At (1.62, 0) the gradient is (3.24, 0). Step: x = 1.62 - 0.162 = 1.458.
The y coordinate collapsed to 0 in one step because 1 - 0.05 \cdot 20 = 0. But x only shrinks by the factor 1 - 0.05 \cdot 2 = 0.9 each step. After 3 steps x = 1.458, after 10 steps 2 \cdot 0.9^{10} \approx 0.697. The steep direction is done immediately, the shallow direction crawls.
RMSProp rescales the crawl
Use \rho = 0.9, \epsilon = 10^{-8}. At the start g = (4, 20), so s = 0.1 \cdot (16, 400) = (1.6, 40). The x step becomes 0.05 \cdot 4 / \sqrt{1.6} \approx 0.158, and the y step becomes 0.05 \cdot 20 / \sqrt{40} \approx 0.158. Both coordinates move about the same distance. That is the point of dividing by \sqrt{s}: it flattens the difference in curvature so the shallow direction no longer lags.
Reading the contour race
Each optimizer leaves a coloured trail with a marker at its current position, and its loss is printed live. Three patterns are worth naming.
- Zigzag
- Sharp back-and-forth across a ravine means the step size is too large for the steep direction. Plain SGD shows this most. Momentum damps it because the alternating gradients cancel in the velocity.
- Stall
- A marker that stops moving sits at a point with near-zero gradient. On the saddle f = x^2 - y^2, a start exactly on the x-axis has gradient (2x, 0) in
y, so it slides to the origin and stops even though the origin is not a minimum. - Divergence
- A trail that flies off the plot with a rising loss means \alpha exceeded the stability limit for that landscape. The tool flags it.
When comparing methods, hold the learning rate fixed and watch which trail reaches the lowest loss first. Adam and RMSProp usually win on the ravine and Rosenbrock because they rescale per coordinate. On the plain convex bowl, all four converge and plain SGD is often as fast, since there is no ill-conditioning to correct.
Common mistakes
The first mistake is reading one landscape as the whole story. Adam can lose to plain SGD on a simple bowl. Try all five surfaces before you conclude anything.
The second is blaming the optimizer for a learning rate problem. If every method diverges, lower \alpha before switching methods. If every method crawls, raise it. The slider is log-scaled precisely because the useful range spans several orders of magnitude.
The third is expecting escape from a saddle without a nudge. Exactly on the symmetry axis the sideways gradient is zero, so the iterate stalls forever in exact arithmetic. In the tool, click slightly off the axis and watch it break free.
These are two-dimensional toy landscapes with exact gradients and a handful of iterations. They show the shape of each update rule clearly, but they say nothing about training a network with millions of parameters and noisy mini-batch gradients. Adam's dominance here does not guarantee it wins on your dataset. Treat the demo as intuition, not evidence.
Related tools
Gradient descent needs a smooth, differentiable objective. When you have neither, other search methods apply. To minimise a discrete tour length with no gradient, see Simulated Annealing TSP, which accepts uphill moves early and cools toward a good tour. For a gradient-free population search on the same kind of test functions used here, see Particle Swarm Optimization. And if your problem is tracking a moving state through noisy measurements rather than minimising a fixed function, the Kalman Filter Visualizer shows the recursive estimator for that case.
Frequently asked questions
What is the difference between gradient descent and stochastic gradient descent?
Full gradient descent computes the gradient over all data at each step. Stochastic gradient descent estimates it from a random subset, which is noisier but far cheaper per step. In this visualizer the loss is a fixed math function, so the gradient is exact; "SGD" here just names the plain update rule \theta - \alpha g without momentum or rescaling.
Why does Adam sometimes lose to plain SGD?
Adam adapts each coordinate's step to its recent gradient history. On a well-conditioned convex bowl there is nothing to adapt, and Adam's per-coordinate scaling can undershoot compared with a single well-tuned learning rate. Its advantage appears on ill-conditioned or curved landscapes like the ravine and Rosenbrock.
How do I choose the learning rate?
Start below the stability limit 2/\lambda for the steepest curvature you expect, then increase until the trail begins to oscillate, then back off. In the tool, raise the slider until one method diverges and pick a value a factor of 2 or 3 below that.
Why does one optimizer get stuck at a saddle point?
A saddle has zero gradient without being a minimum. On f = x^2 - y^2 a start on the x-axis has no force in the escaping y direction, so plain SGD stalls at the origin. Momentum and any small sideways offset break the symmetry and let the iterate roll down the y valley.
What do beta and rho mean in the formulas?
They are decay rates for running averages, both near 0.9. A value of 0.9 means each new gradient contributes 10 percent and the old average keeps 90 percent, so the average reflects roughly the last 10 gradients. Adam's \beta_2 = 0.999 averages over roughly the last 1000, giving a very smooth denominator.
Related tools
Kalman Filter Visualizer
Track a moving target through noisy measurements with a full linear Kalman filter.
Simulated Annealing TSP
Watch simulated annealing untangle a travelling-salesman tour as the temperature falls.
Particle Swarm Optimization
Fly a swarm of particles across classic test functions and watch it converge on the minimum.
AI Learns to Drive (Neuroevolution)
Watch a population of neural-network cars evolve from crashing into corners to lapping a race track.
Backpropagation Step-by-Step
Walk through one forward and backward pass of a tiny neural network, number by number.
Decision Tree Builder
Grow a CART decision tree split by split and watch the partition map and tree diagram build together.