Gradient Boosting Step-by-Step
Gradient boosting builds a strong regressor as a sum of weak ones: each stage fits a shallow regression tree to the residuals of the current ensemble and adds it, scaled by the learning rate. Choose a true function (sine, step or kinked), the noise level and the number of points, then step through stages and watch the blue prediction curve bend toward the data while the orange residual bars shrink; the chart below traces train and test MSE per stage, with test error eventually turning upward as the ensemble starts fitting noise. A stage slider scrubs back through the whole history, and a toggle swaps the depth-2 trees for decision stumps to show how weaker learners need more stages.
Runs 100% in your browser — models are trained and computed locally on your device.
Read the full guide to this tool
Notes
- Update rule: Fₘ = Fₘ₋₁ + ν·hₘ, where hₘ is fitted to the residuals y − Fₘ₋₁(x) — the negative gradient of squared loss — and ν is the shrinkage.
- A smaller learning rate needs more stages to reach the same training fit but usually generalises better.
- Training error only ever decreases with stages; test error is U-shaped, which is why real boosters rely on early stopping.
- With squared loss, stage one fits the data mean, so the ensemble always starts from a flat line.
- Runs 100% in your browser — models are trained and computed locally on your device.
Related tools
Decision Tree Builder
Grow a CART decision tree split by split and watch the partition map and tree diagram build together.
Function Sketch Finder
Sketch a curve on a zoomable coordinate grid and search dozens of function families for the equations that fit it best.
Linear Regression Playground
Place points, watch the least-squares line follow, and see gradient descent crawl across the loss surface.
Overfitting Sandbox
Crank a polynomial’s degree on noisy data and watch the train/test error curves split apart.
Logistic Regression Playground
Paint two classes of points and watch logistic regression carve out its probability boundary by gradient descent.
Naive Bayes Classifier Demo
Score your own message with a word-count spam filter, or watch Gaussian naive Bayes classify 2D points.