Feature Scaling, Explained

After reading this you will know why k-means and gradient descent break when one feature is measured on a bigger scale, how standardization, min-max and robust scaling fix that, and how to read the difference off numbers you can check by hand.

What feature scaling does and one example that shows why

Feature scaling is a rescaling of each column of your data so that no single feature dominates just because it happens to be measured in larger units. Distance-based methods (k-means, k-nearest neighbors, SVMs with an RBF kernel) and gradient-based methods (anything trained by gradient descent) silently assume the features share a comparable range. When they do not, the algorithm answers a different question than the one you asked.

Here is the hook. Take two clean clusters in 2-D. Cluster A sits near (1, 0.5) and cluster B near (3, 0.7). The true grouping is left versus right along feature 1. Now stretch feature 2 by a factor of 1000, so those 0.5 and 0.7 values become 500 and 700. Run k-means with k = 2. The Euclidean distance is now dominated by feature 2, and k-means splits the data along that axis instead. The clusters it returns have almost nothing to do with the real structure.

Scale both features to comparable ranges first and the same algorithm recovers the true left-versus-right split. The algorithm did not change. Only the units did.

When scaling matters and when it does not

Scaling changes the answer for any method that compares features using a distance or a gradient in the raw coordinate space.

Needs scaling

k-means, k-nearest neighbors, SVMs, PCA, and any linear or neural model trained by gradient descent. All of these treat "one unit of feature 1" and "one unit of feature 2" as directly comparable.

Indifferent to scaling

Decision trees, random forests and gradient-boosted trees. They split one feature at a time at a threshold. A monotone rescaling (multiply by 1000, take a log) moves the threshold but never changes which points fall on each side, so the tree is identical.

The three scalers and their formulas

Standardization centers each feature at zero and divides by its standard deviation:

Here x_i is one raw value, \bar{x} is the feature mean and \sigma is its standard deviation. The result has mean 0 and variance 1. A value one standard deviation above the mean becomes +1 regardless of the original units.

Min-max scaling maps the observed range onto [0, 1]:

The smallest value becomes 0, the largest becomes 1, everything else lands in between. This preserves the shape of the distribution but is sensitive to the extremes, because a single large outlier sets x_{\max}.

Robust scaling uses the median and the interquartile range (IQR), the gap between the 75th and 25th percentiles:

The median and IQR ignore the far tails, so a handful of outliers cannot move the center or blow up the divisor. That is the whole reason robust scaling exists.

A worked example on the demo data

The next chart shows the same six points before and after standardization. Raw feature 2 dwarfs feature 1; standardized, they are comparable.

Why gradient descent zigzags on raw data

For linear regression the loss is a bowl. Its curvature is described by the Hessian, and how fast gradient descent converges depends on the condition number \kappa, the ratio of the largest to the smallest eigenvalue of that Hessian.

When one feature has a scale 1000 times another, the bowl becomes a long thin valley. The eigenvalue along the stretched direction is roughly the square of the scale ratio times the eigenvalue along the other, so \kappa can reach the order of 10^6. Gradient descent with a fixed step size must keep the step small enough for the steep direction (the largest safe rate shrinks as 1/\lambda_{\max}), which leaves it crawling along the shallow direction. The path bounces back and forth across the valley.

The number of iterations to reach a fixed accuracy grows roughly in proportion to \kappa. Scale the features and the eigenvalues become nearly equal, so \kappa \approx 1, the contours are near-circles, and the descent runs almost straight to the minimum.

Reading the two panels and the numbers

The demo reports two figures per panel: iterations to converge for gradient descent, and cluster accuracy for k-means. Read them as a pair.

Cluster accuracy near 0.5 on the raw side is the signature of a two-way split that ignores the true labels: the algorithm is grouping by the dominant axis and getting the true grouping right only by chance. On the scaled side, accuracy near 0.99 means the recovered clusters match the true ones almost everywhere. For gradient descent, the ratio of iterations (about 7455 / 12 = 620) tracks the condition number, not the sample size.

Common mistakes

Scaling the target instead of, or alongside, the features by accident. Scale the inputs. If you also scale the target you must remember to undo it before reading predictions in original units.

Using min-max when the data has outliers. One extreme value sets x_{\max}, so the entire remaining range gets squashed into a narrow band near zero. Add a single point at feature 2 = 5000 to the demo and watch min-max collapse the real spread. Robust scaling keeps the median and IQR steady because it reads the middle of the sorted data, not the tail.

Assuming scaling helps a tree model. It does not. Time spent scaling before a random forest is wasted, and it can make the model harder to interpret because splits no longer read in original units.

Related tools

Scaling changes distances, which is exactly what fails as dimensions grow: the Curse of Dimensionality demo shows nearest neighbors breaking down for a related reason. Once a model is trained, tune its decision boundary with the ROC & Threshold Explorer, split its error into bias and variance with the Bias-Variance Decomposition, and confirm your evaluation is honest with the Cross-Validation Visualizer, which is where you enforce the "fit the scaler on training data only" rule fold by fold.

Related tools

ROC & Threshold Explorer

Drag a decision threshold and watch the confusion matrix, ROC curve and expected cost respond in real time.

Bayesian Updating Visualizer

Flip a coin and watch a Beta prior morph into a sharpening posterior in real time.

Bias–Variance Decomposition

See bias², variance and noise split a model’s error as 50 fits wobble around the truth.

Cross-Validation Visualizer

Animate k-fold splits and see why cross-validation beats a single train/test split.

Calibration & Reliability Diagram

Check whether predicted probabilities mean what they say, then fix them with temperature scaling.

Curse of Dimensionality

Watch distances concentrate and nearest neighbors fail as dimensions pile up.