ABC Tools

PCA, Explained

After reading this you will know how principal component analysis finds the directions of largest variance in a point cloud, how to compute those directions by hand for a small example, and how to read the variance-explained bars and reconstruction error the tool shows.

What PCA does, with one example

Principal component analysis rotates your data so that the new axes point along the directions of largest spread, then lets you keep only the first few axes. It is a change of coordinates followed by an optional truncation.

Picture a thin, tilted cloud of points on a plane. The cloud stretches along a diagonal from lower-left to upper-right and is narrow across that diagonal. Your original x and y axes each capture some of the spread, but neither points along the cloud. PCA finds the diagonal (the first principal component) and the direction across it (the second). If you keep only the first, you flatten the two-dimensional cloud onto a line while losing as little information as possible.

That is the whole idea. In two dimensions it looks like a small trick. In 50 or 500 dimensions it becomes the standard way to compress data, remove correlated noise, and draw high-dimensional structure on a flat screen.

When to use it, and when not to

Reach for PCA when your features are numeric, roughly on comparable scales, and correlated with each other. Correlation is what PCA exploits: if two features move together, one combined axis carries most of their information and the other can be dropped.

Do not expect PCA to help with these cases:

  • Nonlinear structure. A spiral or a curled sheet has variance in every direction, so no straight axis unfolds it. Use a nonlinear method instead.
  • Wildly different scales. A feature measured in millimeters and one measured in kilometers will not compare fairly until you standardize.
  • Interpretable individual features. Each principal component is a weighted mix of all original features, so it rarely maps to one nameable quantity.

PCA is unsupervised: it never looks at labels. The direction of largest variance is not always the direction that separates your classes. A feature that spreads points widely can still be useless for prediction, and a low-variance feature can be the one that matters.

The covariance matrix and its eigenvectors

Center the data first by subtracting the mean \mu of each feature. Then build the covariance matrix.

C = \frac{1}{n} \sum_{i=1}^{n} (x_i - \mu)(x_i - \mu)^{\top}

Here x_i is one data point written as a column vector, \mu is the mean vector, n is the number of points, and C is a square matrix whose diagonal holds the variance of each feature and whose off-diagonal entries hold the covariance between feature pairs. For two features C is 2 \times 2.

PCA finds the eigenvectors and eigenvalues of C:

C v = \lambda v

An eigenvector v is a direction that C stretches without turning, and its eigenvalue \lambda is the amount of stretch, which equals the variance of the data measured along v. Sort the eigenvectors by eigenvalue from largest to smallest. The first is the first principal component, the direction of maximum variance. Because C is symmetric, the eigenvectors are mutually orthogonal, so the new axes are perpendicular.

The tool draws each principal axis as an arrow scaled by \sqrt{\lambda}, the standard deviation along that axis, so the arrow length is in the same units as the data.

A worked example you can reproduce

Five points by hand

Take five points chosen so the arithmetic stays clean:

(2, 1), (3, 3), (4, 3), (5, 5), (6, 3)

  1. Mean. The x values sum to 20, so \mu_x = 4. The y values sum to 15, so \mu_y = 3. The mean is (4, 3).
  2. Center. Subtract the mean to get (-2, -2), (-1, 0), (0, 0), (1, 2), (2, 0).
  3. Covariance entries. With n = 5: the x variance is (4 + 1 + 0 + 1 + 4)/5 = 2.0. The y variance is (4 + 0 + 0 + 4 + 0)/5 = 1.6. The covariance is (4 + 0 + 0 + 2 + 0)/5 = 1.2.
  4. Matrix. So C = \begin{bmatrix} 2.0 & 1.2 \\ 1.2 & 1.6 \end{bmatrix}.
  5. Eigenvalues. Solve \lambda^2 - 3.6\lambda + (2.0 \cdot 1.6 - 1.2^2) = 0, that is \lambda^2 - 3.6\lambda + 1.76 = 0. The roots are \lambda = 1.8 \pm \sqrt{1.8^2 - 1.76} = 1.8 \pm \sqrt{1.48}, giving \lambda_1 \approx 3.017 and \lambda_2 \approx 0.583.
  6. First eigenvector. For \lambda_1 = 3.017, solve (2.0 - 3.017)v_x + 1.2 v_y = 0, so v_y / v_x = 1.017 / 1.2 \approx 0.847. Normalized, v_1 \approx (0.763, 0.646).

The total variance is 2.0 + 1.6 = 3.6, which equals \lambda_1 + \lambda_2 = 3.017 + 0.583. That is always true: the trace of C is preserved, so the eigenvalues split the total variance among the components.

The first component explains 3.017 / 3.6 \approx 0.838, about 84 percent of the variance. The second explains the remaining 16 percent.

PC1 carries 0.838 of the total variance and PC2 carries 0.162. The two bars always sum to 1.

Reconstruction error and what the bars mean

When you press Project, every point slides onto the first principal component and the tool reports the mean squared reconstruction error: the average squared distance each point moved. That error is not arbitrary. It equals the sum of the eigenvalues you discarded.

\text{MSE} = \sum_{j=k+1}^{d} \lambda_j

Here d is the original number of dimensions, k is how many components you keep, and the sum runs over the dropped eigenvalues. In the worked example, keeping only PC1 discards \lambda_2 = 0.583, so the mean squared error of the projection is exactly 0.583. No other line through the cloud gives a smaller error. That optimality is the reason PCA is the default linear compressor.

To choose how many components to keep, add the variance-explained bars from the left until the running total crosses a threshold you decide in advance, often 0.90 or 0.95. In the example, PC1 alone reaches 0.838, so if your threshold is 0.90 you keep both components.

See variance rotate under your control

Variance along a direction is a smooth function of the angle of that direction. It peaks exactly at the first principal component and bottoms out at the second, ninety degrees away. The widget lets you sweep the angle and watch the projected variance rise and fall.

For the five-point example, projecting onto a line at angle θ gives variance that rises to 3.017 at θ ≈ 40.3° (the PC1 direction) and falls to 0.583 at θ ≈ 130.3° (the PC2 direction). The two extremes are exactly 90° apart and sum to the total variance 3.6.

Common mistakes

Skipping standardization
PCA reads raw variance. Multiply one feature by 100 and its variance grows by 10000, so it dominates PC1 for no good reason. Convert each feature to zero mean and unit standard deviation first unless the units are already comparable.
Forgetting to center
If you skip subtracting the mean, the first component points toward the data's center of mass rather than its direction of spread. The tool marks the mean for exactly this reason.
Reading sign into the components
An eigenvector and its negative are both valid, so PC1 may point up-right in one run and down-left in another. Only the axis matters, not the arrow's head.
Expecting curved structure to unfold
PCA only rotates and projects. If the interesting structure is a curve, projecting onto a straight axis destroys it.
Treating variance as importance
A high-variance component is not automatically the useful one for a downstream task. Variance measures spread, not relevance to a label.

Related tools

PCA is often a preprocessing step before clustering, because clustering in a lower-dimensional space is faster and less noisy. After you flatten a cloud, try k-Means Clustering to partition it into groups, or DBSCAN Clustering and Hierarchical Clustering & Dendrogram when the cluster count is unknown. For soft, elliptical clusters, Gaussian Mixture & EM models the same covariance structure PCA reads off.

When PCA fails because the structure is nonlinear, compare it against t-SNE Explorer for laying out clusters in 2D, or a Self-Organizing Map for a grid that learns the data's shape. If your goal is finding rare points rather than compressing, the Anomaly Detection Playground shows how reconstruction error becomes an outlier score.

Frequently asked questions

How many principal components should I keep?

Keep enough to reach a variance-explained total you set in advance, such as 0.95. Sum the eigenvalues from largest down and stop when the cumulative fraction crosses your target. In the five-point example, one component reaches 0.838 and two reach 1.0.

Is PCA the same as linear regression?

No. Regression minimizes vertical distance from points to a line, treating y as a target. PCA minimizes perpendicular distance and treats all features symmetrically. On the same data the two lines usually differ.

Why do the principal axes have to be perpendicular?

Because the covariance matrix is symmetric, and symmetric matrices always have orthogonal eigenvectors. Perpendicular axes also mean the projected components are uncorrelated, which is part of what makes the compression efficient.

Does PCA lose information?

Only when you drop components. Keeping all d components is a lossless rotation with zero reconstruction error. Dropping the smallest ones loses an amount exactly equal to the sum of their eigenvalues.

What does the arrow length in the tool mean?

Each principal axis is drawn scaled by \sqrt{\lambda}, the standard deviation of the data along that axis. A longer arrow means the cloud spreads more in that direction, so PC1's arrow is always at least as long as PC2's.

Try it yourself in the tool

Related tools

Distribution Sketcher

Sketch the shape of a probability density — or paste raw data — and search 16 distributions for the ones that fit it best.

k-Means Clustering

Step through Lloyd’s algorithm and watch centroids chase the middles of your point clouds.

Anomaly Detection Playground

Flag outliers three different ways and see where each score breaks down.

DBSCAN Clustering

Grow density-based clusters point by point and watch noise get left behind.

Gaussian Mixture & EM

Fit soft, elliptical clusters with the expectation–maximization algorithm.

Hierarchical Clustering & Dendrogram

Merge points bottom-up and cut the dendrogram wherever you like.