ABC Tools

Calibration and Reliability Diagrams, Explained

After reading this you will be able to tell whether a classifier's probabilities mean what they say, measure the gap with ECE and Brier score, and fix a common failure mode with temperature scaling.

What calibration means

A classifier that outputs probabilities makes two separate claims. First, it ranks examples: higher score means more likely positive. Second, it puts a number on that likelihood. Calibration is only about the second claim. A model is calibrated when the numbers are honest: of all the times it says 0.70, about 70% turn out positive.

Here is the hook. Suppose a weather model predicts rain with probability 0.80 on 100 days. If it rained on 62 of those days, the model is overconfident: it claimed 80% but delivered 62%. The ranking might still be perfect (rainier days do get higher scores), yet the number 0.80 is a lie by 18 points. Calibration is what catches this.

Accuracy and AUC will not catch it, because both depend only on ranking. You can shift every probability toward 0 or 1 without changing which examples score higher, so accuracy stays fixed while calibration falls apart. That is exactly why calibration needs its own measurement.

When to check calibration, and when not to

Check calibration whenever the probability itself feeds a decision. Expected-cost thresholds, risk scores shown to a doctor, betting odds, and any pipeline that multiplies probabilities together all depend on the number being true, not just ordered correctly.

You can skip it when only the ranking matters. If you sort leads and call the top 100, or if a downstream stage re-thresholds anyway, then miscalibrated but well-ranked scores do no harm. In that case spend your effort on the threshold instead, which the ROC and Threshold Explorer covers.

Modern neural networks with softmax outputs are usually overconfident after training. A network that reaches 95% accuracy often reports average confidence near 0.99. Do not trust the raw softmax number as a probability until you have measured its calibration on held-out data.

Two numbers that summarize calibration

The reliability diagram is the picture, but you also want single numbers. Two are standard.

Expected calibration error (ECE) bins the predictions, then averages the gap between what each bin claimed and what it delivered, weighted by how many predictions land in the bin.

\text{ECE} = \sum_{b=1}^{B} \frac{n_b}{N} \, \left| \text{acc}_b - \text{conf}_b \right|

Here B is the number of bins, n_b the count of predictions in bin b, N the total, \text{acc}_b the observed positive frequency in the bin, and \text{conf}_b the mean predicted probability in the bin. ECE of 0 means perfect calibration. ECE of 0.1 means the probabilities are off by 10 points on average.

Brier score is the mean squared error of the probabilities themselves.

\text{Brier} = \frac{1}{N} \sum_{i=1}^{N} (p_i - y_i)^2

Here p_i is the predicted probability and y_i is the outcome, either 0 or 1. Lower is better. A model that always predicts 0.5 on balanced data scores 0.25. Brier rewards both calibration and sharpness: it drops when probabilities are honest and when they are confident and correct.

Temperature scaling, the fix

Temperature scaling recalibrates without touching the ranking. It takes the logit (the pre-sigmoid score) and divides by a single positive scalar T before the sigmoid.

\hat{p} = \sigma\!\left( \frac{z}{T} \right), \qquad z = \sigma^{-1}(p) = \ln\frac{p}{1 - p}

The logit z is recovered from the original probability p. Because T \gt 0 and dividing by a positive constant preserves order, the ranking never changes, so accuracy and AUC are untouched. Only the spread of the probabilities changes.

  • T \gt 1 softens: it pulls probabilities toward 0.5, fixing overconfidence.
  • T \lt 1 sharpens: it pushes probabilities toward 0 and 1, fixing underconfidence.
  • T = 1 does nothing.

You fit T by minimizing the negative log-likelihood on held-out data, which is a smooth 1-D search:

\text{NLL}(T) = -\sum_{i=1}^{N} \left[ y_i \ln \hat{p}_i + (1 - y_i) \ln (1 - \hat{p}_i) \right]

Because there is only one parameter, a simple scan over T from about 0.3 to 5 finds the minimum quickly. The fitted T also diagnoses the model: a value like 1.8 tells you the model was overconfident by that factor in logit space.

With temperature T = 1 the curve of recalibrated probability versus original probability is the identity diagonal. Raising T to 2 bows the curve toward the horizontal line at 0.5, so an input of 0.90 maps to about 0.71. Lowering T to 0.5 bows it the other way, so 0.90 maps to about 0.99. The endpoints at 0 and 1 stay fixed.

A worked example you can reproduce

Load the tool with its default settings (the demo button uses the field defaults: a Beta skew, an overconfident distortion at moderate strength, and sampled outcomes). To see the arithmetic by hand, take a smaller slice: five predictions with their true outcomes.

Computing ECE and Brier on five predictions

Five overconfident predictions and their outcomes
Prediction p_iOutcome y_iBinSquared error
0.9210.9 to 1.00.0064
0.8800.8 to 0.90.7744
0.8510.8 to 0.90.0225
0.7000.7 to 0.80.4900
0.6510.6 to 0.70.1225
  1. Brier score is the mean of the squared errors: (0.0064 + 0.7744 + 0.0225 + 0.4900 + 0.1225) / 5 = 0.2832.
  2. For ECE, group by bin. The 0.8 to 0.9 bin has two predictions with mean confidence (0.88 + 0.85)/2 = 0.865 and observed frequency 1/2 = 0.5, a gap of 0.365.
  3. The 0.9 to 1.0 bin: confidence 0.92, frequency 1.0, gap 0.08.
  4. The 0.7 to 0.8 bin: confidence 0.70, frequency 0.0, gap 0.70.
  5. The 0.6 to 0.7 bin: confidence 0.65, frequency 1.0, gap 0.35.
  6. Weight by bin size and average: (1(0.08) + 2(0.365) + 1(0.70) + 1(0.35)) / 5 = 1.86 / 5 = 0.372.

An ECE of 0.372 is large, but five points is far too few to bin reliably. This is why the tool samples hundreds of outcomes: with tiny bins, one flipped label swings the frequency by a lot.

Before fitting, observed frequency sags below the diagonal at high confidence (0.9 claimed, 0.72 delivered). After fitting T = 1.8 the green bars hug the diagonal.

Reading the reliability diagram

The diagram bins predictions along the x axis and plots observed frequency on the y axis. The diagonal is perfection. Where you sit relative to it tells you the failure mode.

Bars below the diagonal at high confidence
Overconfident. The model claims 0.9 but delivers 0.72. Fit T \gt 1 to soften.
Bars above the diagonal at high confidence
Underconfident. It claims 0.7 but delivers 0.85. Fit T \lt 1 to sharpen.
Bars above on the left and below on the right
A shift. The whole curve is rotated or offset around 0.5. One temperature cannot fix this.

Read the confidence histogram alongside it. If almost all predictions pile up in the 0.9 to 1.0 bin, then the ECE is dominated by that bin and the other bins are noise from a handful of points. Sharpness and calibration are different properties: a model can be perfectly calibrated by always predicting the base rate, which is useless, so you want honest probabilities that are also spread out.

Common mistakes

Never fit temperature on the same data you evaluate on. Fit T on a validation split, then report ECE and Brier on a separate test split. Fitting and scoring on one set understates the true error, the same leakage trap that the Cross-Validation Visualizer demonstrates.

Three more traps show up constantly.

  • Too many bins. ECE with 50 bins on 500 points averages 10 points per bin, so each observed frequency is noisy and ECE is inflated. Ten to fifteen bins is the usual range.
  • Trusting a one-parameter fix on a shift. Temperature scaling only rescales symmetrically around 0.5. If the miscalibration is asymmetric, the fitted curve will improve one end and worsen the other. Use Platt scaling (two parameters) or isotonic regression instead.
  • Confusing calibration with accuracy. Temperature scaling leaves accuracy exactly unchanged because it preserves ranking. If someone reports that recalibration boosted accuracy, something else changed.

Related tools on this site

Calibration is one facet of understanding a classifier. Pair this with the ROC and Threshold Explorer to see how ranking and thresholds behave independently of the probability values. The Bias-Variance Decomposition shows a different error split for regression. The Beta distribution that generates true probabilities here also drives the Bayesian Updating Visualizer, which is worth a look if the skew slider caught your interest.

Frequently asked questions

Does temperature scaling change my model's accuracy?

No. Dividing every logit by the same positive number cannot change which example scores higher, so the argmax and any fixed threshold decision are identical. Only the probability values move. If accuracy changed, you changed something besides temperature.

What is a good ECE value?

It depends on scale and stakes, but below 0.05 is usually considered well calibrated for a 10-bin estimate on a few thousand points. A raw deep network commonly starts above 0.10 and drops to around 0.02 after temperature scaling. Always report the number of bins and the test set size alongside it.

Why use Brier score if I already have ECE?

ECE measures only calibration and ignores sharpness, so a model that always predicts the base rate can score a low ECE while being useless. Brier score penalizes that: it rewards probabilities that are both honest and confident when correct. Report both.

My reliability diagram looks fine but ECE is high. Why?

Look at the confidence histogram. ECE weights each bin by its population, so a small gap in the crowded high-confidence bin can dominate while the eye is drawn to the sparse, empty-looking bins. The number and the picture emphasize different things.

When should I choose isotonic regression over temperature scaling?

Use isotonic regression when the miscalibration is not a simple symmetric stretch, for example an S-shaped or shifted reliability curve, and when you have enough validation data (a few thousand points) to fit the non-parametric step function without overfitting. On small data the single-parameter temperature is more stable.

Try it yourself in the tool

Related tools

ROC & Threshold Explorer

Drag a decision threshold and watch the confusion matrix, ROC curve and expected cost respond in real time.

Bayesian Updating Visualizer

Flip a coin and watch a Beta prior morph into a sharpening posterior in real time.

Bias–Variance Decomposition

See bias², variance and noise split a model’s error as 50 fits wobble around the truth.

Cross-Validation Visualizer

Animate k-fold splits and see why cross-validation beats a single train/test split.

Curse of Dimensionality

Watch distances concentrate and nearest neighbors fail as dimensions pile up.

Feature Scaling Demo

Run k-means and gradient descent on raw versus scaled features and watch scaling decide the outcome.