ABC Tools

Shannon Entropy and Diversity, Explained

After reading this you can turn a list of category counts into three numbers that describe how varied your data is: Shannon entropy, evenness, and the Simpson index. You will also know how to read each one and where they mislead.

What these numbers measure

Suppose you sort 100 items into categories and count how many land in each. Shannon entropy, evenness, and the Simpson index all answer a single question in different ways: how spread out are the items across categories?

Two extremes make the idea concrete. If all 100 items fall in one category, there is no variety. Pick an item at random and you know its category with certainty. If the items split evenly, say 25 in each of four categories, variety is at its maximum for four categories. Pick an item and you have the least chance of guessing right.

Entropy puts a number on that uncertainty. Higher entropy means more spread and less predictability. The demo counts 10, 20, 30, 40 sit between the two extremes, and by the end of this article you will be able to reproduce their entropy of about 1.279 bits by hand.

When to use it, and when not to

Reach for these measures when your data is a set of counts across unordered categories: species in a field survey, tickets by support queue, words by type, files by extension. The categories have no natural order and no distance between them, so a mean or standard deviation makes no sense. Diversity indices are built for exactly this case.

Do not use them on ordered or continuous data. If your values are heights, prices, or test scores, summarize them with the Descriptive Statistics Calculator instead. Entropy would ignore the order and throw away most of the information.

Entropy depends only on the proportions, not on the labels or the raw totals. The counts 10, 20, 30, 40 and 100, 200, 300, 400 give the identical entropy, because both describe the shares 0.1, 0.2, 0.3, 0.4.

The formula and the intuition

Let there be k categories. Turn the counts into proportions p_i by dividing each count by the total. Shannon entropy is:

H = -\sum_{i=1}^{k} p_i \log_2 p_i

Here p_i is the share of items in category i, and the sum runs over all categories with a positive count. Using \log_2 gives entropy in bits. Some fields use the natural log (units called nats) or \log_{10} (units called bans). The shape of every conclusion is the same; only the scale changes.

Why the -\log_2 p_i term? It is the "surprise" of seeing category i. A rare category with p_i = 0.01 carries -\log_2 0.01 \approx 6.64 bits of surprise. A common one with p_i = 0.5 carries only 1 bit. Entropy is the average surprise, weighted by how often each category appears.

The maximum entropy for k categories is \log_2 k, reached when every category is equally likely. For 4 categories that ceiling is \log_2 4 = 2 bits. Evenness rescales entropy against that ceiling:

J = \frac{H}{\log_2 k}

Evenness J runs from 0 (all in one category) to 1 (perfectly even). It strips out the effect of category count so you can compare a survey with 4 categories against one with 40.

The Simpson index takes a different angle. The Gini-Simpson form is:

D = 1 - \sum_{i=1}^{k} p_i^2

The sum \sum p_i^2 is the chance that two items drawn at random land in the same category. Subtract from 1 and you get the chance they differ, which is another honest measure of variety.

Worked example with the demo data

Entropy of 10, 20, 30, 40

  1. Total the counts: 10 + 20 + 30 + 40 = 100.
  2. Convert to proportions: 0.1, 0.2, 0.3, 0.4.
  3. Compute each term -p_i \log_2 p_i. For 0.1: -0.1 \times \log_2 0.1 = -0.1 \times (-3.3219) = 0.3322. For 0.2: 0.4644. For 0.3: 0.5211. For 0.4: 0.5288.
  4. Add the terms: 0.3322 + 0.4644 + 0.5211 + 0.5288 = 1.2465 bits.

The calculator reports about 1.2465 bits. Divide by the ceiling \log_2 4 = 2 to get evenness J = 1.2465 / 2 = 0.623. The data is fairly uneven: the largest category holds 40 percent of the items, and the smallest holds 10.

For Simpson, sum the squares: 0.01 + 0.04 + 0.09 + 0.16 = 0.30. So D = 1 - 0.30 = 0.70. Two random items differ 70 percent of the time.

The rarest category (0.1) contributes the most surprise per item but appears rarely, so its entropy term is the smallest. The terms stay close because none of the four shares is extreme.

Watch entropy respond to the shares

With four categories, entropy is 0 bits when one category holds everything and rises to its maximum of 2 bits when all four shares equal 0.25. The demo split of 0.1, 0.2, 0.3, 0.4 gives 1.2465 bits, or an evenness of 0.623.

Reading the results

Read the three numbers together, since each hides something the others reveal.

Shannon entropy (bits)
Average uncertainty per item. Compare it to the maximum \log_2 k for your category count. A value of 1.2465 out of a possible 2 means you are using about 62 percent of the available variety.
Evenness
Entropy divided by its maximum, so it lives on a fixed 0 to 1 scale regardless of how many categories you have. Use it to compare datasets with different numbers of categories.
Simpson diversity
Probability that two randomly picked items differ. It is less sensitive to rare categories than entropy, so it emphasizes the common ones.

A useful reframe of entropy is the "effective number of categories," 2^H. For the demo, 2^{1.2465} \approx 2.37. The four real categories behave like about 2.4 equally common ones, because two of them dominate.

Common mistakes

The errors below change the numbers in ways that are easy to miss.

Do not feed proportions that already sum to 1 alongside raw counts and expect them to mix. The calculator divides by the total, so keep the input in one consistent unit. Counts like 10, 20, 30, 40 are fine; do not append a stray 0.5.

Comparing entropy across datasets with different category counts is unfair. A survey with 40 categories can reach \log_2 40 \approx 5.32 bits, while a 4-category survey caps at 2. The larger survey looks more diverse even if its evenness is lower. Compare evenness, not raw entropy, when k differs.

Empty categories with a count of 0 contribute nothing, since 0 \times \log_2 0 is defined as 0. But whether you include known-but-unobserved categories changes k and therefore the evenness ceiling. Decide once and stay consistent.

Finally, small samples inflate apparent unevenness. With only 5 items across 4 categories, one category will grab a large share by chance. Entropy from tiny counts is noisy; gather more before you compare.

Related tools

Once you have summarized category structure, these calculators cover the neighboring questions. Use the Descriptive Statistics Calculator for ordered or continuous measurements. The Outlier Detector flags extreme numeric values that a diversity index would ignore. The Percentile Calculator ranks numeric data. For relationships between two numeric variables, try the Covariance Calculator. And the Benford's Law Checker tests first-digit frequencies, another way to probe how counts are distributed.

Frequently asked questions

Why is my entropy in bits and not the same as someone else's?

Different logarithm bases give different units. Base 2 gives bits, natural log gives nats, base 10 gives bans. To convert, divide: entropy in nats equals bits times 0.6931. The relative pattern across datasets does not change.

What is the highest entropy I can get?

For k categories it is \log_2 k bits, reached only when every category has the same share. With 4 categories that is 2 bits; with 8 it is 3.

How do entropy and the Simpson index differ in practice?

Entropy weighs rare categories more heavily through the \log term, so it responds strongly when you split off a small new group. The Simpson index leans on the common categories through the p_i^2 term, so a rare category barely moves it. Report both when rare groups matter.

Can entropy be negative?

No. Every proportion is between 0 and 1, so each \log_2 p_i is negative or zero, and the leading minus sign makes the sum zero or positive. The lowest value is exactly 0, when one category holds everything.

Do I need the counts to sum to 100?

No. Any positive total works, because the first step converts counts to proportions. The counts 3, 6, 9, 12 give the same entropy as 10, 20, 30, 40, since both reduce to shares of 0.1, 0.2, 0.3, 0.4.

Try it yourself in the tool

Related tools

Descriptive Statistics Calculator

Mean, median, quartiles, variance, skewness, outliers and more.

Kaplan–Meier Survival Calculator

Survival curve, life table and median survival from time-to-event data with censoring.

Outlier Detector

Grubbs' test, Tukey fences and z-scores on your data.

Time Series Lab: Decomposition, ETS & ARIMA Forecast

Paste a CSV: seasonal decomposition, ETS and ARIMA fits, forecast intervals and rolling-origin cross-validation.

Autocorrelation Calculator

ACF by lag with white-noise bands and the Ljung–Box test.

Covariance Calculator

Sample and population covariance with Pearson correlation.