Cohen's Kappa, Explained
After reading this you can compute Cohen's kappa from a two-rater agreement table by hand, judge whether the agreement is better than chance, and avoid the traps that make kappa look worse or better than it should.
What Cohen's kappa measures
Cohen's kappa (written \kappa) tells you how much two raters agree when both sort the same items into the same set of categories. The trick is that it removes the agreement you would expect from random guessing.
Suppose two radiologists each read 50 scans and label them "normal" or "abnormal". They agree on 35 of them. Raw agreement is 35/50 = 0.70. That sounds decent, but if both doctors call almost everything "normal", they would agree most of the time by luck alone. Kappa answers the sharper question: of the agreement that was actually available to earn beyond chance, how much did they earn?
Kappa ranges from \kappa = 1 (perfect agreement) down through \kappa = 0 (agreement no better than chance) and can go negative when raters agree less than chance would predict.
When to use it, and when not to
Use kappa when you have exactly two raters, each item is placed in one of a fixed set of unordered categories, and the same items are rated by both people. The classic cases are two clinicians reading the same records, or two coders tagging the same text passages.
Do not use plain kappa in these situations:
- More than two raters. Use Fleiss' kappa instead.
- Ordered categories where a near miss should count more than a far miss (mild, moderate, severe). Use weighted kappa.
- A rating scale you want to check for internal consistency across many items. That is a different question, answered by the Cronbach's Alpha Calculator.
Kappa is not a substitute for observed agreement. Always report both. Two raters can show 90% raw agreement and a kappa near 0 if one category dominates. The kappa alone would hide how often they actually matched.
The formula and the intuition
Kappa compares observed agreement with expected agreement:
Here p_o is the observed proportion of items the raters agreed on, and p_e is the proportion expected by chance if each rater kept their own category rates but assigned labels independently.
The numerator p_o - p_e is the agreement beyond chance. The denominator 1 - p_e is the most agreement that was left to earn once chance is accounted for. Their ratio is the fraction of that available margin the raters captured.
To get p_e, take the diagonal category totals. For each category, multiply the fraction of items rater 1 put there by the fraction rater 2 put there, then add across categories:
where r_i is the row total for category i (rater 1's count), c_i is the column total for category i (rater 2's count), and N is the total number of items.
Worked example with the demo data
Load the demo matrix on the calculator:
| R2: Yes | R2: No | Row total | |
|---|---|---|---|
| R1: Yes | 20 | 5 | 25 |
| R1: No | 10 | 15 | 25 |
| Column total | 30 | 20 | 50 |
- Total items:
20 + 5 + 10 + 15 = 50. - Observed agreement is the diagonal over the total:
(20 + 15) / 50 = 35/50 = 0.70. So p_o = 0.70. - Chance term for "Yes": rater 1 said Yes
25/50 = 0.50of the time, rater 2 said Yes30/50 = 0.60. Product is0.50 \times 0.60 = 0.30. - Chance term for "No": rater 1
25/50 = 0.50, rater 220/50 = 0.40. Product is0.50 \times 0.40 = 0.20. - Expected agreement: p_e = 0.30 + 0.20 = 0.50.
- Kappa:
(0.70 - 0.50) / (1 - 0.50) = 0.20 / 0.50 = 0.40.
So \kappa = 0.40. The raw agreement of 0.70 looked healthy, but half of that was expected by chance. On the Landis and Koch scale, 0.40 sits at the top of "fair".
See how the base rate moves kappa
The same raw agreement can produce very different kappa values depending on how balanced the categories are. When one category dominates, chance agreement climbs and kappa falls. Move the slider below to feel this effect.
Reading and interpreting the result
Kappa has no natural units, so people lean on descriptive bands. The Landis and Koch labels used on the calculator are:
- < 0.21
- Slight agreement.
- 0.21 to 0.40
- Fair.
- 0.41 to 0.60
- Moderate.
- 0.61 to 0.80
- Substantial.
- > 0.80
- Almost perfect.
These cutoffs are conventions, not laws. A kappa of 0.55 in one field may be routine and in another may be unacceptable. Report the number, the band, and the raw agreement together so a reader can judge for themselves.
A negative kappa means the raters disagreed more than chance predicts. In the widget, a prevalence of 0.90 with raw agreement of 0.80 yields a kappa near -0.11. That is a warning that the two raters are systematically pulling in different directions, or that the category mix is too lopsided for kappa to behave sensibly.
Common mistakes
Three problems account for most bad kappa reports.
Feed the calculator a square matrix of raw counts, not proportions and not row percentages. The chance term uses the marginal totals, so scaling any row changes the answer. If your data are percentages, multiply back up to counts first.
The kappa paradox. When almost every item lands in one category, kappa can be low even with 95% raw agreement. This is not a flaw in your raters; it is the definition working as designed. Report prevalence alongside kappa so the paradox is visible.
Wrong tool for ordered categories. If your categories are mild, moderate, severe, plain kappa treats a mild-versus-severe disagreement the same as mild-versus-moderate. Use weighted kappa so near misses cost less.
Confusing the diagonal. Cell (i, j) counts items rater 1 called category i and rater 2 called category j. The diagonal cells are the agreements. If you transpose the matrix by mistake, the off-diagonal counts swap but kappa itself stays the same, since it depends only on the diagonal and the marginals.
Related tools
Kappa checks whether two raters agree. If your question is instead whether the items in a questionnaire measure one thing consistently, use the Cronbach's Alpha Calculator. Both live under reliability and agreement, but they answer different questions: kappa is about rater-to-rater matching, alpha is about item-to-item consistency within a scale.
Frequently asked questions
What is a good Cohen's kappa value?
By the Landis and Koch bands, anything above 0.60 is substantial and above 0.80 is almost perfect. In the worked example a kappa of 0.40 counts as fair. What passes as "good" depends on the stakes of the decision being coded.
Why is my kappa low when the raters agree most of the time?
This is the kappa paradox. If both raters assign most items to one category, chance agreement is already high, so there is little room to score above it. With 90% of items in one category and 80% raw agreement, kappa can be negative.
Can Cohen's kappa be negative?
Yes. A negative value means observed agreement fell below the chance level p_e. The theoretical minimum depends on the marginals but is bounded below by -1.
Does kappa need equal numbers in each category?
No. Kappa uses whatever marginal totals your data show. But very unequal marginals inflate p_e and push kappa down, so always report the category counts.
What if I have three or more raters?
Cohen's kappa is defined for two raters only. For three or more, use Fleiss' kappa, which extends the same chance-corrected idea to any number of raters.
Related tools
Cronbach's Alpha Calculator
Internal consistency of a questionnaire or scale.
Binomial Distribution Calculator
Probabilities of k successes in n independent trials.
Chi-Square Test of Independence
Test association between two categorical variables from a contingency table.
Combinations & Permutations Calculator
nCr and nPr, with and without repetition.
Confidence Interval Calculator
CIs for a mean (z or t) and for a proportion (Wald & Wilson).
Correlation & Linear Regression Calculator
Pearson r, Spearman ρ, R² and the least-squares line.