Statistical & Math
Fleiss’ Kappa (kappa) Multi-Rater Categorical Agreement Formula
Calculate chance-corrected multi-annotator reliability Fleiss’ Kappa = (P_bar - P_e) / (1 - P_e) across N items rated by n >= 3 raters into k categories for ML data-labeling QA and clinical radiology panels.
Generalize chance-corrected inter-rater reliability beyond Cohen’s 2-rater limit!
Cohen’s Kappa only compares two fixed raters—so how do ML data-annotation teams, medical board examiners, and radiology panels audit reliability when n = 5 annotators classify each item into k = 3 categories (Positive, Neutral, Negative)? Joseph Fleiss (1971) generalized Scott’s pi to any number of raters n >= 2: for each item i, pairwise agreement among n*(n - 1) rater pairs is P_i = (SUM_j(n_ij^2) - n) / (n * (n - 1)), averaged across all N items to give observed agreement P_bar, while expected chance agreement is P_e = SUM_j(p_j^2) from marginal category proportions p_j!
Look at a 4-item ML labeling audit rated by n = 5 annotators across 3 classes: item vote counts [(5,0,0), (4,1,0), (3,2,0), (0,1,4)] yield row agreements P_i = [1.0, 0.6, 0.4, 0.6] -> mean observed agreement P_bar = 0.6500, while marginal class shares [12/20 = 0.60, 4/20 = 0.20, 4/20 = 0.20] yield chance agreement P_e = 0.60^2 + 0.20^2 + 0.20^2 = 0.4400—giving Fleiss’ Kappa = (0.65 - 0.44) / (1 - 0.44) = 0.3750 ("Fair Agreement", flagging ambiguous class guidelines before training the model)!
| Audit Item (n = 5 Raters) | Class Votes (Cat A, Cat B, Cat C) | Item Pairwise Agreement Pi | Summary Reliability |
|---|---|---|---|
| Item 1 (Unanimous) | 5, 0, 0 | 1.0000 | Marginal Shares: 0.60, 0.20, 0.20 |
| Item 2 & Item 4 (4-to-1 Split) | 4, 1, 0 and 0, 1, 4 | 0.6000 each | Chance Agreement Pe = 0.4400 |
| Item 3 (3-to-2 Split) | 3, 2, 0 | 0.4000 | P_bar = 0.6500 -> Fleiss kappa = 0.3750 |
Mean Observed Multi-Rater Pairwise Agreement P_bar Across 4 Items (n = 5)
Uses SUMPRODUCT(B2:D5, B2:D5) to sum squared votes across the 4x3 vote matrix: since each row sums to n = 5, average P_i equals (SUMPRODUCT(B2:D5, B2:D5) - N * n) / (N * n * (n - 1)).
=ROUND((SUMPRODUCT(B2:D5, B2:D5) - 4 * 5) / (4 * 5 * (5 - 1)), 4)Chance-Corrected Fleiss’ Kappa = (P_bar - P_e) / (1 - P_e)
Computes P_e = (SUM(B2:B5)/20)^2 + (SUM(C2:C5)/20)^2 + (SUM(D2:D5)/20)^2 and normalizes (P_bar - P_e) / (1 - P_e).
=ROUND((((SUMPRODUCT(B2:D5, B2:D5) - 20) / 80) - ((SUM(B2:B5) / 20) ^ 2 + (SUM(C2:C5) / 20) ^ 2 + (SUM(D2:D5) / 20) ^ 2)) / (1 - ((SUM(B2:B5) / 20) ^ 2 + (SUM(C2:C5) / 20) ^ 2 + (SUM(D2:D5) / 20) ^ 2)), 4)Try the example
Use the original values or change them to test your understanding.