Statistical & Psychometrics
Gwet’s AC1 Inter-Rater Agreement Coefficient (Solving the Kappa Paradox)
Calculate Kilem Gwet’s (2008) paradox-resistant first-order agreement coefficient AC1 = (p_a - p_eγ) / (1 - p_eγ) vs. Cohen’s κ when trait prevalence is skewed near 0% or 100%.
Why 90% raw agreement collapses to Cohen’s κ = 0.44—and how Gwet’s AC1 restores truth!
In pathology screening, radiology QC, and content moderation where positive or negative cases dominate (85%–95% prevalence), Feinstein and Cicchetti (1990) exposed a devastating flaw in Cohen’s kappa: TWO raters agreeing on 90% of cases (a = 85, b = 5, c = 5, d = 5; N = 100) receive a mediocre Cohen’s κ = 0.4444 solely because skewed marginals inflate Cohen’s chance-agreement penalty to 0.8200!
Kilem L. Gwet (British Journal of Mathematical and Statistical Psychology, 2008) proved that chance agreement occurs ONLY on hard-to-classify subjects, deriving p_eγ = 2 * π_1 * (1 - π_1) where average positive propensity π_1 = (2*a + b + c) / (2*N)! On that exact [85, 5, 5, 5] table, π_1 = 0.90 gives p_eγ = 0.1800 and Gwet’s AC1 = (0.90 - 0.18) / (1 - 0.18) = 0.8780—accurately reflecting 90% agreement while coinciding with κ = 0.8000 on balanced [45, 5, 5, 45] cohorts!
| 2×2 Rater Table [a, b, c, d] (Raw p_a = 90%) | Cohen’s κ | Gwet’s AC1 | Prevalence Sensitivity |
|---|---|---|---|
| [85, 5, 5, 5] — 90% Skewed Prevalence | 0.4444 (Kappa Paradox!) | 0.8780 | AC1 resists marginal distortion |
| [45, 5, 5, 45] — 50% Balanced Prevalence | 0.8000 | 0.8000 | AC1 & κ agree identically at π = 0.5 |
| Empty Table (N <= 0) | Undefined | "Empty Contingency Table" | Sample-Size Guard |
Gwet’s AC1 Paradox-Resistant Agreement Coefficient (2×2 Binary)
Computes (p_a - p_eγ) / (1 - p_eγ) with p_a = (a + d) / N and p_eγ = 2 * π_1 * (1 - π_1), π_1 = (2*a + b + c) / (2*N) across cell counts B2:B5.
=IF(OR(MIN(B2:B5) < 0, SUM(B2:B5) <= 0), "Empty Contingency Table", ROUND((((B2 + B5) / SUM(B2:B5)) - (2 * (((2 * B2 + B3 + B4) / (2 * SUM(B2:B5)))) * (1 - ((2 * B2 + B3 + B4) / (2 * SUM(B2:B5)))))) / (1 - (2 * (((2 * B2 + B3 + B4) / (2 * SUM(B2:B5)))) * (1 - ((2 * B2 + B3 + B4) / (2 * SUM(B2:B5)))))), 4))Standard Unadjusted Cohen’s Kappa κ (For Paradox Audit Comparison)
Evaluates traditional Cohen’s κ on the same 2×2 cells B2:B5 to expose prevalence-induced κ attenuation.
=IF(OR(MIN(B2:B5) < 0, SUM(B2:B5) <= 0), "Empty Contingency Table", ROUND((((B2 + B5) / SUM(B2:B5)) - ((((B2 + B3) * (B2 + B4)) + ((B4 + B5) * (B3 + B5))) / (SUM(B2:B5) ^ 2))) / (1 - ((((B2 + B3) * (B2 + B4)) + ((B4 + B5) * (B3 + B5))) / (SUM(B2:B5) ^ 2))), 4))Try the example
Use the original values or change them to test your understanding.