Brennan-Prediger Free-Marginal Kappa and PABAK Formula

Reproduce why 92% rater agreement collapses to 0.29 Cohen’s kappa on rare defects

When screening rare clinical findings or low-defect manufacturing lots, two raters can agree on 92% of cases yet receive a dismal Cohen’s kappa below 0.30! On N = 100 inspections across q_cat = 2 classes (B6 = 2) with n_11 = 90 Normal (B2), n_12 = 4 (C2), n_21 = 4 (B3), and n_22 = 2 Defect (C3), observed agreement is P_o = 92 / 100 = 0.9200, yet skewed Normal prevalence (94%) drives Cohen’s chance agreement to 0.94^2 + 0.06^2 = 0.8872, crushing kappa to (0.92 - 0.8872) / 0.1128 = 0.2908!

When raters are NOT forced to assign fixed quotas per category, Brennan and Prediger (1981) benchmark P_o against uniform random guessing 1 / q_cat: kappa_n = (P_o - 1 / q_cat) / (1 - 1 / q_cat). For q_cat = 2, kappa_n equals Byrt’s Prevalence-Adjusted Bias-Adjusted Kappa PABAK = 2 * P_o - 1 = 0.8400, reported alongside Prevalence Index PI = |n_11 - n_22| / N = 0.8800.

Brennan-Prediger Free-Marginal Kappa (Any Category Count q >= 2) Formula

Accept observed agreement P_o and nominal category count q_cat >= 2 so the exact same formula works for 2x2 PABAK (q = 2) and multi-class rubrics (q = 3, 4, 5).

=LET(n_tot, SUM(B2:C3), q_cat, B5, IF(OR(n_tot <= 0, q_cat < 2), "Invalid Rubric Input", LET(p_obs, (B2 + C3) / n_tot, p_uni, 1 / q_cat, ROUND((p_obs - p_uni) / (1 - p_uni), 4))))

Multi-Class Rubric Generalization Check (q = 4)

Verify multi-category scaling: if two graders achieve the same 92% exact agreement across a 4-category rubric (set B5 = 4, so uniform guessing is 1/4 = 0.25), Brennan-Prediger kappa_n rises from 0.8400 to (0.92 - 0.25) / 0.75 = 0.8933 because matching by pure chance across 4 classes is harder.

=ROUND(ABS(B2 - C3) / SUM(B2:C3), 4)

Further reading

• Arte Sheets Lesson — Scott’s Pi Pooled-Marginal Agreement vs Cohen’s Kappa
• Arte Sheets Lesson — Fleiss’ Kappa Multi-Rater Categorical Agreement Formula