Psychometrics & Inter-Rater Reliability
Scott’s Pi Pooled-Marginal Agreement Formula vs Cohen’s Kappa Bias
Expose systematic rater threshold bias by comparing Scott’s (1955) pooled-marginal pi against Cohen’s (1960) separate-marginal kappa on 2x2 annotation tables.
Spot the rater-bias paradox where Cohen’s kappa rewards mismatched thresholds
When two annotators classify N = 100 tickets as Escalate vs Routine, Cohen’s kappa computes expected chance agreement from the product of their SEPARATE marginal rates P_e,Cohen = p_A1*p_B1 + p_A2*p_B2, whereas Scott’s (1955) pi pools their marginals P_e,Scott = ((p_A1 + p_B1)/2)^2 + ((p_A2 + p_B2)/2)^2. By algebra, P_e,Scott - P_e,Cohen = 0.5 * (p_A1 - p_B1)^2 >= 0 ALWAYS — meaning Cohen’s kappa paradoxically RISES when one rater is lenient and the other is strict!
Verify by hand on N = 100 items: n_11 = 30 (B2), n_12 = 30 (C2, Rater A = 60% Escalate), n_21 = 0 (B3), n_22 = 40 (C3, Rater B = 30% Escalate). Observed agreement is P_o = 70 / 100 = 0.7000. Cohen’s chance is 0.60*0.30 + 0.40*0.70 = 0.4600, giving kappa = 0.24 / 0.54 = 0.4444. Scott’s pooled rates (0.45 and 0.55) give P_e,Scott = 0.45^2 + 0.55^2 = 0.5050 (+0.0450 higher!), penalizing the 30-point threshold bias down to pi = 0.1950 / 0.4950 = 0.3939 (13 / 33).
| Agreement Metric (N = 100) | Formula Definition | Sample Output |
|---|---|---|
| Observed Agreement P_o | (n_11 + n_22) / N | 0.7000 |
| Byrt Rater Bias Index BI | ABS(n_12 - n_21) / N | 0.3000 (30-point threshold shift) |
| Cohen’s Separate-Marginal Kappa | (P_o - (p_A1*p_B1 + p_A2*p_B2)) / (1 - P_e,C) | 0.4444 (Excuses bias) |
| Scott’s Pooled-Marginal Pi | (P_o - (p_bar1^2 + p_bar2^2)) / (1 - P_e,S) | 0.3939 (Penalizes bias) |
Scott’s Pooled-Marginal Pi Formula
Pool row and column totals into p_bar1 = (R_1 + C_1) / (2 * N) and p_bar2 = (R_2 + C_2) / (2 * N) and guard unanimous single-category tables where P_e,Scott = 1.
=LET(n_tot, SUM(B2:C3), IF(n_tot <= 0, "Empty Table", LET(p_obs, (B2 + C3) / n_tot, pb_1, ((B2 + C2) + (B2 + B3)) / (2 * n_tot), pb_2, 1 - pb_1, pe_sc, pb_1^2 + pb_2^2, IF(pe_sc >= 0.999999, "Single Category", ROUND((p_obs - pe_sc) / (1 - pe_sc), 4)))))Cohen’s Kappa Comparison & Zero-Bias Equality Check
Balance the off-diagonal disagreements by setting C2 = 15 and B3 = 15 (keeping P_o = 0.7000 and pooled Yes rate = 0.45): rater bias BI drops to 0 and BOTH Cohen’s kappa and Scott’s pi converge identically to 0.3939!
=LET(n_tot, SUM(B2:C3), p_obs, (B2 + C3) / n_tot, pa_1, (B2 + C2) / n_tot, pb_1, (B2 + B3) / n_tot, pe_co, pa_1 * pb_1 + (1 - pa_1) * (1 - pb_1), ROUND((p_obs - pe_co) / (1 - pe_co), 4))Try the example
Use the original values or change them to test your understanding.