Statistical Analysis
Intraclass Correlation Shrout-Fleiss ICC(2,1) Agreement vs ICC(3,1) Consistency
Calculate single-measure and k-rater Intraclass Correlation Coefficients ICC(2,1), ICC(3,1), and ICC(2,k) from two-way ANOVA Mean Squares (MS_R, MS_C, MS_E) to isolate systematic rater bias.
Why Pearson r and ICC(3,1) miss harsh-vs-lenient rater offset that ICC(2,1) catches
When k = 3 clinicians or QA inspectors score n = 10 subjects, two-way ANOVA partitions variance into Between-Subjects Mean Square MS_R, Between-Raters Mean Square MS_C, and Residual Error MS_E. Patrick Shrout and Joseph Fleiss (1979) distinguished two fundamentally different reliability questions: Consistency ICC(3,1) = (MS_R - MS_E) / (MS_R + (k - 1) * MS_E) ignores MS_C completely, asking only whether raters rank subjects additively in parallel.
By contrast, Absolute Agreement ICC(2,1) = (MS_R - MS_E) / (MS_R + (k - 1) * MS_E + (k / n) * (MS_C - MS_E)) adds systematic rater offset (k / n) * (MS_C - MS_E) into the denominator! On MS_R = 25, MS_C = 13, MS_E = 1 (n = 10, k = 3), consistency looks high at 0.8889 while true single-rater interchangeability drops to 0.7843—until rater calibration lowers MS_C to 1, where both converge to 0.8889.
| Shrout-Fleiss Model | ANOVA Mean-Square Formula | Score (n=10, k=3, MSR=25, MSC=13, MSE=1) |
|---|---|---|
| ICC(2,1) Single-Rater Agreement | (MSR - MSE) / (MSR + (k-1)*MSE + (k/n)*(MSC - MSE)) | 0.7843 (Penalizes rater bias MSC=13) |
| ICC(3,1) Single-Rater Consistency | (MSR - MSE) / (MSR + (k-1)*MSE) | 0.8889 (Rank/additive consistency) |
| ICC(2,k) Mean-of-k Agreement | (MSR - MSE) / (MSR + (MSC - MSE)/n) | 0.9160 (Averaging all 3 raters) |
Two-Way Random Single-Measure Absolute Agreement ICC(2,1) Formula
Include (k / n) * (MS_C - MS_E) in the denominator so harsh/lenient judge offsets appropriately reduce single-rater interchangeability.
=ROUND((B4 - B6) / (B4 + (B3 - 1) * B6 + (B3 / B2) * (B5 - B6)), 4)Consistency ICC(3,1) & Average-Rater ICC(2,k) Formulas
Compare ICC(3,1) against ICC(2,1) to quantify rater-bias loss, and evaluate ICC(2,k) when operational workflow averages all k judges.
=ROUND((B4 - B6) / (B4 + (B5 - B6) / B2), 4)Try the example
Use the original values or change them to test your understanding.