Statistical & Psychometrics
Spiegelhalter’s Z-Test for Probability Calibration (Exact Brier Decomposition)
Test whether ML or clinical risk probabilities p_i are calibrated without arbitrary Hosmer-Lemeshow decile binning via David Spiegelhalter’s (1986) exact Z-statistic!
Bin-free calibration testing—why biostatisticians prefer Spiegelhalter’s Z over Hosmer-Lemeshow deciles!
Hosmer-Lemeshow goodness-of-fit changes p-values capriciously depending on whether you group predicted probabilities into 8, 10, or 12 bins! Sir David J. Spiegelhalter (Statistics in Medicine, 1986) proved that under the null hypothesis of perfect calibration E[Y_i] = p_i, the Brier score B = MEAN((Y_i - p_i)^2) has exact expectation E_0[B] = MEAN(p_i * (1 - p_i)) and variance V_0[B] = SUM((1 - 2*p_i)^2 * p_i * (1 - p_i)) / N^2—yielding the bin-free asymptotic standard normal statistic: Z_Spieg = SUM((Y_i - p_i) * (1 - 2*p_i)) / SQRT(SUM((1 - 2*p_i)^2 * p_i * (1 - p_i))) (note (Y_i - p_i)^2 - p_i*(1 - p_i) = (Y_i - p_i)*(1 - 2*p_i) for binary Y_i in {0, 1}!)!
Audit 5 risk predictions p = [0.15, 0.25, 0.65, 0.80, 0.90] against well-calibrated outcomes Y = [0, 0, 1, 1, 1]: Z_Spieg = -1.0808 (|Z| < 1.96, well calibrated!). Invert outcomes to overconfident miscalibration Y = [1, 1, 1, 0, 0]: Z_Spieg surges to +4.1717 (|Z| >= 1.96, severe miscalibration p < 0.0001)!
| Binary Outcome Vector Y vs p = [0.15, 0.25, 0.65, 0.80, 0.90] | Spiegelhalter Z | Calibration Verdict (α = 0.05) |
|---|---|---|
| Y = [0, 0, 1, 1, 1] (Aligned Outcomes) | -1.0808 (|Z| < 1.96) | Well-Calibrated Probabilities |
| Y = [1, 1, 1, 0, 0] (Overconfident Miscalibration) | +4.1717 (|Z| >= 1.96) | Significant Miscalibration (Reject H0) |
| All Predictions Exactly p = 0.50 | "Degenerate 0.50 Predictions" | Denominator (1 - 2p) = 0 Guard |
Spiegelhalter Exact Brier Calibration Z-Statistic
Divides SUM(D2:D6) = Σ (Y_i - p_i)*(1 - 2*p_i) by SQRT(SUM(E2:E6)) = SQRT(Σ (1 - 2*p_i)^2 * p_i * (1 - p_i)).
=IF(SUM(E2:E6) <= 0, "Degenerate 0.50 Predictions", ROUND(SUM(D2:D6) / SQRT(SUM(E2:E6)), 4))Two-Sided α = 0.05 Probability Calibration Decision
Flags statistically significant miscalibration whenever ABS(Z_Spieg) >= 1.95996.
=IF(SUM(E2:E6) <= 0, "Degenerate 0.50 Predictions", IF(ABS(SUM(D2:D6) / SQRT(SUM(E2:E6))) >= 1.95996, "Significant Miscalibration (|Z| >= 1.96)", "Well-Calibrated Probabilities (|Z| < 1.96)"))Try the example
Use the original values or change them to test your understanding.