Statistical & Math
Expected Calibration Error (ECE) & Expected-to-Observed (E/O) Risk Ratio
Audit clinical risk scores and ML classifiers across probability bins using Expected Calibration Error ECE = SUM(n_b * |p_pred - p_obs|) / N and overall Expected-to-Observed event ratio E/O = SUM(n_b * p_pred) / SUM(n_b * p_obs).
High AUC discrimination does NOT guarantee calibrated probabilities!
A credit-default or ICU mortality classifier can achieve a 0.88 ROC-AUC purely by ranking risky patients above healthy ones—even if its predicted probabilities are systematically 2x too high! Biostatisticians (following David Hosmer and Stanley Lemeshow’s binned goodness-of-fit framework) and machine-learning engineers audit probability calibration across B risk bins using two complementary metrics: Expected-to-Observed Ratio E/O = Expected_Events / Observed_Events (ideal calibration-in-the-large = 1.0000; E/O > 1 over-predicts risk, E/O < 1 under-predicts risk) and Expected Calibration Error ECE = SUM_b((n_b / N) * |p_pred,b - p_obs,b|)!
Across N = 400 validation patients partitioned into 4 equal bins of n_b = 100 with mean predicted probabilities [0.08, 0.25, 0.60, 0.85] vs. observed event rates [0.10, 0.20, 0.52, 0.88]: expected events = 178 vs. observed events = 170 yields E/O = 1.0471 (4.71% mild overall over-prediction), while bin-weighted absolute gap ECE = 0.0450 (4.50 percentage points mean calibration error)!
| Calibration Bin (n_b = 100 Patients Each) | Mean Predicted p_pred | Observed Event Rate p_obs | Absolute Calibration Gap |pred - obs| |
|---|---|---|---|
| Bin 1 (Low Risk) & Bin 2 (Low-Med) | 0.08 / 0.25 | 0.10 / 0.20 | 0.02 / 0.05 |
| Bin 3 (Med-High) & Bin 4 (High Risk) | 0.60 / 0.85 | 0.52 / 0.88 | 0.08 / 0.03 |
| Overall Cohort Summary (N = 400) | Expected = 178 | Observed = 170 (E/O = 1.0471) | Weighted ECE = 0.0450 (4.50%) |
Sample-Weighted Expected Calibration Error (ECE) Across 4 Probability Bins
Weights each bin’s absolute probability discrepancy |C_b - D_b| by its bin sample count B_b divided by total N = SUM(B2:B5).
=IF(SUM(B2:B5) <= 0, "Empty Validation Cohort", ROUND((B2 * ABS(C2 - D2) + B3 * ABS(C3 - D3) + B4 * ABS(C4 - D4) + B5 * ABS(C5 - D5)) / SUM(B2:B5), 4))Calibration-in-the-Large Expected-to-Observed Ratio (E / O)
Divides total expected events SUMPRODUCT(B2:B5, C2:C5) by total observed events SUMPRODUCT(B2:B5, D2:D5).
=IF(SUMPRODUCT(B2:B5, D2:D5) <= 0, "Zero Observed Events", ROUND(SUMPRODUCT(B2:B5, C2:C5) / SUMPRODUCT(B2:B5, D2:D5), 4))Try the example
Use the original values or change them to test your understanding.