Forecasting & Demand Planning
Diebold-Mariano and HLN Forecast Accuracy Test Formula
Test whether two competing models have equal out-of-sample forecast accuracy using Diebold-Mariano loss differentials d_t and Harvey-Leybourne-Newbold’s finite-sample correction.
Stop declaring a forecast winner from raw RMSE differences without testing d_t variance!
Comparing Model 1 RMSE against Model 2 RMSE on a single holdout window tells you nothing about whether the gap is statistically distinguishable from luck! Diebold and Mariano (1995) form period-by-period loss differentials d_t = L(e_{1t}) - L(e_{2t}) (squared error e_{1t}^2 - e_{2t}^2 or absolute error |e_{1t}| - |e_{2t}|) and test H0: E[d_t] = 0 via DM = d_bar / SQRT(Omega_d / T), where positive DM means Model 2 has lower loss (beats Model 1).
Because raw DM over-rejects in short holdout samples and multi-step horizons h >= 1, Harvey, Leybourne, and Newbold (1997) multiply DM by exact finite-sample factor k_HLN = SQRT((T + 1 - 2*h + h*(h - 1)/T) / T) and compare against Student-t(df = T - 1)! Verify on T = 8 1-step (h = 1 in G2) holdout periods with squared-error differentials d_t = {4, 6, 3, 5, 7, 2, 5, 8} in D2:D9: mean loss gap d_bar = 5.0000, population variance VARP(d) = 3.5000 -> raw DM = 5 / SQRT(3.50 / 8) = 7.5593, and at h = 1, k_HLN = SQRT((8 + 1 - 2 + 0) / 8) = SQRT(7 / 8) = 0.9354 -> HLN-adjusted statistic = 7.0711 (matching standard 1-sample t = 5 / (STDEV(d)/SQRT(8)) = 5 * SQRT(2) = 7.0711 identically at h = 1!).
| Forecast Comparison Metric (T=8, h=1) | Formula Expression | Sample Output |
|---|---|---|
| Mean Loss Differential d_bar | AVERAGE(e_1t^2 - e_2t^2) | +5.0000 (Model 2 beats Model 1) |
| Asymptotic Diebold-Mariano DM | d_bar / SQRT(VARP(d_t) / T) | +7.5593 (Overstated in small T) |
| HLN Finite-Sample Factor k_HLN | SQRT((T + 1 - 2*h + h*(h-1)/T) / T) | 0.9354 (SQRT(7/8) at h = 1) |
| HLN-Modified Statistic DM_HLN | k_HLN * DM | +7.0711 (5 * SQRT(2)) |
Harvey-Leybourne-Newbold Finite-Sample Scaling Factor k_HLN Formula
Verify horizon bounds 1 <= h < T so T + 1 - 2*h + h*(h - 1)/T remains strictly positive.
=LET(t_len, ROWS(D2:D9), h_hor, G2, num_k, t_len + 1 - 2 * h_hor + (h_hor * (h_hor - 1)) / t_len, IF(OR(h_hor < 1, num_k <= 0), "Horizon Too Long for Sample", ROUND(SQRT(num_k / t_len), 4)))HLN-Adjusted Diebold-Mariano Statistic DM_HLN Formula
Evaluate raw DM (for 1-step errors using VARP(D2:D9)) multiplied by k_HLN. Notice how increasing horizon h in G2 from 1 to 3 shrinks k_HLN from 0.9354 down to SQRT((8 + 1 - 6 + 6/8)/8) = SQRT(3.75/8) = 0.6847, appropriately penalizing multi-step forecast overlap down to DM_HLN = 5.1755!
=LET(t_len, ROWS(D2:D9), h_hor, G2, v_pop, VARP(D2:D9), num_k, t_len + 1 - 2 * h_hor + (h_hor * (h_hor - 1)) / t_len, IF(OR(h_hor < 1, num_k <= 0, v_pop <= 0), "Invalid DM Series", ROUND((AVERAGE(D2:D9) / SQRT(v_pop / t_len)) * SQRT(num_k / t_len), 4)))Try the example
Use the original values or change them to test your understanding.