Statistical & Clinical Analysis
Bradley-Terry Paired-Comparison Formula: LLM Arena, Sports & Sensory Win Probability
Compute pairwise head-to-head win probabilities P(i beats j) = EXP(beta_i) / (EXP(beta_i) + EXP(beta_j)) and worth ratios EXP(beta_i - beta_j) using Ralph Bradley & Milton Terry’s 1952 model powering Chatbot Arena LLM leaderboards and NCAA/sports rankings.
From 1952 sensory taste panels to LMSYS Chatbot Arena & Elo ratings: the Bradley-Terry model!
Ralph Allan Bradley and Milton E. Terry (1952, Biometrika — originally developed by Ernst Zermelo in 1929 for chess tournaments, and today the exact maximum-likelihood engine behind LMSYS Chatbot Arena LLM human-preference leaderboards!) assign each competitor i a latent log-strength beta_i (or positive "worth" pi_i = EXP(beta_i)) so that the probability competitor i beats competitor j in a head-to-head comparison is P(i > j) = pi_i / (pi_i + pi_j) = EXP(beta_i) / (EXP(beta_i) + EXP(beta_j)) = 1 / (1 + EXP(-(beta_i - beta_j)))!
Notice that Elo ratings are simply a base-10 rescaling of Bradley-Terry log-strengths (Elo_i - Elo_j = 400 * LOG10(EXP(beta_i - beta_j)))! When Model A has log-strength beta_A = 1.80 and Model B has beta_B = 0.60 (gap +1.20 logits -> odds ratio EXP(1.20) = 3.3201 : 1), Model A wins 76.85% of blind pairwise prompts (reversing to 23.15% when beta_A = 0.60 and beta_B = 1.80)!
| Head-to-Head Log-Strengths (beta_A, beta_B) | Win Odds Ratio EXP(beta_A - beta_B) | Bradley-Terry Win Probability P(A > B) |
|---|---|---|
| Model A = 1.80, Model B = 0.60 (+1.20 gap) | 3.3201 : 1 | 0.7685 (76.85% Win Rate) |
| Reversed: Model A = 0.60, Model B = 1.80 | 0.3012 : 1 | 0.2315 (23.15% Win Rate; sums to 1.0000) |
| Equal Strength beta_A = beta_B = 1.20 | 1.0000 : 1 | 0.5000 (50.00% Coin Toss) |
Bradley-Terry Head-to-Head Win Probability P(A > B) = EXP(beta_A) / (EXP(beta_A) + EXP(beta_B))
Evaluates exact pairwise win probability from log-worth parameters beta_A (B2) and beta_B (B3).
=IF(OR(ABS(B2) > 20, ABS(B3) > 20), "Log-Strength Outside [-20, +20] Bounds", ROUND(EXP(B2) / (EXP(B2) + EXP(B3)), 4))Pairwise Worth / Win-Odds Ratio pi_A / pi_B = EXP(beta_A - beta_B)
Computes how many times more likely A is to defeat B than B is to defeat A.
=IF(OR(ABS(B2) > 20, ABS(B3) > 20), "Log-Strength Outside [-20, +20] Bounds", ROUND(EXP(B2 - B3), 4))Try the example
Use the original values or change them to test your understanding.