A/B test conversion lift and statistical significance (two-proportion Z-score)

Relative lift vs statistical noise

If Control A converts 160 of 2,000 visitors (8.0%) and Variant B converts 200 of 2,000 visitors (10.0%), relative lift is (10% - 8%) / 8% = +25%.

Under the null hypothesis of equal conversion probabilities, compute the pooled rate, then the standard error and Z-score shown below. An absolute unrounded Z-score above approximately 1.96 rejects equality at the two-sided 5% significance level under the test assumptions. This does not prove that the observed lift is real or predict its future size.

Assume independent randomized groups, one binary outcome per unit, and adequate expected successes and failures in both groups (commonly at least 10 each). Validate positive integer sample sizes and conversions between zero and sample size. All-zero or all-converted data have zero standard error here; zero baseline conversion makes relative lift undefined. Repeated peeking or many comparisons needs an adjusted testing plan.

Two-proportion Z-score and lift formulas with LET

With pPool = 360 / 4,000 = 0.09, SE = SQRT(0.09 * 0.91 * (1/2000 + 1/2000)) = 0.00905, giving Z = 0.02 / 0.00905 = 2.21 (>= 1.96 -> significant).

=LET(pA, B2/A2, pB, D2/C2, p, (B2+D2)/(A2+C2), se, SQRT(p*(1-p)*(1/A2 + 1/C2)), ROUND((pB - pA) / se, 2))

Relative conversion lift formula

(D2/C2 - B2/A2) / (B2/A2) returns 0.25 (+25% relative improvement over Control).

=ROUND(((D2 / C2) - (B2 / A2)) / (B2 / A2), 4)

Further reading

• Microsoft Support — SQRT function
• Penn State — Inference for two samples