A/B Test Significance Calculator

A variant that converts at 6.8% against a control at 5.0% looks like a win, but with a thousand visitors in each group that 18-percentage-point gap has a real chance of being noise. A/B Test Significance Calculator runs the actual statistical test, a two-proportion z-test, so the answer is a p-value instead of a feeling about which bar is taller.

Use it once a test has run its full planned duration, not as a dashboard you refresh every morning looking for green. Enter visitors and conversions for both groups and the result updates immediately.

Running Your First Significance Test

1. Fill in Visitors and Conversions for Control (A) and Variant (B).
2. Read the p-value and the Significant / Not Significant badge next to it.
3. Check the note underneath: it tells you whether the result clears the 95% threshold, the 99% threshold, or neither.

How the Math Works: The Two-Proportion Z-Test

The pooled conversion rate across both groups sets a shared standard error, and the z-score is the gap between the two observed rates divided by that error. A z of 1.96 corresponds to a two-tailed p-value of 0.05, the standard 95% cutoff; a z of 2.576 corresponds to 0.01. Converting a z-score into a p-value needs the normal distribution’s CDF, which this computes with the Abramowitz-Stegun error-function approximation, checked during development against Python’s math.erf at five points and matching to 6 decimal places every time.

Accuracy and What This Doesn’t Account For

This is one test run once, and it assumes exactly that: checking the result daily and stopping the moment it turns green (a practice called peeking) inflates the real false-positive rate well past 5%, even though each individual check looks legitimate. It also doesn’t adjust for testing multiple variants at once or for a control and variant that ran in different weeks with different traffic mixes. Decide the sample size and duration before you start, run the test once, and read the result here a single time at the end.