A/B test significance calculator

A standard two-proportion z-test. The p-value uses a pooled standard error (it assumes, for the sake of the test, that there is truly no difference between A and B, then asks how surprising your actual result would be under that assumption). The confidence interval on the difference uses an unpooled standard error, because it is describing the observed gap rather than testing a hypothesis about it. The normal distribution itself is computed properly via the Abramowitz-Stegun approximation to the error function, not read off a lookup table -- so the result should match any other correctly-built calculator, including Evan Miller's, to within rounding.

Two-tailed is the honest default and this tool starts there. A two-tailed test asks "is B different from A, in either direction?" -- which is the true state of your knowledge before a test runs, because a change can make things worse as easily as better. A one-tailed test only asks "is B better?", which makes it slightly easier to call a result significant. That is exactly why it is easy to misuse: switching to one-tailed after you have already seen the data, because it happens to turn a borderline result significant, is a form of p-hacking.

A higher observed rate is not the same as a real, repeatable difference. With small numbers of visitors or conversions, a run of ordinary luck can easily produce a gap this size even when the two versions perform identically in the long run. The p-value is an estimate of how often that would happen by chance alone -- if it is not small enough, the honest conclusion is "we do not yet know", not "B lost".

Because it tells you what you needed before you started, which is the only time it can actually help. If your test never reached that sample size, a significant-looking p-value part-way through is much more likely to be noise, and a non-significant one does not mean B failed -- it may mean the test never had a real chance to detect the effect you were hoping for (the test was "underpowered"). Most A/B tests run on low-traffic sites or pages fall into exactly this trap: the traffic simply is not there to answer the question being asked of it.

No. Statistical significance only tells you the difference is probably real, not that it is large enough to matter commercially, not that it will hold up next month, and not that it is worth the engineering cost of keeping two versions running. A tiny but "significant" uplift on a low-value page may not be worth acting on; a promising but not-yet-significant result on a high-value page may be worth extending the test rather than abandoning it.

Peeking is checking a test's significance repeatedly while it is still running and stopping as soon as it first looks significant. Each look is a fresh chance for random noise to cross the threshold, so a test checked daily and stopped on the first green result has a real false-positive rate far higher than the 5% a single, planned check implies -- often several times higher. The fix is deciding your sample size in advance (the second calculator on this page) and waiting until you reach it before treating the result as final.