The formula
How to calculate statistical significance
This is the standard two-proportion z-test used to decide whether an A/B test result is real or noise. It compares the observed difference against how much difference random variation alone would produce.
The normal tail probability is computed with the Zelen and Severo series, which is accurate to about seven decimal places — the same method statistics packages use when no exact routine is available.
The calculator asks for:
- Control conversion rate (%)
- Control visitors
- Variant conversion rate (%)
- Variant visitors
Results appear immediately — there is nothing to submit. Changing a field rewrites the link, so you can share the exact scenario you are looking at.
The calculation runs on exactly the numbers currently in the fields above, recomputed in full each time — there is no dependency on the order values are entered in, so adjusting one input to test a scenario and then changing it back leaves the result exactly where it started.
Why statistical significance matters
The formula behind statistical significance is standard and appears in the same form across textbooks and reference material; what a calculator adds is speed and the ability to see instantly how the result responds to a change in any one of the inputs, which is far slower to do by hand.
It is useful for checking a manual calculation before submitting or acting on it, and equally useful for building intuition about a formula by adjusting one input at a time and watching how the result moves in response — a much faster way to understand a relationship than working through several versions of the algebra by hand.
A formula like this one is rarely the last step in a piece of work — the figure it produces usually feeds into a further calculation, a comparison against a published value, or a write-up that needs to state both the result and how confident it is. Getting this step right the first time, rather than propagating a small arithmetic slip through several more steps, is the main practical reason to check a manual calculation against a tool like this one before building on top of it.
It is worth keeping a note of which inputs were used to produce a given result, particularly where the figure is going into a report or a further calculation — reproducing a result later, or explaining how it was reached, is far easier with the original inputs to hand than by trying to reverse-engineer them from the output alone.
Worked example
Here is the calculation with the starting values:
- Control conversion rate: 4.2 %
- Control visitors: 5,200
- Variant conversion rate: 5.1 %
- Variant visitors: 5,150
That gives:
- z-score: 2.1742
- Two-tailed p-value: 0.0297
- Relative lift: 21.4286 %
The figures above are the calculator's own default values, shown purely so the working is visible rather than hidden — the same steps apply exactly to your own numbers, entered in the fields at the top of this page.
Reading the result
A p-value below 0.05 is the conventional threshold for significance, corresponding to a z-score beyond ±1.96. It means a difference this large would occur by chance less than 5% of the time if the variants were identical.
Where this goes wrong. Checking the test repeatedly and stopping when it turns significant inflates the false positive rate dramatically. Fix the sample size in advance, or use a sequential testing method designed for continuous monitoring.
A result that is wrong by an exact factor of ten, a hundred or a similar round number is almost always a units error rather than a mistake in the formula itself — checking each input against the unit stated beside it is the fastest way to track it down.
That the observed difference is unlikely to have arisen from random variation alone. It says nothing about whether the difference is large enough to matter commercially.
Until the pre-calculated sample size is reached, and for at least one full week to cover day-of-week effects. Stopping early on a promising result is the most common way to reach a wrong conclusion.
The headline figure is z-score. With 4.2 % control conversion rate, 5,200 control visitors and 5.1 % variant conversion rate, that comes to 2.1742. Change any field and the figure moves with it.
Generally, no more than the least precise input justifies — a result reported to six decimal places from inputs measured to two significant figures is implying a precision the calculation does not actually have. The calculator shows full precision so you can round appropriately for your own use.
Yes — the equation shown in the formula section above is the standard form used in textbooks and reference material for this calculation, not a simplified or approximate version.
Yes, in the sense that it applies the correct standard formula and returns an accurate result for the inputs given — but check your own course or publication's requirements for how results should be rounded, presented and referenced, since those conventions vary and are not something a calculator can know on your behalf.