A/B Testing Glossary

Every term the experiment results page uses, in plain English. None of these require a statistics background to act on.

Sample size

How many shoppers a result is based on. In NavonaAI experiments, sample size is counted in interventions shown per group — not visitors, not carts.

A shopper who never saw a popup contributes nothing, because both versions treated them identically. Counting them would make the sample look bigger while making the result weaker.

Statistical significance

A result is significant when the gap between the two versions is large enough that it is unlikely to be a coincidence of who happened to land in which group.

It is a statement about luck, not about size. A significant result can still be a tiny difference — significance tells you the difference is probably real, not that it is worth acting on. Always look at the actual numbers alongside the label.

Not significant

The difference is smaller than your traffic can resolve. That is all it means.

It does not mean the two versions perform the same. Three completely different situations produce this same label:

• The two versions really are equivalent.
• One is genuinely better, but by less than this test could detect.
• One is much better, and the test simply has not collected enough data yet.

The result cannot tell you which of the three you are in. See Reading Your Results.

Confidence

How sure we are that a difference is real rather than luck.

NavonaAI uses 95% confidence — the standard threshold across the industry — so a result here means the same thing it would mean anywhere else. Rate metrics are compared with a two-proportion z-test: the standard way to check whether a gap between two rates is bigger than the random wobble you would expect at that sample size. (Flip 100 coins and get 56 heads and nothing has happened; flip 4,000 and get 56% and something has.)

Higher confidence needs more data. Demanding near-certainty from a low-traffic store is how tests end up running for years.

Power

The chance that a test finds a real difference, assuming one exists.

Confidence protects you from seeing a difference that is not there. Power protects you from missing one that is. NavonaAI sizes tests for 80% power, again the industry standard — meaning that if the improvement you are looking for is real and as large as you specified, roughly four tests in five will catch it. The fifth will come back "not significant" anyway. That is a designed-in limit, not a malfunction.

Minimum detectable effect (MDE)

The smallest improvement a test is capable of noticing at your traffic.

If your MDE is 10%, a test that finishes without a winner has ruled out improvements larger than roughly 10% — it has said nothing at all about a 3% improvement. Smaller MDE means more traffic and a longer test. This is the single knob that most changes how long a test runs, because required data grows with the square of how small an effect you want to catch: halving the MDE roughly quadruples the wait.

Primary metric

The one number that decides the winner. For NavonaAI experiments this is prevention rate — orders divided by interventions shown.

Fixing this in advance is deliberate. If you pick the winning metric after seeing the results, you will eventually pick whichever one happened to look good by chance.

Guardrail metric

A second number that can veto a win, but never award one.

NavonaAI's guardrail is net revenue per intervention. If a version converts better but earns less money per intervention, the guardrail catches it and the recommendation is withheld rather than pointing you at a change that would cost you money.

Cannibalization

Giving a discount to a shopper who would have bought anyway.

Those orders still count as conversions, so cannibalization is invisible in conversion rate — it looks like a win. It shows up only in revenue per intervention, which is why that metric is the guardrail.

Control and treatment (Group A and Group B)

Control (Group A) is the version you are comparing against, usually your current setup. Treatment (Group B) is the change you want to evaluate. Shoppers are split between them.

Holdout

A test where the control group has prevention turned off entirely, so nothing is shown to them.

It answers a different question from a normal A/B test: not "which popup is better" but "does the popup do anything at all". See What You Can Test.

Net revenue

Order revenue with the discount you gave away subtracted.

It is not profit. It does not subtract your cost of goods, so it overstates the value of extra orders. See Reading Your Results.

Average order value (AOV)

Total revenue divided by number of orders. Useful as an explanation of why a version won or lost, but too noisy to decide a winner on — a single unusually large order can move it substantially.