Reading Your Results

An A/B test result is only useful if you read it correctly. This page covers the four things that most often get misread — and the one situation where NavonaAI deliberately refuses to tell you what to do.

One recommendation, not three scoreboards

You see all three metrics, each with its own status: either a verdict, or an estimate of how much longer it needs.

But you only ever get one recommendation, and it names the metric it came from.

That is on purpose. If you watched three metrics and acted on whichever one turned green first, you would eventually act on pure chance — with three chances to find a "winner", something will look like one sooner or later. Fixing the deciding metric in advance is what stops that.

What "significant" means

A result is significant when the gap between the two versions is large enough that it is unlikely to be an accident of who happened to land in which group.

Two things it does not mean:

• It does not mean the difference is big. With enough traffic, a 0.3% improvement can be significant. Significance says the difference is probably real; it says nothing about whether it is worth having. Read the actual numbers next to the label.
• It does not mean it will hold forever. It means the difference was real during the window you tested, with the traffic you had. Seasonality, a change to your catalogue, or a different ad mix can all change the answer.

"Not significant" does not mean "no difference"

This is the single most important line on this page.

"Not significant" means the difference is smaller than your traffic can resolve. It is a statement about the limits of your data, not a finding about your two popups.

Three completely different situations produce this same label:

Nothing in the result distinguishes them.

If what you actually want is a confident "these are equivalent, stop worrying about it", that is a harder question than finding a winner and needs considerably more traffic. At most store volumes it is not answerable.

Why an inconclusive result is not a broken feature

Before releasing A/B testing we ran four A/A tests — both groups configured identically, so there was genuinely nothing to find. A trustworthy system has to come back empty on those.

All four did. Every metric on every one of the four came back non-significant, with p-values spread between 0.22 and 0.97 — nowhere near the threshold. The leftover gaps were small and pointed in inconsistent directions, which is exactly what pure noise looks like.

So when your test says "not significant", the machinery is working. It is reporting the limit of your data rather than inventing a winner to look useful.

When a recommendation is withheld

Sometimes the primary metric and the guardrail disagree: Version B converts more shoppers, but earns less money per intervention.

When that happens, NavonaAI does not recommend B. It tells you the two metrics point in opposite directions and leaves the decision with you, with both numbers on screen. Something like:

Keep A — B wins on acceptance (58% vs 48%) but earns less per intervention ($2.10 vs $2.35).

An automated "B wins!" in that situation would be pointing you at a change that loses money.

Why that happens: cannibalization

A more generous offer converts more shoppers. Some of those shoppers were going to buy anyway.

You have now paid a discount for an order you already had. That is cannibalization, and it is invisible in conversion rate — those orders still count as conversions, so they look like a win. It shows up only when you divide money by interventions shown.

This is the entire reason net revenue per intervention exists as a guardrail. It is the one number that can tell "we won more customers" apart from "we paid our existing customers to stay".

Revenue is not profit

Net revenue per intervention subtracts the discount you gave away. It does not subtract your cost of goods.

That matters, because extra orders bring extra cost:

Because cost of goods is missing, the metric overstates the value of extra orders. A version that wins narrowly on net revenue may not be winning on profit at all.

One huge order can distort revenue

Order values are heavy-tailed: most orders cluster around your average, and a few are enormously larger. Our production data contains a genuine single order of $153,400.

One order like that, landing by chance in one group, can swamp every other order in the comparison. In an internal analysis it produced a spurious "+26% average order value" that vanished once corrected.

So for the revenue comparison we cap unusually large orders at the 99th percentile before running the test, and report both the capped and the uncapped figures. If those two disagree sharply, one or two large orders are driving your result — treat it with suspicion.

Revenue is also checked differently from the two rate metrics. Accept rate and prevention rate are percentages, and are compared with the standard test for percentages. Net revenue per intervention is an amount, which behaves very differently, so it uses a test built for amounts. This is part of why it needs so much more data before it will say anything.

Two limits worth knowing about

Experiment numbers will not match your Analytics page

They are not measuring the same population, and they are not supposed to.

Your Analytics dashboard covers every cart and order in the period. An experiment covers only the shoppers who were actually placed into a group — and it can only count an order if that order's cart was part of the test.

The gap is not small. Across the four stores we measured, 12% to 28% of all orders happened on carts that were never part of the experiment — because the cart was never tracked, or never reached the point where a group is assigned. Those orders are invisible to the test, in both the top and the bottom of every rate.

A few repeat shoppers land in both groups

Shoppers are split by cart, not by person. A shopper who comes back later and starts a new cart gets assigned again — and about half the time, to the other group.

Measured across the same four stores, this affects roughly 0.5% to 2% of carts, and the share grows the longer a test runs. The effect is to shrink whatever real difference exists by at most about 2% of its size — a true 10% improvement would read as roughly 9.8%. That is well inside the noise at any store's real traffic, so it does not change decisions. We are telling you because it is a real property of the measurement, not because you need to act on it.

A checklist before you act on a result

1. Is the primary metric significant? If not, you do not have a winner — see How Long a Test Takes.
2. Does the guardrail agree? If not, the win may be costing you money.
3. Is the difference big enough to bother with? Significant and meaningful are different things.
4. Would it survive your margin? The metric is revenue, not profit.
5. Did the test run at least one full week? Day-of-week effects are real.
6. Do the capped and uncapped revenue figures agree? If not, a few large orders are driving it.

Related

• Metrics & Results — the three numbers and their formulas
• How Long a Test Takes — traffic and time to a verdict
• Glossary — plain-language definitions