Metrics & Results
The three numbers an A/B test reports, the formula behind each, and why the verdict is measured per intervention.
A/B Testing is coming soon. This page describes how experiments will work once the feature launches. Want early access when it's ready? Let us know at support@navona.ai.
An A/B test reports three numbers for each group. They answer three different questions and they are deliberately not interchangeable — one of them decides the winner, one can veto that decision, and one is an early warning.
All three share the same denominator: interventions shown. An intervention is one prevention popup shown to one shopper — the same thing your dashboard's Interventions Shown card counts. That choice of denominator does more to determine whether your test is usable than anything else, and Why we measure per intervention explains it.
The three metrics
| Metric | Formula | Its job |
|---|---|---|
| Accept rate | Accepted / Interventions shown | Leading indicator — moves first |
| Prevention rate | Orders / Interventions shown | Primary — this decides the winner |
| Net revenue per intervention | (Revenue − Discount cost) / Interventions shown | Guardrail — can veto a win |
These are computed only on the shoppers in this test, so they will not match the same-named figures on your Analytics dashboard, which cover your whole store. Same formulas, different populations. Why they differ.
Accept rate — the leading indicator
Accept rate = (Offers accepted / Interventions shown) x 100Did the shopper take the offer? This is the popup's immediate job, and it needs the least data of the three, so it moves first. If a change is going to do anything at all, this is where you see it earliest.
Accept rate is gameable, so it never picks the winner on its own. A bigger discount almost always wins on accept rate. A test judged on acceptance alone would keep pushing you toward larger and larger giveaways, and would call each one a success.
Treat it as a signal that something is happening, not as evidence that it is happening in your favour.
Prevention rate — the primary verdict
Prevention rate = (Orders / Interventions shown) x 100Of the shoppers who were shown a prevention popup, how many went on to complete a purchase.
This is the metric that names a winner. It is a real business outcome rather than a proxy for one — accepting an offer is only interesting if it ends in an order — and it is reachable at real store traffic, which the money metric usually is not.
Net revenue per intervention — the guardrail
Net revenue per intervention = (Order revenue − Discount cost) / Interventions shownThe bottom line, in one number. It already contains conversion, basket size and discount cost, so a version that buys its extra orders with too much margin will show up here even while it looks like a winner on prevention rate.
It is the metric you would most want to decide on. At real store traffic it is usually far too slow to resolve — it needs roughly three times the data that prevention rate needs — so it guards the verdict instead of setting it. See How Long a Test Takes.
This is revenue, not profit. It subtracts the discount you gave away but not your cost of goods, so it overstates the value of extra orders. Full explanation.
Why we measure per intervention
Most of your carts never see a popup. Those shoppers get an identical experience in both groups — there is nothing being tested on them. Counting them makes your sample look bigger while making your result weaker.
Here is one real NavonaAI store, over a single experiment window:
| Stage | Carts |
|---|---|
| Assigned to a group | 3,553 |
| Shown a prevention popup | 1,218 (34%) |
| Accepted the offer | 590 (48% of those shown) |
Two out of every three carts never saw a popup. Include them in the denominator and you are averaging the real difference across a crowd that could not possibly show one. On this store that dilutes the effect by about 2.9x.
The damage is worse than it sounds, because the data you need does not grow by 2.9x — it grows with the square of the dilution, roughly 8x.
Same store, same change, three different denominators:
| Measured on | Denominator | Time to a verdict |
|---|---|---|
| Accept rate | 1,218 shoppers shown a popup | ~2 weeks |
| Prevention rate | 1,218 shoppers shown a popup | ~6 weeks |
| Store-wide cart-to-order rate | 3,553 carts | ~11 months |
These are one real store's numbers, not a typical store's. The change modelled here is a 10 percentage-point move in accept rate, from 48% to 58% — a much larger change than the 10% relative improvement used on the How Long a Test Takes page, which is why the durations there are longer. The absolute weeks depend on how large the real difference turns out to be; the ratios between the rows are the robust part, because the gap between six weeks and eleven months is a property of the denominator, not of the example.
This is why store-wide conversion is not offered as an experiment verdict. It is not hidden from you: in advanced mode it appears in the list with the reason it is unavailable at your traffic. It is simply not a question most stores can afford to ask.
Choosing what decides the winner
Simple mode is the default and has no metric picker. You choose two offer configurations and start; the winner is decided on prevention rate, guarded by net revenue per intervention.
Advanced mode lets you pick the primary metric from the three above and set your own minimum detectable effect and planned duration. Store-wide conversion is shown, disabled, with the estimated time it would need at your store — so you can see for yourself why it is not the default.
Which should you pick? Prevention rate, unless you have a specific reason not to. It is the default because it is the only one of the three that is both a genuine business outcome and reachable at ordinary store traffic.
Supporting numbers
Alongside the three headline metrics, each group reports the figures that explain why it won or lost:
| Figure | What it is |
|---|---|
| Interventions shown | Sample size for this group — the denominator for all three metrics |
| Offers accepted | How many shoppers took the offer |
| Orders | How many shoppers shown a popup went on to purchase |
| Revenue | Total order value attributed to this group |
| Average order value | Revenue / Orders |
| Discount cost | Total value of NavonaAI discounts redeemed by this group |
These decompose the headline. A version that wins on net revenue per intervention is winning through some combination of more acceptances, more conversions, larger baskets, or cheaper discounts — and this table tells you which.
Average order value is useful as an explanation and unreliable as a verdict. It is dominated by a handful of large orders, which is why it never decides a winner. See One huge order can distort revenue.
Daily performance
The daily breakdown tracks each metric for both groups, day by day. It is there to help you spot:
- Trends — is one version consistently ahead, or did it spike on a single day?
- Convergence — are the two getting closer over time? Early gaps usually shrink.
- Anomalies — did a sale, an ad campaign or an outage hit one day disproportionately?
Daily figures are noisy by construction — a single day rarely has enough interventions to mean anything on its own. Use them to sanity-check the overall result, not to call it early.
Next steps
- Reading Your Results — what the verdict does and does not tell you
- How Long a Test Takes — traffic, time, and feasibility
- Glossary — plain-language definitions