A/B Testing Overview
Run controlled experiments to find the prevention settings that convert best for your store
A/B Testing is coming soon. This page describes how experiments will work once the feature launches. Want early access when it's ready? Let us know at support@navona.ai.
A/B testing lets you compare two different prevention offers side by side and let real shopper behaviour decide between them. Instead of guessing whether a 10% or a 15% discount performs better — or whether offering a discount at all makes any difference — you run a controlled test and read the answer off the data.
What an A/B test actually does
The mechanic is simple:
- You set up two versions of your prevention offer — Group A (your control, usually your current setup) and Group B (the change you want to try). What varies is the offer and its rules, not the popup's wording or design; see What You Can Test.
- NavonaAI splits your incoming shoppers between them, roughly half and half.
- Both groups shop your store normally. Neither knows it is in a test.
- NavonaAI counts what happened in each group — how many accepted the offer, how many bought, and how much money each version made per intervention.
- When there is enough data to be confident the difference is real and not luck, you get a verdict.
That last step is the one that trips people up. "Enough data" is not a formality — at low traffic it can be more data than your store will produce this year. How Long a Test Takes is the page to read before you start one.
How shoppers are split
Shoppers are assigned by cart. The first time NavonaAI loads for a cart, that cart is placed in Group A or Group B and it stays there — a cart never switches groups mid-session, so the shopper gets one consistent experience.
The split alternates rather than flipping a coin, so the two groups stay evenly sized as the test runs rather than drifting apart.
Assignment happens when NavonaAI first loads for a cart, which is earlier than the moment a popup is shown. Most assigned carts never trigger a popup at all — those shoppers got an identical experience in both groups. This is why results are measured per intervention rather than per cart; see Why we measure per intervention.
Only one experiment can run per store at a time.
What you get at the end
Three numbers per group, one recommendation, and an honest statement of what the data can and cannot support:
- Accept rate — the earliest signal, and the easiest to game.
- Prevention rate — the metric that decides the winner.
- Net revenue per intervention — the guardrail that can veto a win.
If the primary metric and the guardrail disagree, NavonaAI says so rather than picking for you. If there is not enough data, it says that too — and does not dress it up as "no difference".
Next Steps
- What You Can Test — configurable settings per variant
- Experiment Lifecycle — from draft to winner promotion
- Metrics & Results — the three KPIs and the formula behind each
- How Long a Test Takes — traffic, time, and whether a test is feasible at all
- Reading Your Results — what the verdict does and does not tell you
- Glossary — plain-language definitions
- Best Practices — tips for running effective experiments