Best Practices
Tips for running effective A/B testing experiments
Run for at Least 7–14 Days
Shopper behavior varies significantly by day of week. Running for less than a week can skew results based on which days happen to fall in your test window. A two-week minimum captures at least two full weekly cycles.
Ensure Enough Traffic
Experiments need a sufficient sample size to produce reliable results. As a rule of thumb, aim for at least a few hundred cart events per variant before drawing conclusions.
Low-traffic stores benefit most from the volume-based termination rule — set a target sample size rather than a fixed duration, so the experiment runs until you have enough data regardless of how long that takes.
Test One Variable at a Time
Changing multiple settings between control and treatment makes it impossible to know which change caused the outcome. Keep all other settings identical between your two variants.
Good: Control at 10% discount, treatment at 15% discount (same everything else)
Risky: Control at 10% percentage, treatment at €5 fixed amount with a minimum purchase (three changes at once)
Don't End Experiments Early
It's tempting to stop when one variant looks like it's winning after a few days, but early results can be misleading. A small number of high-value orders in one variant can make it look dominant when the real long-term performance is different. Let the experiment run to its intended duration or volume target.
Start with a Holdout Test
The most valuable first experiment: control with AI prevention off vs. treatment with AI prevention on. This gives you a clean baseline measurement of how much lift NavonaAI generates for your store — and the data to prove it.
Keep a Record
After promoting a winner, note what you tested and the results. This helps you build institutional knowledge about what works for your customers and informs future experiments.
A/B testing is coming soon. Want early access when it launches? Let us know at support@navona.ai.