Best Practices
Tips for running effective A/B testing experiments
Run for at Least 7–14 Days
Shopper behavior varies significantly by day of week. Running for less than a week can skew results based on which days happen to fall in your test window. A two-week minimum captures at least two full weekly cycles.
Ensure Enough Traffic
Experiments need a sufficient sample size to produce reliable results. As a rule of thumb, aim for at least a few hundred cart events per variant before drawing conclusions. Your account manager can advise on the right volume for your store.
Low-traffic stores benefit most from the volume-based termination rule — set a target sample size rather than a fixed duration, so the experiment runs until you have enough data regardless of how long that takes.
Test One Variable at a Time
Changing multiple settings between control and treatment makes it impossible to know which change caused the outcome. Keep all other settings identical between your two variants.
Good: Control at 10% discount, treatment at 15% discount (same everything else)
Risky: Control at 10% percentage, treatment at €5 fixed amount with a minimum purchase (three changes at once)
Don't End Experiments Early
It's tempting to stop when one variant looks like it's winning after a few days, but early results can be misleading. A small number of high-value orders in one variant can make it look dominant when the real long-term performance is different. Let the experiment run to its intended duration or volume target.
Start with a Holdout Test
The most valuable first experiment: control with AI prevention off vs. treatment with AI prevention on. This gives you a clean baseline measurement of how much lift NavonaAI generates for your store — and the data to prove it.
Keep a Record
After promoting a winner, note what you tested and the results. This helps you build institutional knowledge about what works for your customers and informs future experiments.
Ready to run an experiment? Reach out to your account manager and they'll handle the setup. Self-service experiment creation is coming soon.