A/B test

An A/B test splits visitors at random between two versions of a page and measures which converts better. It's the most reliable way to know whether a change caused an improvement.

Why it matters

Opinions about pages are often wrong, and conversion rates move for many reasons besides your change. A properly run A/B test isolates the effect of one change. The cost is time: most pages need weeks of traffic per test.

How to get it right

• Work out the sample size before you start, and wait for it.
• Test one idea per test, and make it big enough to matter.
• Run for whole weeks so weekdays and weekends are both included.
• Decide what to test with faster methods, so each slow test counts.

Common mistakes

• Stopping the test as soon as one version looks ahead.
• Testing tiny changes on pages without enough traffic to detect them.
• Changing several things at once and not knowing which mattered.