A/B testing: the complete method for testing without fooling yourself

A/B testing is the only method that replaces opinions with facts: two versions of a page, randomly split traffic, and your conversions as referee. But it must be practised rigorously — a badly run test produces false certainty, which is worse than no test at all.

The principle, in one minute

You show version A (the original) to half the traffic, version B (the variant) to the other half. The split is random and both groups live through the same period — promotions, seasonality and news affect both equally. If B converts significantly better than A, you ship B. The method's power lies in the word significantly: telling a real effect from statistical noise.

Before testing: the hypothesis

A good test starts from an observation, not a whim. The healthy pipeline:

1. Detect: your funnels show a step leaking abnormally — say 70% abandonment between basket and delivery.
2. Understand: heatmaps and session replay show visitors going back and forth over shipping costs, discovered late.
3. Formulate: "Showing shipping costs on the product page will reduce basket abandonment, because the bad surprise disappears." A hypothesis = one change + an expected effect + a mechanism.

Testing without a hypothesis ("let's try a green button") is rolling dice with your traffic.

Sample size: the calculation that prevents 80% of mistakes

Fix three parameters before launch:

• The page's current conversion rate (e.g. 3%)
• The minimum effect worth shipping (e.g. +15% relative, i.e. 3% → 3.45%)
• The confidence level (standard: 95%, with 80% power)

A sample-size calculator then gives you the number of visitors required per variant — around 15,000 per branch in this example. Two practical consequences:

• Low traffic? Aim for bigger effects (section redesign rather than button shade) or test higher in the funnel, where volume lives.
• The sample sets the duration. You stop at the planned point, not when the curve looks pleasing.

The four deadly sins of A/B testing

1. Stopping early. Checking the test daily and stopping the moment it turns "significant" guarantees false positives: over a test's life, significance fluctuates. Fix the sample, wait, conclude.
2. The multi-change test. If B changes the headline, the image and the price, even a positive result teaches you nothing reusable. One test = one hypothesis.
3. Forgetting segments… or torturing them. The overall result comes first. Slicing after the fact into 15 segments until one looks like a "winner" is p-hacking. Segments are defined before the test — or become the next test.
4. Ignoring the day of the week. A test launched Tuesday and stopped Friday saw neither a weekend nor a Monday. Two full weeks minimum.

What to test first? The impact hierarchy

At equal traffic, not all tests pay alike. By decreasing observed impact:

1. The offer and its wording — the hero promise, pricing structure, free trial with or without a card. Double-digit gains almost always come from here.
2. Structural friction — number of fields, funnel steps, the moment costs appear. This is the territory of fixes diagnosed through replay.
3. Information hierarchy — section order, CTA position relative to the 50% scroll line (see our heatmap guide).
4. Cosmetics — colours, button wording, images. Real but small effects: keep them for very high-traffic pages, the only ones able to detect them.

Starting with cosmetics on a 10,000-visits/month site is looking for pennies under a streetlight.

Interpreting results: three scenarios

The variant wins clearly. Ship it, then re-measure at 30 days: the novelty effect (regulars noticing the change) fades and sometimes shaves part of the gain. The true number is the cruising-speed one.

The test is neutral. A useful result, not a failure: you now know this element is not a lever, and you saved a pointless deployment. Document it and move to the next hypothesis — a healthy testing programme runs 50–70% neutral.

The variant loses. The most instructive of the three. A clear loser on a strong team conviction is gold: it reveals a gap between your mental model of the visitor and reality. Dig into replays before burying the idea — sometimes the execution failed, not the hypothesis.

Multivariate tests and sequencing

The temptation of the multivariate test (MVT) — headline × image × CTA simultaneously — collides with arithmetic: 3 × 2 × 2 = 12 combinations, hence twelve times the required sample. Outside very large traffic, prefer sequencing: successive A/B tests, from the strongest lever to the weakest, each building on the previous winner. Less elegant on paper, but convergent in practice — and every result stays interpretable on its own.

Targeting intelligently: behavioural personas

Not all visitors are alike, and some tests only make sense for a segment: new mobile visitors from search don't share the frictions of returning desktop users. Modern targeting builds on real behavioural data — device, source, country, operating system — rather than demographic guesses.

In practice with Mirage — A/B testing is built into the analytics: a visual no-code variant editor, targeting through personas built from your real traffic data (source, device, OS, country — with an estimate of the audience share covered), results measured directly against your conversion goals, with lift estimation. No third-party cookies, no extra tool. Free 30-day trial.

A/B testing and GDPR

Two points of attention:

• Variant assignment. It must be stable (a visitor always sees the same version) without relying on an advertising cookie or a consent-requiring identifier. Anonymous mechanisms exist; privacy-first tools use them natively.
• Measurement. Aggregated, anonymous counts: the frame is that of consent-exempt measurement. Google Optimize shut down in 2023; US alternatives raise the transfer question — our comparison covers the stakes.

FAQ

How long should an A/B test run?

At least two full weekly cycles (14 days), even if significance shows up earlier: behaviour varies strongly by day of the week. And never less than the time needed to reach the sample size calculated before launch.

How many conversions does a reliable A/B test need?

Order of magnitude: 150 to 300 conversions per variant to detect a substantial effect (+15–20% relative). Detecting a small effect (+5%) takes several thousand conversions per variant. If your page converts little, test higher in the funnel or aim for bolder changes.

Is A/B testing compatible with GDPR without a banner?

Yes, if variant assignment does not rely on a consent-requiring tracker and measurement is anonymous. A test that splits traffic without identifying individuals and counts aggregated conversions stays within the same frame as consent-exempt audience measurement.

Why doesn't my winning test hold up afterwards?

Three usual suspects: the test was stopped as soon as it turned "significant" (a lucky peak), the sample was too small, or the novelty effect wore off. Hence the rules: sample size fixed in advance, minimum duration respected, and when in doubt, re-test.