A/B testing
A method of comparing two versions of a marketing asset (a page, email, headline, button, or ad) by splitting traffic between them and measuring which version produces better results on a chosen metric. Sometimes called split testing. A/B testing replaces opinion-driven decisions with data-driven ones, and is the foundation of conversion rate optimization.
Why A/B testing changes how you decide
Three reasons it's the foundation of every conversion-rate program, not just a tactic teams use occasionally.
It replaces opinion with data
The loudest voice in the room usually wins the headline argument; the testing data settles it. The team stops debating which version is better and starts asking which version made more money. That alone changes how marketing decisions get made.
Small wins compound
A 10% lift on the opt-in page, 8% on the sales page, and 12% on the email subject line are small alone but together more than double the funnel's output. A testing program is the only way to find these stacked gains; one-off campaign launches don't produce them.
Audiences surprise you
The variant you expected to lose often wins. Testing surfaces counterintuitive truths about what your audience actually responds to, which makes every future page, email, and ad more accurate. Without testing, you're optimizing for the audience you imagined.
How to run an A/B test from hypothesis to winner
Five steps. Skip any of them and the result either takes longer to reach or isn't actually trustworthy.
-
Pick one variable to test
Headline, button text, image, price display, subject line, call-to-action color. One thing at a time. Change more than one element and you won't know which change drove the result. The variable should be something you can actually act on once you know the answer.
-
Form a clear hypothesis
"Changing the headline from feature-focused to benefit-focused will increase opt-in rate by at least 15%." A real hypothesis names the change, the metric, and the expected direction. Without it, you'll cherry-pick whichever interpretation flatters the variant you preferred.
-
Split traffic evenly
50/50 split, randomly assigned. Same traffic source, same time window, same everything except the one variable. If variant B only runs on weekends and variant A only on weekdays, you're testing the day of the week, not the change.
-
Reach statistical significance
Wait until the difference between variants is unlikely to be random noise. The standard threshold is 95% confidence. Most testing tools display this live; stop the test only when significance crosses 95%, not when one variant looks ahead. Early leads reverse all the time.
-
Ship the winner, then test the next variable
The winning variant becomes the new baseline. Then you pick the next variable: if the headline won, test the subhead next; then the image; then the CTA. A continuous testing pipeline produces the compounding gains. One test, one ship, one new test.
What an A/B test actually looks like
Three real-world scenarios from e-commerce, email, and a course funnel. The lifts are typical, not exceptional.
Headline test on a product page
A skincare brand tests their product page headline. Variant A: "Premium vitamin C serum." Variant B: "Visible results in 14 days or your money back." Both run for 12 days with 4,200 visitors per variant. Variant B wins with a 22% lift on add-to-cart rate and 97% confidence.
Subject line test on a broadcast
A newsletter writer sends two subject lines to 1,000-contact samples. Variant A: "This week's top 3 ideas." Variant B: "The mistake I keep making with podcast guests." The winner (B) gets a 38% open rate vs 21% for A; the platform auto-sends B to the remaining 18,000 contacts.
Video vs text sales page
A course creator tests a 12-minute video sales letter against a long-form written sales page. Same offer, same price, same traffic. Video wins for cold traffic from ads (+18% conversion); text wins for warm traffic from email (+14% conversion). The creator now serves different versions by traffic source.
What to watch during an A/B test
Eight numbers that tell you whether the test is real, the winner is real, and the lift will hold up after you ship it.
Conversion rate per variant
The primary metric: percentage of visitors who completed the conversion goal. Display side by side for A and B with absolute and relative difference.
Statistical significance
The probability the difference between variants is real. 95% is the standard threshold for declaring a winner; below that, the result is inconclusive.
Sample size per variant
Number of visitors or sends per variant. Drives how reliable the result is. Use a sample size calculator to set the target before starting the test.
Confidence interval
The plausible range for the true lift. "Variant B is 22% better, plus or minus 5%" is informative; "B is 22% better, plus or minus 35%" means you don't know yet.
Relative lift
Percentage improvement of the winner over the baseline. A 0.5 percentage point gain on a 2% conversion rate is a 25% relative lift; expressed both ways for clarity.
Test duration
How long the test has run. Most tests need at least 7 days to absorb weekday/weekend variance; long sales cycles need much longer.
Secondary metric impact
Does the winning variant hurt anything downstream? A page that converts more visitors but produces lower-paying customers is not actually a win.
Traffic split balance
Visitors should be 50/50 across variants. A drifting split (60/40, 70/30) means the random assignment is broken and the result isn't trustworthy.
Related glossary terms
A/B testing touches almost every conversion concept. These are the terms that change meaning once you start testing.
How systeme.io handles A/B testing
Native A/B testing for funnel steps and email subject lines. Automatic traffic split, live conversion-rate comparison, auto-winner. Included on Startup and above.
Funnel-step A/B testing
Duplicate any sales page, opt-in page, or order form as variant B and start splitting traffic. systeme.io tracks conversion rate per variant in the dashboard with significance reporting built in.
Email subject line testing
Send variant A to a sample of the list, variant B to another sample, then automatically send the winning subject line to the rest based on open rate. No manual coordination required.
Automatic traffic split
50/50 random assignment by default with adjustable ratios. Visitors are sticky: a contact who saw variant A keeps seeing variant A on return visits, so the test data stays clean.
Live per-variant analytics
Conversion rate, sample size, and confidence level update in real time. Watch the test reach significance instead of guessing when to stop.
Winner promotion
Once a test reaches significance, promote the winner to 100% of traffic with one click. The losing variant is archived; the winner becomes the new baseline for the next test.
No third-party tool needed
A/B testing is built into the funnel and email modules. No Google Optimize replacement to configure, no JavaScript snippets, no flickering page loads while a test script swaps the variant.
Frequently asked questions
Common questions about A/B testing, with the practical answer for small and mid-size marketing teams.
A/B testing is a method of comparing two versions of a marketing asset by randomly splitting traffic between them and measuring which version performs better on a chosen metric. The two versions (A and B) differ in one element: a headline, a button color, an email subject line, an image, a price. Whichever version produces more conversions, opens, or clicks wins, and that becomes the new baseline for the next test. A/B testing replaces guesswork with data.
The honest answer: it depends on your baseline conversion rate and the size of the lift you want to detect. A useful rule of thumb is at least 100 conversions per variant before drawing conclusions, and at least 1,000 visitors per variant for low-conversion-rate pages. Small lists need bigger relative differences to detect a real winner. Use a sample size calculator to get a real number rather than guessing; testing tools usually have one built in.
Long enough to reach statistical significance (usually 95% confidence), and long enough to capture at least one full business cycle. For most pages and emails, that means a minimum of 7 days, often 14, to absorb the difference between weekday and weekend behavior. Stopping a test the moment one variant looks ahead is the most common A/B testing mistake; early leads frequently reverse as the sample grows.
Statistical significance is the probability that the difference between your two variants is real, not random noise. The standard threshold is 95% confidence, which means there's a 5% chance the winner you picked is actually no better than the loser. Below that threshold, the result is inconclusive: you don't know if B is better than A, you just have a number that hasn't stabilized yet. Tools usually display significance live; only declare a winner once it crosses 95%.
A/B testing changes one element at a time and compares two versions; multivariate testing changes multiple elements simultaneously to find which combination performs best. Multivariate is more efficient when you have heavy traffic and many elements to optimize, but it needs much larger sample sizes (often 10x or more). For most small and mid-size businesses, A/B testing produces better results because you can actually reach significance. Start A/B, graduate to multivariate when traffic justifies it.
systeme.io includes native A/B testing for funnel steps and email subject lines. Create variant B of a sales page, opt-in page, or order form; systeme.io splits traffic automatically and shows conversion rate per variant live. For emails, send variant A to a sample of the list, variant B to another sample, then auto-send the winner to the rest based on open rate. Included on the Startup plan and above, no third-party tools required.
Run your first A/B test inside systeme.io
Native A/B testing for sales pages, opt-in pages, and email subject lines. Live conversion-rate reporting, automatic winner promotion, no third-party tools.
Start for free now