When It Comes to Email Marketing, What Do We Mean by the Term A/B Testing?

The only difference is four words in the subject line. That single data point changes how you write every subject line from that moment forward and the cost of finding it out was a few minutes of setup. That is what A/B testing brings to email marketing. Here is what it means, what to test and how to make sure the results are worth acting on.

What Does A/B Testing Mean in Email Marketing

A/B testing in email marketing means sending two slightly different versions of the same email to a portion of your audience to find out which performs better. Version A goes to one randomly selected group, and version B goes to another. You measure the results, identify the winner and send that version to the rest of your list.

The principle comes straight from the scientific method of changing one variable, keeping everything else identical and observing what happens. In the context of email campaigns, that variable could be the subject line, the button text or the time of day you hit send. When the element you’re testing is something you use in every campaign, knowing which version resonates has a compounding effect on your results over time.

According to HubSpot’s email marketing benchmarks, the average email open rate across industries sits at around 42%. This means the gap between a well-optimised campaign and a mediocre one is real and measurable and subject line testing is one of the fastest ways to close it.

What Can You Test in an Email Campaign?

More than most people think. But some elements produce clearer, more actionable results than others, so they are the right place to start.

Does Your Subject Line Really Drive Open Rates

Subject lines are the most tested element in email marketing and the most impactful. They are the single biggest driver of open rates. Small changes like a question instead of a statement, adding the recipient’s first name or dropping in a specific number, can swing results by several percentage points. In a channel where attention is limited and inboxes are crowded, those details matter.

Is Preview Text Worth Testing

Preview text is the short line that appears next to the subject line in the inbox. Many senders leave it on default or ignore it entirely, which is a missed opportunity. Testing it is low effort and frequently produces a measurable lift in open rates, particularly on mobile where inbox space is tight.

Can Button Wording Change Your Click-Through Rate

Call-to-action wording is a lever most email campaigns leave unpulled. The text on your button or link shapes how people interpret what happens when they click. “Book a free call” and “Get started today” can produce very different click-through rates, even when they point to the same page. If your opens are healthy but your clicks are flat, this is the first place to look.

Does It Matter What Time You Send Email Campaigns

Send time and day affect whether your email catches people at the right moment. A Tuesday morning send and a Friday afternoon send to the same list can tell completely different stories, depending on your audience’s habits and routines. It takes almost no effort to test and the results are often striking.

Does Email Length Affect Engagement

Email length is something many senders assume they have already figured out. Some audiences engage better with short and punchy whilst others want detail and context before they click. A test will tell you which your subscribers prefer, rather than leaving you to rely on instinct.

Does It Matter What Time You Send Email Campaigns

Send time and day affect whether your email catches people at the right moment. A Tuesday morning send and a Friday afternoon send to the same list can tell completely different stories, depending on your audience’s habits and routines. It takes almost no effort to test and the results are often striking.

How Do Open Rates and Click-Through Rates Tell You Which Version Won

The two metrics most used to evaluate A/B test results are open rates and click-through rates. Knowing which one to focus on and when, is what separates a meaningful test from a confusing one.

What Do Open Rates Tell You

Open rates measure the percentage of recipients who opened your email. If you changed the subject line between version A and version B, open rate is the metric to watch. A meaningful difference tells you that one subject line did a better job of earning attention in a crowded inbox. A small or inconsistent difference suggests the change was not effective enough to make a noticeable improvement.

What Do Click-Through Rates Tell You

Click-through rates measure the percentage of recipients who clicked a link inside the email. If you have changed the body copy, the button text, or the layout, click-through rate is where to look first. A higher click-through rate on one version tells you that something in that version was more effective at driving people toward the action you wanted them to take.

Campaign Monitor’s email marketing benchmarks put a healthy click-through rate at between 2% and 3%, which gives you a useful reference point for judging whether a test result represents a meaningful improvement or just normal variation. That matters because the value those clicks generate for your business is the real measure of whether a campaign is working.

Which Metric Should You Focus On

Neither is more important in isolation. The right one depends entirely on what you have changed. If you tested the subject line and nothing else, don’t let a difference in click-through rates distract you. Match the metric to the variable and you will always know exactly what your test is telling you.

How Does an A/B Test Work in Practice

The process is more straightforward than most people expect. You choose one element to test, create two versions of your email that differ only in that element and let your email marketing platform handle the rest.

Most platforms split a portion of your list into two groups at random, send each group one version, wait a set amount of time and declare a winner based on whichever metric you nominated. Some then send the winning version to the remainder of your list automatically. Others ask you to review the results and send them manually.

Either way, the random split is what makes the test meaningful. Email marketing delivers an average ROI of 36:1 according to Campaign Monitor, which makes it one of the highest-returning digital channels available and A/B testing is the most reliable way to push that return higher over time.

What Makes an A/B Test Reliable Enough to Act On

Three things determine whether a test gives you results you can trust and build on.

Why Should You Only Change One Thing at a Time

Testing one variable at a time is the most important discipline in A/B testing. If you change the subject line, the send time and the button colour in the same test, you won’t be able to attribute any difference in results to a specific cause.

How Big Does Your List Need to Be for a Meaningful Test

List size matters more than most people realise. A test run on 200 subscribers rarely produces statistically significant results. You need a large enough sample in each group for any difference to mean something. As a rough guide, aim for at least 1,000 recipients per variant. Most platforms will indicate whether your test has reached statistical significance and that signal is worth paying attention to rather than acting on a small number and hoping it holds.

How Long Should You Wait Before Picking a Winner

Open rates typically peak in the first few hours after sending, but giving the test a full 24 hours is usually the sensible approach before declaring a winner, especially if your audience spans different time zones or has varied daily routines. Patience here saves you from locking in a result that would have been made too quickly.

What Are the Most Common Mistakes People Make with A/B Testing in Email

Most of the mistakes that undermine A/B testing are entirely avoidable once you know what to watch out for. These are the ones that come up most often:

• Testing too many variables at once: It’s been mentioned already and it’s worth mentioning again, because it’s the mistake that quietly ruins more tests than anything else. If you change the subject line, the send time and the button colour in the same test, you won’t be able to attribute any difference in results to a specific cause.
• Acting on too small a sample: A two-point difference in open rates on a list of 300 people isn’t a finding, it’s a coincidence. Treating small-sample results as confirmed insights is how bad habits become ingrained. The confidence you can place in a result increases with the size of the audience it is drawn from.
• Stopping after one winner: Litmus research found that 55% of marketers rarely or never A/B test at all which means the ones who do it consistently already have a significant edge. Running the same test again, or iterating on the winning version, builds a far clearer picture over time. The campaigns that improve most consistently aren’t built on a single experiment, they’re built on a habit of ongoing testing.
• Only ever testing subject lines: They’re a strong place to start, but they’re not the whole story. The gains available in body copy, layout and call-to-action wording are just as important. The further into your emails you’re willing to test, the more you learn about what your audience responds to.

The good news is that none of these are difficult to avoid once they’re on your radar. A simple rule of one variable per test, a minimum audience threshold before you act on results and a commitment to running tests consistently rather than occasionally will put you ahead of many email marketers who are still guessing.

Is A/B Testing Worth It if Your Email List Is Smaller

Yes, with realistic expectations about what the results can tell you. Smaller lists produce noisier data, but they still produce directional information that is worth having and acting on.

If your Monday sends consistently outperform your Thursday sends across three or four tests, that pattern is informative even if it falls short of a strict statistical significance threshold. It’s a reason to lean one way, not a reason to discard the data. Patterns that hold across multiple tests on a smaller list are more reliable than a single test on a large one.

Some variables are better suited to smaller lists than others. Send time and day don’t require a large audience to show a clear directional trend. Subject line testing is harder to draw firm conclusions from on a small list, but running the tests still sharpens your understanding of your audience. That understanding then improves the quality of your email campaigns long before the data becomes fully conclusive. On a smaller list, treat A/B testing as a learning tool rather than a proof mechanism.

Northern Media’s email marketing services give you the strategy, testing framework and analysis to put these insights into practice from the very first campaign, with a team that knows how to read the data and turn it into copy and structure that converts. Get in touch to talk through what that could look like for your business.