By Kaleigh Moore May 16, 2023
In a world where people are bombarded with countless emails on a regular basis, it’s more important than ever to craft emails with purpose.
In 2022, over 333 billion emails were sent and that figure is expected to rise to a staggering 376 billion by 2025.
These days it’s not enough to assume you know what type of email your audience will want to open — let alone read through it entirely.
Creating great emails requires a lot of hard work, researching, and strategizing. The best emails are crafted not only with goals in mind, but also with the target audience at the forefront.
So how can you be sure which email will be more successful than others?
You’re not the first person to ask that question.
What if there was a way to be sure that one version of an email would generate more engagement, lead to more landing page views, and/or provoke more sign ups?
Well . . . there is – with testing.
Testing your emails is a brilliant way to determine what resonates with your audience and what sparks their interest. With email A/B testing, you can gather data-backed proof of the effectiveness of your email marketing.
In this guide to email A/B testing you’ll learn
- What is A/B testing
- Why you need to split test your emails
- Setting goals for A/B testing
- What you should test
- Email A/B test case study
- Best practices for email testing
- Setting up an A/B email test
- Tracking and measuring A/B test results
- Get started with your own email test
And by the end you’ll know how to set up a successful email split test.
But before we get started, it’s important to know what email marketing A/B split testing is.
What is A/B testing?
A/B testing, also known as split testing, is a method that lets you scientifically test the effectiveness of an experiment, in this case, an email experiment.
When split testing, you create two versions (called variants) of an email to determine which email statistically performs better. Once you find which email variant performs best, you can update your email strategy based on what you learned about the winning email.
This allows you to identify what emails engage your subscribers best, which can ultimately help you increase conversions and revenue.
Why you need to split test your emails
Split testing is an effective way to find out what’s working and what’s not in your email marketing. Rather than assuming your customers would prefer one kind of email over another, you can run a split test to find out in a methodical way.
The more you split test, the more information you’ll have on hand for your future emails. And while a once-and-done test, or even an occasional test, can yield information that will expand your marketing knowledge, regular testing can provide you with a long-term successful email marketing strategy.
Setting goals for A/B testing
Like anything in digital marketing, having a clear goal and purpose for testing is essential. Sure, you can run a quick email test and obtain useful results, but having a more precise testing strategy will yield more powerful data.
A/B testing your emails is a great tool to use at any time, but it can be especially useful if you want to gain insight on a new campaign or email format. Before you begin your test, first establish what you are testing and why.
A few questions that can help guide you at this stage include:
- Why are we testing this variable?
- What are we hoping to learn from this?
- What is the impact this variable has in relation to the performance of this email?
In theory, you could test any element of an email, but some variables will give you more insight into your subscribers’ minds than others.
The beauty of split testing is that no variable is too small to test.
What you should test
It can be tricky to identify what test can help you improve key metrics. From subject line strategies to sound design principles, there are many components that make up a successful email. Understanding each email key performance indicator (KPI) and the email components that impact those KPI’s helps identify what you should be testing.
Open rate
Your open rate is the percentage of customers who opened your email. It’s calculated by dividing the number of unique opens by the number of emails delivered.
If you have a low open rate you should be testing your subject line or preheader (the preview snippet of text that is next to your subject line or below it (on mobile) in your inbox).
Subject lines are crucial because they’re the first things people see in their inbox. Split testing your subject lines can help make for more successful emails.
Subject line test ideas:
- Short vs long subject line
- More urgent language
- Try an emoji
- All capitalized words vs sentence case
- With and without punctuation marks
- Single word subject lines
- Statement vs question
The better your subject line, the more likely your subscribers are to open the email and read through. Having a solid subject line is like getting your foot in the door.
In addition to testing subject lines, try sending the test emails at different times of day and see if that has an impact on your open rate. Your subscribers may be more inclined to open an email in the morning or at night after dinner instead of during the middle of a workday.
Example:
Click-through to open rate
Your click through to open rate is the percentage of unique clicks in an email divided by the number of unique opens.
There are several elements within the body of your email which you need to look at if you have a low click through to open rate or if you’re looking to improve an already strong email.
Keep subscribers interested throughout the email by providing eye-catching, engaging content. If it’s your click-through rate you want to improve, make sure you create clickable content. Consider how interactive content, information gaps (missing pieces of info that spark a reader’s curiosity), or contests could boost your in-email engagement.
There are also many variables you can test to optimize for click-through rate — a strong call to action, intriguing anchor text, personalization, spacing, or bold imagery. Just remember to test one at a time to ensure you know precisely why subscribers are clicking more (or less).
This is why testing is so important. You can look at an email and make some assumptions as to why your performance is low. But if a change is made without testing and that theory is wrong, then you’re setting your email efforts back even further.
Email body test ideas:
- Different color call to action button
- Image vs no image
- Email message length
- Soft sell vs hard sell
- GIF vs no GIF
- Personalization vs no personalization
Design elements like colors, fonts, images, templates, and spacing are just as crucial to an email as the copy and links.
Did you know that 47% of emails are opened on mobile devices? With this in mind, think about how your email visually appeals to subscribers and what they need to get the best reading experience.
Test different templates, layouts, and formats to see which yields the best results for your email campaigns.
Examples:
Opt-out (or unsubscribe) rate
Your unsubscribe rate is the percentage of customers who opt-out of receiving future emails from you.
If your unsubscribe rate is high, then you may be sending too many emails or the email content isn’t relevant.
So make sure you test the frequency of emails and the relevancy as discussed above.
Email A/B test case study
AWeber customer and photo sharing community Light Stalking split their email subject lines to gauge the success of one versus the other.
As a result, they were able to increase their web traffic from the winning subject line email by 83%.
How’d they do it?
Light Stalking wanted to run an email A/B test on the subject line of their weekly challenge email, which asked subscribers to send in a photo of a silhouette.
The test was simple: they created two identical versions of the same email, changing only the subject lines. The first email used a straightforward subject line, “The Weekly Challenge is Live!” and the second email was just one word and hinted at the nature of the challenge, “Silhouettes.”
The email with the shorter headline (“Silhouettes”) was the winner. The email yielded an above-average click-through rate, which drove more people to the Light Stalking website and increased overall engagement levels.
Impressive, right? And simple. This is a perfect example of how email A/B testing helps you make data-backed decisions.
Best practices for email testing
Email A/B testing seems pretty straightforward, right?
It is, but like any experiment, if you don’t solidify the details and ensure your test is valid, your results may turn out to be useless.
Keep these things in mind when creating your split test:
1 – Identify each variable you want to study
Prioritize your tests. Run split tests for your most important and most frequently sent emails first. And know what you want to fix about your emails before you run tests.
Create a split testing plan where you conduct one email split test a week or one email split test per month.
2 – Test one element at a time
Never test more than one change at a time. Have a control email that remains the same and a variant with one change — like a different color CTA button, or a different coupon offer — you want to test. If you have multiple variables, it’ll become difficult to identify which one caused a positive or negative result.
3 – Record the test results
Keep records of the email split tests you’ve performed, the results of those tests, and how you plan to implement your learnings. Not only will this keep you accountable for implementing changes, it will allow you to look back on what did and didn’t work.
4 – Use a large sample to get as close to statistically significant as possible
Achieving statistical significance means that your finding is reliable. The larger the sample pool for your test, the more likely you are to achieve statistically significant results. You can be more confident that your findings are true.
5 – Make sure your sample group is randomized
Tools like AWeber’s split testing make sure that your sample group is completely randomized.
Setting up an A/B email test
You have the basics of email A/B testing down, so let’s next discuss how to set one up properly.
1 – Determine your goals
First things first: Identify the intentions behind the campaign you want to test.
Your goals will act as your compass when figuring out the details of your email A/B test. Every component of your campaign should trace back to your end goals.
2 – Establish test benchmarks
Once you have defined your goals, take a look at your current email data and examine how your previous email campaigns have fared. From there, use your findings as benchmark numbers.
These numbers will be significant when it comes time to analyze your email A/B test data so you can gauge early success. These numbers should also help you decide on the variables you want to test moving forward.
3 – Build the test
You have your goals and your benchmark data; now it’s time to build your test. Remember to test only one variable at a time.
Bonus: Did you know AWeber Lite and Plus customers can automatically split test their email campaigns (and can test up to three emails at a time)?
4 – How big should your test sample size be?
You want your test list to be large enough that you can gauge how the rest of the subscribers will likely react without using the entire list, but just small enough that you can send the winning version to a large portion of your audience. The goal is to get accurate, significant results, so bigger lists typically work the best.
However, keep in mind that you should be using a sample that represents the whole list, not just a specific segment.
There are many ways to approach this. You can figure out a generic sample size with a calculation that factors in your email list confidence level, size, and confidence interval.
Or, if you’re an AWeber customer, you can manually select the percentage of your list that will receive each version of the split test.
Either way, make sure you select a viable percentage of your list to send your test emails to so you have enough data to analyze. Often this is in the 10% to 20% range.
5 – How long should an email A/B test run?
The answer to this question depends on your list size. If you have a large list, then you may only need to send a single email marketing campaign. Bottom line is you want to make sure that you receive enough opens or clicks (depending on the goal of the email campaign) to ensure the results are statistically significant.
You want to make sure that your test results are at least 90% statistically significant in order to confidently conclude that your test is a winner or loser.
Run your test results through an A/B testing significance calculator to determine the percentage of confidence your test results will hold when you implement your test in future campaigns.
Once your test has ended and as you begin analyzing your data, keep detailed notes of your findings. Ask yourself:
- What metrics improved?
- What elements of the email flat-out didn’t work?
- Were there any patterns that correlated with past tests?
Maintaining records and tracking results will help guide future campaign optimizations.
Put together a testing roadmap or a detailed record of what you’ve tested, the results, and what you plan on testing in the future. That way, you’ll have a detailed account of your tests and won’t leave any stone unturned in the process.
Tracking and measuring A/B test results
With so many elements to test, you might be thinking, “How can I verify that a campaign is successful or that a test yielded helpful data?”
The answer: Think back to your goals. Your goals will tell you what metrics you should pay the most attention to and what you should work on improving – open rate, click rate, delivery rate.
For example, if generating more leads from email campaigns is your goal, you’ll want to focus on metrics like open rate, click-through rate, and form fills.
It’s also important to look at your metrics as a whole to see the big picture of how an email performed. Being able to track that data and refer back to it will also help you optimize future campaigns.
Get started with your own email test
Email A/B testing is imperative to the success and optimization of any email campaign. It allows you to gain real insight that can help you make decisions about existing and future emails.
Email marketing is always changing, and as subscribers’ attention spans seem to get shorter, it’s vital to know what will yield the most success.
The important thing to remember when it comes to creating an email A/B test is that it doesn’t have to be a complicated process. Email A/B testing is designed to deliver powerful, straightforward insights without a bunch of confusing variables.
Not sure about what font to use for the body of the email? Test it. Going back and forth between a few colors for the CTA button? Test it.
The bottom line: You can and should test different variables of your email campaign before launch so you can optimize for success. Just be sure you’re testing only one variable at a time to get the most accurate and useful results possible.
Download your free email planning template
This email marketing planning template (available in both Excel and Google Sheets) is set up so you can quickly and easily measure the performance of all your email sends and tests. Download it today.
AWeber is here to help
Already an AWeber customer? Start executing your A/B testing strategy today. Our email A/B testing tool allows you to test just about any element of your email (subject line, calls-to-action, colors, templates, preheaders, images, copy, and more!).
Not an AWeber customer yet. Then give AWeber Free a try today.
Keep reading:
FAQs
How to do ab testing for emails? ›
In your email marketing tool, you simply set up 2 emails that are exactly the same except for 1 variable, such as a different subject line. You then send the 2 emails to a small sample of your subscribers to see which email is more effective. Half of your test group receives Email A and the other half gets Email B.
Which aspect of the email body can be evaluated with a b email testing? ›A/B testing can be done on all individual elements of an email like subject line, preview text, from name, or on the whole email such as changing the template, long copy vs short copy. In this guide, we will look at what you should test, and tips for running an effective a/b test.
What is email ab testing results? ›A/B testing, in the context of email, is the process of sending one variation of your campaign to a subset of your subscribers and a different variation to another subset of subscribers, with the ultimate goal of working out which variation of the campaign garners the best results.
What is a B split testing in email marketing? ›A/B testing, also known as split testing, is a way of working out which of two campaign options is the most effective in terms of encouraging opens or clicks. In an A/B test you set up two variations of the one campaign and send them to a small percentage of your total recipients.
What is the AB test email subject line? ›A/B testing is a way of comparing how version A of something, like your email subject line, performs compared to version B, within an audience, like your contact list: An email with subject line A is sent to up to 50% of your selected contacts, while a second email with subject line B is sent to an equal portion.
What are the 5 sections of an email to evaluate? ›Emails can be broken into 5 major parts: the sender, subject line, salutations, body, and CTA. These pieces make up 99% of emails and provide an optimal format for engaging with clients and optimising conversions.
What are the 4 main parts of an email body? ›- The subject line. Arguably the most important component of the email, the subject line is the deciding factor in whether your message is read or deleted. ...
- The salutation. ...
- The bit in the middle. ...
- The ending.
Email is not totally secure, so you should avoid sending sensitive information like credit card numbers, passwords, and your Social Security Number.
What percentage of ab tests fail? ›A properly designed experiment will be successful no matter how the results turn out, negative, positive or simply flat. While your 'success' rate can happily hover at 10% (and it absolutely will sometimes be lower), you need to ensure that your 'learning rate' is at 100%.
Why is it important to a b test your follow up email templates? ›A/B testing your follow up emails can help you optimize your email strategy and increase your sales results. By testing different elements of your email, such as subject lines, opening lines, call to action, tone, length, and timing, you can identify what works best for your target audience and your goals.
Why do people send test emails? ›
Testing your emails will help you fix broken email designs, incorrect or untracked links, choose a subject line with higher open rates, eliminate spelling and grammar errors, broken images or missing alt text, and more.
What is a B testing examples? ›What is A/B testing? A/B testing is a controlled experiment in which you run two different product or website versions simultaneously and see which one performs better. For example, you might run your current sales page against a new version with a section that addresses objections.
What is an example of split testing? ›One of the most famous examples of split testing is the Moz landing page case study done by Conversion Rate Experts. In this case, they tested between a short form (original) and a long-form sales page (improved version).
What is the main problem with a B split testing? ›Split Testing the Wrong Page
One of the biggest problems with A/B testing is testing the wrong pages. It's important to avoid wasting time, resources, and money with pointless split testing.
- Subject Lines are Important. ...
- Use Bullet Points and Highlight Call to Action. ...
- Keep it Short. ...
- Don't Muddle Content. ...
- Be Collegial. ...
- Watch Your Tone. ...
- Avoid Too Many Exclamation Marks and No Emojis. ...
- Avoid Quotes That Could be Offensive to Others.
- Test Period. It might sound like a no-brainer to you, but make sure to always include the test period and exact dates of when the test did run. ...
- A/B Test Variations. ...
- Hypothesis. ...
- Most Important Results. ...
- Relevant Side Analysis. ...
- Predicted Uplift in Revenue or Margin. ...
- Conclusion. ...
- Learnings.
- Add personalization. Use merge tags to personalize your subject lines with each recipient's name or location. ...
- Be descriptive. Sometimes, it's better to be direct and descriptive than trendy. ...
- Keep it short. ...
- Limit punctuation. ...
- Use emojis carefully.
To help me accomplish that task, I distilled the writing advice I've read and received over the years into the four Cs—clear, concise, correct, and compelling. Below are the points I keep in mind for each.
What are the 5 C's of email? ›Emails need to be written as clearly as possible to avoid causing confusion with colleagues, partners or stakeholders. Here are 5 'C's to keep in mind for clear, concise, and competent emails. Complete: State your purpose up front and provide the right amount of information.
What are the 7 C's of email writing? ›- Clear.
- Concise.
- Concrete.
- Correct.
- Coherent.
- Complete.
- Courteous.
What are the 3 essential components of an email structure? ›
In this section, we will examine the three parts that make up a mail message: the header, body, and envelope. But before we do, we must first demonstrate how to run sendmail by hand so that you can see what a message's parts look like.
What are the basic elements of email? ›The email message envelope consists of three items: sender, subject line, and preheader. All three items are visible in the inbox of the user's email application.
What is the most important part of an email? ›The subject line is the most important section of your email. If you use the same line repeatedly, customers will stop opening and reading your emails. You should always try to use a subject line that describes the content of your email.
Which of the following cannot be using email? ›The correct answer is space.
What should you always check for before sending an email? ›- Make sure it's not emotional. “Email is for information and tasks,” Crenshaw said in his course. ...
- Check the subject line and make sure it's on topic and typo-free. The subject line is the most important part of the email. ...
- Check the body for clarity. ...
- Check attachments and links.
- Limited Research before running an A/B test. ...
- Testing changes that are too small. ...
- Stopping the A/B test too early. ...
- Not using segmentation. ...
- Testing wrong elements or not important steps of the funnel. ...
- Not Running a follow-up A/B test.
Letting your tests run long enough will help you be more confident that you're choosing the right winner. We recommend waiting at least 2 hours to determine a winner based on opens, 1 hour to determine a winner based on clicks, and 12 hours to determine a winner based on revenue.
How long should ab testing last? ›For you to get a representative sample and for your data to be accurate, experts recommend that you run your test for a minimum of one to two week. By doing so, you would have covered all the different days which visitors interact with your website.
How effective are follow-up emails? ›It brings the highest reply rate — even about 40% higher comparing to the initial email. Definitely worth the effort. For example, if your initial emails have an average reply rate of 6%, a single follow-up may increase the total average reply rate to 8.5%.
How many contacts do you need on your list to run an ab test? ›To A/B test a sample of your list, you need to have a decently large list size — at least 1,000 contacts. If you have fewer than that in your list, the proportion of your list that you need to A/B test to get statistically significant results gets larger and larger.
What is a minimum test segment size for an email a b split test? ›
To run a valid A/B test, the larger the sample size, the better. As a general guideline, test results are valid when you achieve at least, 30,000 visitors per variant with at least 3,000 conversions on that variant. However, generally even higher numbers are preferred.
Can you tell if your email is being monitored? ›Most server side trackers send their email or website URL along with the email, you can look for it to check for tracking. To detect if your email is being tracked through this method, go to your email service and look for Show Original Message option that shows server side data.
Why are test emails going to spam? ›When sending preview emails, make sure that your test content looks as much like your real campaign as possible. If you use filler words or filler content, then this may trigger the email client's spam filter.
How many times can you send a test email in Mailchimp? ›Free plan users can send up to 12 test emails per campaign, but no more than 24 total test emails in a 24-hour period. Tests can include up to six addresses at once. Paid users can send up to 70 test emails per campaign, but no more than 200 total test emails in a 24-hour period.
What is the difference between split testing and a B testing? ›The term 'split testing' is often used interchangeably with A/B testing. The difference is simply one of emphasis: A/B refers to the two web pages or website variations that are competing against each other. Split refers to the fact that the traffic is equally split between the existing variations.
What is the goal of a B testing? ›An A/B test, also known as a split test, is an experiment for determining which of different variations of an online experience performs better by presenting each version to users at random and analyzing the results.
What is a B testing strategy? ›A/B testing, also known as split testing, refers to a randomized experimentation process wherein two or more versions of a variable (web page, page element, etc.) are shown to different segments of website visitors at the same time to determine which version leaves the maximum impact and drives business metrics.
Why is split testing important? ›Essentially, you can split test everything that your target audience sees from your business, and the more you can do so, the better grip you'll likely have on what engages your audience the most.
How does split testing work? ›Split testing is a solution for carrying out A/B tests. By comparing several versions of your web pages, such as your landing pages or homepages, a split test helps you identify which one has a better conversion rate for your visitors.
What is a B testing quizlet? ›A/B Testing, also known as split testing, is a method of website optimization in which the conversion rates of two versions of a page — version A and version B — are compared to one another using live traffic.
What are the limitations of a B testing? ›
Another limitation of A/B testing is that it requires the tester to extend the results indefinitely into the future. Traditional A/B tests assume an unchanging world view and don't take into account changes in trends and consumer behavior and the impact of seasonal events, for example.
Can you do AB testing on Mailchimp? ›It's easy to conduct A/B testing in Mailchimp. To get started, create an email campaign like you normally would, select A/B Test when prompted, and choose the audience or contact segment you want. You'll then move on to the Variables step, so you can decide what you want to test.
How to do ab testing on google? ›- Go to your Optimize Account (Main menu > Accounts).
- Select your container.
- Click Create experiment.
- Enter an experiment name (up to 255 characters).
- Enter an editor page URL (the web page you'd like to test).
- Click A/B test.
- Click Create.
- Focus on what you're trying to understand about the candidate's knowledge and skills. ...
- With product management roles, focus on how they make decisions. ...
- With data science roles, focus on statistical skills. ...
- Design questions that are relevant to your company's day-to-day.
You can also use A/B Testing with the Notifications composer to test multiple variants on your re-engagement campaign before you roll it out to all users. With an experiment that uses the Notifications composer, you can use an Analytics event to define the goal of your experiment and compare experiment variants.
Can you do AB testing with Google Analytics? ›For landing pages optimization, an A/B test is essential. One of the easiest and most cost effective ways is split testing with Google Analytics A/B testing features. The allows you to use the existing set of data and users behavior to create landing page experiments.
How many test emails can I send Mailchimp? ›Free plan users can send up to 12 test emails per campaign, but no more than 24 total test emails in a 24-hour period. Tests can include up to six addresses at once. Paid users can send up to 70 test emails per campaign, but no more than 200 total test emails in a 24-hour period.
What is an example of a B testing in Google? ›A/B testing is where you test two (or more) variations of a change. For example, you may test different fonts on a button to see if you can increase button clicks.
Is Google AB testing free? ›Engage your website visitors like never before. Create personalized experiences and run website tests — for free.
How does AB testing software work? ›A/B testing—also called split testing or bucket testing—compares the performance of two versions of content to see which one appeals more to visitors/viewers. It tests a control (A) version against a variant (B) version to measure which one is most successful based on your key metrics.