It’s Time For A/B Testing: 10 Reasons Why You Should Start Now
It’s Time For A/B Testing: How To Set Your Campaign Right
A/B testing is a method of comparing two versions of an email to see which one performs better on a specific metric, such as Open Rate, Click Rate or Conversion Rate.
First of all, we have prepared a list of the most important features of A/B Testing in Email Marketing:
- Compares the behaviour of your target audience
- Validates the idea that converts best
- Helps optimize the campaign
- Influences and makes decision-making easier
- Increases conversion
What can you test with A/B Testing?
- Email topics
- Offers
- Images
- Content
- Call-to-action
- Link positioning
- Date and time of sending campaigns
Start by defining objectives
To begin with, it is very important to define the goal of your A/B Testing campaign.
For example, objectives can take many forms:
- registrations for an event or course
- purchase of a product or service
- downloads of materials
- a calendar booking
- viewing the landing page
A/B Testing – example applied to product sales
Let’s say you’re the owner of an online store that sells clothing and you want to test two different offers to see which converts better.
We propose to create two versions, with different content for testing.
| Variant A Price reduction offer | Variant B Free transport offer |
| Subject: “20% Discount on the Entire Autumn Collection” | Subject: “Free Shipping For Any Order” |
| Content: Include an image of a few items from the fall collection, highlights their quality and style, and offers a 20% off code for your first purchase. | Content: Include attractive images of your most popular products and emphasizes that anyone who places an order will receive free shipping without the need for a discount code. |
How do you set up your A/B Testing campaign correctly?
Choose one part of your subscriber list to send version A and another part to send version B.
Make sure that these two groups are demographically or behaviourally comparable..
Suppose you have a list of subscribers to your online clothing store and this list includes people from different geographic regions and age groups. In order to conduct A/B testing for your offerings, you should make sure that group A and group B have a similar distribution in terms of these demographic characteristics. More applied: Demographics: If the majority of your subscribers are women between the ages of 25 and 35, make sure that both Group A and Group B have a similar proportion of women in this age group. Behavioral: If you have subscribers who are frequent buyers and others who buy less often, try to ensure that both groups contain a similar proportion of each type of customer. |
The main idea is to eliminate demographic or behavioural variables that might influence the results in undesirable ways.
In this way, you can more accurately assess the impact of your offer on their behaviour, knowing that the differences found are more likely to be related to the offer, not other variable factors.
Send variant A to the first part of the list and variant B to the other part, at the same time and on the same day, to minimise differences in timing.
How do you segment lists in the NewsMAN platform?
The NewsMAN email marketing platform has advanced segmentation features, so defining the audience for A/B Testing can be done easily.
Following the example above, you can segment your list according to several criteria:
- gender
- city
- age
- product preferences
- average order value
- many others
The NewsMAN platform also allows you to segment your subscriber list based on their actions in the newsletter or on the website.
- page views/product categories
- click actions
- “buy” or “add to cart” actions
- newsletter subscribers in the last month
- etc.
Depending on the subscriber list data you have available, you can start segmentation for A/B Testing.
| 1. Fixed subscriber properties Subscriber ID Domain Location by IP Country by IP Region by IP Name First name Stars 1-5 Last activity Last click Browser Device Operating System | 2. Custom variables: first name, last name, date of birth, gender, age, etc. Date of subscription Subscription method By open rate percentage/total By newsletter click rate/percent total Clicked on link X Is/is not in the segments Received / did not receive newsletter X Opened / did not open newsletter X Remarketing Filled out form X |
To create a segment in the list, go to → Segmente → Create segment → Name segment (Segment name) → Choose one or more rules from Segment rules → Click Add, to add the condition → Click Save. → After clicking Save, the subscriber upload operation will run. → When the segment has loaded, the wheel showing that the process is running will disappear. |
We have prepared several video-tutorials for segmentation.
Determine how you will measure results
The next important step in running the right test campaign is to define the metrics you will track and analyze based on the test results.
Below, we’ve identified the most important metrics you might want to track.
- email opens
- clicks on certain links or buttons
- conversion of traffic from the email campaign
How do you determine which is A and which is B?
Which is which?
Well, if you fail to define from the start what you want to look at in version A and vice versa, you will have trouble defining the result.
You need to get it right from the start what the purpose of each variant will be.
Version A is the control version, the one that keeps the original email.
Version B is the variation, i.e. the email with one or more changes based on the variable you set out to test.
For example, if you want to see how your subject line performs, you can adjust the message in the subject line with a sales-oriented call to action.
| version A | version B |
| [Valid today only] -50% On Your Favourite Products | Discount 50% On Your Favourite Products |
Make sure the two versions are identical, except for the variable you’ve set out to test.
Create step-by-step A/B Testing campaign
We’ve prepared a step-by-step guide to creating your A/B Testing campaign.
- In the NewsMAN platform, go to Newsletters > Drafts > click on the details button corresponding to the desired newsletter
- Select the segment/list or segments to which the email will be sent
- Schedule the email to be sent – now or later
- Uncheck “Delete draft after creating the newsletter”, then click on “Send newsletter to selected segments / to the whole list”.
- Click on “Add A/B test version”
- This will open the draft edit page, where you can change one of the items mentioned above.
- Click on “Continue”.
- The newly created draft will be sent to subscribers of the basic newsletter.
- After clicking on “Send newsletter to selected segments / to the whole list”, the basic newsletter or the parent newsletter must be confirmed. To do this, you have to click on the name of the parent newsletter in “This newsletter is a variant of [parent newsletter name]”.
- Once you have clicked on the newsletter name, click on the Confirm button.
For more details about setting up your campaign in the platform, see the guide here.
How do you measure the results of your A/B Testing campaign?
For A/B Testing, you can track the most relevant metrics:
- Open rate
- Click rate
- Conversions from email traffic (we also track sales generated from email marketing campaigns, if you have an integration in your shop with NewsMAN)
For each email campaign, we give you detailed reports on open rate, clicks, bounces, complaints and unsubscribes.
What’s more, all reports can be exported to Excel for more advanced data analysis.
Track, analyse and optimise the impact of your email marketing campaigns
NewsMAN’s analytics and reporting tools help you maximize campaign results. NewsMAN integrates with several software you already use, such as Shopify and Google Analytics.