Email – A/B Testing

Overview

With A/B Testing, you can test which subject line or email content gives you the highest open and click-thru rate and send the winning content to your customers. This allows you to achieve a higher conversion and keep your leads and customers engaged. To leverage A/B testing, determine the number of splits or versions of email you would like to send to a test audience. Then define the size of the test audience and the winning metric. Once the test email is sent to the test audience and the winning threshold is reached, the winning content will be sent to the rest of your mailing list.

Here’s some helpful tips about setting up an A/B test:

The Deployment Graph report (available within the deployment delivery report) will provide excellent benchmarks to determine the time interval for the open and click activities to help determine the optimal times for determining the winner.

A/B Testing Tips

Setting Up A/B Testing

1. When you create a deployment, in the Deployment Summary pleat, check the A/B Testing checkbox. If you use A/B Testing, you may not use Triggered deployments, On Demand Send API or Dynamic Content for the same deployment. NOTE: The A/B Testing checkbox must be checked prior to saving the deployment summary or you will be unable to use this functionality on this mailing.

2. In the Audience pleat, select the number of splits you want to create- this is for the A, the B, the C test etc. Each split will be for a different test audience. Notice you can no longer define the distribution of your recipients as your test audience will be evenly distributed across the splits.

3. In the Audience pleat, enter the total Test Audience Size and select whether the split wins by the most unique opens or the most unique clicks. Select the number of hours you want the test to run for or leave the default one hour. Once you are done, click the Save Audience Settings button on the upper right corner of the pleat.

4. Good reference tools can be found here to assist you with determining the optimal test list size: https://returnpath.com/sample-size-calculator/   or  https://www.surveysystem.com/sscalc.htm#one

5. Set up the email content, test and approve your email as you normally would. Once the email is sent and the winning threshold has been met, we will then send the email with the winning content to the rest of your recipients.

6. The Modification History pleat of the sample send will show the results and which was selected as the ‘winner’.

7. When you review your email results, you will notice two deployments: the main deployment that went to most of your audience and the sample or test deployment that had the A/B split and was sent to your test audience. The sample deployment will have “S” appended to the end of the tracking Id and “-sample” appended to the deployment name.