A/B Split Jobs

A/B testing (also known as split testing) is a randomized experimentation process wherein two or more versions of a variable element are presented to different segments of a larger group to determine which of the versions yields the best results.

Translated to the case of an email marketing campaign in LISTSERV Maestro, A/B testing means that you distribute the delivery of two or more variants of a mail job to randomly created segments of your set of recipients.

In LISTSERV Maestro, A/B-testing starts with creating an A/B Split Job. Completing the workflow of an A/B Split Job works in a fashion that is very similar to what you are accustomed to from a Standard Mail Job: You start with defining the recipients, and then you proceed with defining the remaining elements.

A/B-Test Content Variants: The most common A/B-testing is based on variable content, where you define variants based on different subject lines and/or different mail body content.
A/B-Test Sender Definition: You can also A/B-test the sender definition, to see which version of your "From:" information gains the most confidence with your recipients.
A/B-Test Delivery Settings / Schedule: Since the delivery day/time is known to affect email marketing success, LISTSERV Maestro allows you to define varying delivery settings for each of the variants.

Just as with the workflow of a Standard Mail Job, once you have completed all the steps on the workflow screen, your A/B Split Job is ready for authorization.

On the Authorize Delivery screen, you make the decision whether you want to use Sampling Delivery or not:

A/B Split With Sampling

The main reason to do A/B testing is that you are not 100% sure which version of your message is the best. Without A/B testing, you may be forced to guess, which bears the risk of sending the less-than-optimal version to a large group of recipients. To avoid this, you can use LISTSERV Maestro's A/B Split Sampling technology, which allows you to postpone delivery to the majority of your recipients until you have collected enough insights from looking at the tracking results for the sampling subset.

Delivery to the Sampling Subset: Authorizing the delivery of an A/B Split Job with sampling only schedules the delivery to the sampling subset. You will therefore find the job in the job list under "Ongoing Jobs" even if the sampling delivery has already been completed (according to the settings you supplied on the Delivery Settings screen).
Delivery to the Remaining Recipients: This is the crucial part of A/B Split Jobs with sampling. The major part of your recipients (per your input during delivery authorization) did not yet receive any version of the message. To authorize this part of the delivery (and to decide which version of the message they receive and when), select the job in the job list and select Job Delivery Authorize Remaining Recipients... from the menu. The screen you see then shows metric values and content previews that assist you with the decision.

A/B Split Without Sampling

While delivery to the full set of recipients when using A/B Split Sampling is divided in two parts (or more, depending on your input when authorizing the remaining recipients), delivery for an A/B Split Job without sampling is always performed utilizing the whole set of recipients and distributing the variant jobs randomly and evenly.

Such an A/B test also allows you to later compare results of the delivered variants and learn which of the variants was better, but there can be good reasons to not retain delivery for any of the recipients:

Message Urgency: Due to outer circumstances it may be necessary that the message goes out quickly to all recipients without imposing the delay that comes naturally with A/B sampling.
Variant Quality: Previous tests or your experience may tell you that the variants of the A/B test are not too different from one another and that you don't expect truly bad results with any of them.
Number of Recipients: Your set of recipients is very small at least currently. You may for example plan to use A/B testing for a long-running and repeated delivery to a Subscriber List that you have set up recently and that you expect to grow in the future. In such a situation, a reduced subset of your recipients often is simply too small to yield statistically relevant results.