Create an A/B test Email program

[nishanth27]

Hello folks,

Quick question on AB test set up:

I have set a simple A/B test for whole emails in an Email Send program which is tied to a CRM campaign code. Now, under the same program, can I create another AB test possibly nesting another email program housed under the main Email Send program in any way? I understand an email program can have one one AB test in it. 

Please advise.

Thanks

[ilona_buteneers]

Hi @nishanth27 

Each Email Program can only have one single A/B test. And it is not possible to nest one Email Program in another Email Program. 

What you could do is creating a Default Program and nest 2 separate Email Programs in that Default one. 

 

Another way of working with multiple A/B tests in a Program, is working with Smart Campaigns and the following elements: 

1.  Creating samples and adding the records to a Static List (the filter you use in your Smart List criteria is called "Random Sample")
2. A Smart Campaign to send version A to the first random sample you created (Static List n°1) and to send the B version to the second random sample you created (Static List n°2). Of course you can expand this depending on how many versions you'd like to test. 

As you can see, this requires a little more manual effort. 

Hope this helps you going in the right direction. 

Don't hesitate to reach out in case you have more questions on this. 

Happy to help. 

Kind regards, 

Ilona 

[nishanth27]

Thanks @ilona_buteneers Appreciate the response.

Could you kindly explain a little bit more about the other way of creating Ab tests (with Random Samples) with screenshots if possible. I want to make sure I understood right. Not sure if we can add records to a static list from smart lists

Also, looks like this set up sends versions of emails to sample percentage of the audience but the outcome of one the tests will be the winner right. So how is this set up going to send the winner to the rest of the audience?

Winner email has to be manually determined and launched?

Note: my target audience is inside a smart list

Regards,

Nishanth

[ilona_buteneers]

Hi @nishanth27 

Please see attached a screen recording in the Word document of different steps to take. (I was not able to upload video directly into this message). In this set-up I created a separate Smart Campaign to send out the emails, but I believe you can also do everything in one Smart Campaign (creating the samples by adding them to a list - include a wait step - send out the email). 

After you ran the manual A/B test, you also would have to create Email Performance Report to define the winner. 

You can then send the winner to the remainder of the audience and make sure to exclude the samples you created. 

Hope this helps! 

Kind regards, 

Ilona 

[nishanth27]

@ilona_buteneers Thanks! I'll take a look at the document you have shared that outlines the steps.

Should I face any issues, I'll let you know.

Thank you so much for this. Happy Holidays in advance 🙂

Best

Nishanth

[nishanth27]

@ilona_buteneers I am trying to understand why static lists are required. I have created two smart lists with random sample filters like you suggested, then I am creating two batch campaigns to send out emails for smart list samples 1 and 2. And manually I am going to review the performance report for both of these emails and launch the winner email by excluding the members of sample list samples.

Please let me know if there is anything wrong. Totally appreciate the response.

Best

Nishanth

[ilona_buteneers]

Hi @nishanth27 

Valid point! My use case was a little different. I used this way of running an A/B test where I had to abort the initial campaign. 

This was because my A/B test that I configured on the Program Level only had sent out one version of the email in the A/B test. So for this reason I created a separate static list so I could send version B to that newly created list. 

But for your approach I believe you can indeed skip that step. 

If you have other questions or issues , don't hesitate to reach out. 

Kind regards

Ilona 

[nishanth27]

@ilona_buteneers Thanks!