Can you 50/50 test a whole email?

[Nick_Zhao]

@Alex Baumgarten that is an important point. For emails that are timely in which you don't want a winner to be sent hours or days after the original test is sent, and use the 100% test within the A/B testing function in Marketo, make sure you declare the winner manually. 

Otherwise you will be wondering, like I had, why there was a winner email sent. It's not just for duplicate contacts but anyone that qualifies that wasn't sent the email (e.g: reached a communication limit that day) 

[Steven_Vanderb3]

Highly recommend you DON'T do this - in fact, we plan to change how A/B Tests configuration works to not allow this! Setting the test sample size to 100% can cause confusing (and sometimes embarrassing) results after the A/B test is completed and guard rails will be added to prevent this.

If the desire is to randomly split one audience between multiple emails, this can be achieved with a Smart Campaign using the random sample constraint.  A/B tests are intended to be used to test multiple versions of an email with a subset of an audience and then send the best performing email to the remainder of the audience.

[Andrea_Fiorenti]

Hi Steven, thanks for your suggestion but I find it a bit generic. Can you please be more specific about the confusing and/or embarrassing results you mentioned?

[Natascia_Almeid]

Usually in my team we used to create two different programs for making A/B Testing:

Program 1: Sends version A

Program 2: Sends version B

This would be easily displayed on the reporting.

[Nithish_Bhat]

Natascia Almeida​ What about the list/audience? are they same or do you split it ?

[Natascia_Almeid]

We split it

[Natalie_Kremer]

We do this often by just using a Smart Campaign to send the email. We determine the audience in the Smart List like we normally would, but within the Flow step, we choose Send Email and Add Choice. The criteria would be Random Sample is 50 which would send your Email A and then you'd set Email B as the default email to send. After you send you can then run your Email Performance Report to look at the different metrics and determine which performed better. I find this gives you more control than the out of the box A/B test.

Here's an example of the Flow:

[Tiffany_Scott]

This was one of the best tips I've ever seen on here- thanks a lot @Natalie_Kremer !

Would you recommend doing this/ will it even work within an engagement program that uses programs instead of emails to send an email? I fear it may be too taxing on the system if there are too many programs that would need to call for this random sample.