Can you 50/50 test a whole email?

[Michaela_Morale]

I want to set up an A/B test for an email send. I have 2 versions of the email. I want to send it 50/50 to everyone. Can this be set up through the A/B test option? I don't want the winner to be sent out since I want it to all go out at one time and then see which is possible.

[Natalie_Kremer]

We do this often by just using a Smart Campaign to send the email. We determine the audience in the Smart List like we normally would, but within the Flow step, we choose Send Email and Add Choice. The criteria would be Random Sample is 50 which would send your Email A and then you'd set Email B as the default email to send. After you send you can then run your Email Performance Report to look at the different metrics and determine which performed better. I find this gives you more control than the out of the box A/B test.

Here's an example of the Flow:

[Tiffany_Scott]

This was one of the best tips I've ever seen on here- thanks a lot @Natalie_Kremer !

Would you recommend doing this/ will it even work within an engagement program that uses programs instead of emails to send an email? I fear it may be too taxing on the system if there are too many programs that would need to call for this random sample.  

[Natascia_Almeid]

Usually in my team we used to create two different programs for making A/B Testing:

Program 1: Sends version A

Program 2: Sends version B

This would be easily displayed on the reporting.

[Nithish_Bhat]

Natascia Almeida​ What about the list/audience? are they same or do you split it ?

[Natascia_Almeid]

We split it

[Valerie_Whitin2]

Hi Michaela Morales​

You would set the test sample size to 100% for it to work that way. That means it will evenly split your test among 100% of the audience you selected. Which would in turn be a true 50/50 split. It will still make you select a time to "send the winner" but really nothing will happen other than you will get an email with the results of the test. This is how I've almost always run my tests and have never had any issues. I hope this helps answer your question

[Steven_Vanderb3]

Highly recommend you DON'T do this - in fact, we plan to change how A/B Tests configuration works to not allow this! Setting the test sample size to 100% can cause confusing (and sometimes embarrassing) results after the A/B test is completed and guard rails will be added to prevent this.

If the desire is to randomly split one audience between multiple emails, this can be achieved with a Smart Campaign using the random sample constraint.  A/B tests are intended to be used to test multiple versions of an email with a subset of an audience and then send the best performing email to the remainder of the audience.

[Andrea_Fiorenti]

Hi Steven, thanks for your suggestion but I find it a bit generic. Can you please be more specific about the confusing and/or embarrassing results you mentioned?

[Alex_Baumgarten]

Be careful doing a 100% test. If you have a clean database with no duplicates you should have no issues. But, because of how Marketo handles A/B tests, the winning email may be sent out to duplicate contacts once as a test and then again once a winner is declared. You could avoid this by making the winner declaration manual rather than automatic.

[Nick_Zhao]

@Alex Baumgarten that is an important point. For emails that are timely in which you don't want a winner to be sent hours or days after the original test is sent, and use the 100% test within the A/B testing function in Marketo, make sure you declare the winner manually. 

Otherwise you will be wondering, like I had, why there was a winner email sent. It's not just for duplicate contacts but anyone that qualifies that wasn't sent the email (e.g: reached a communication limit that day) 

[Ankit_Dua]

You can send the email via Random sample 50/50, by this the email will be sent to all the people who qualifies for the campaigns and you can run a report to check which email performed better.

You can do it via A/B testing too, as you have shared in the snapshot.

[Natascia_Almeid]

Hi Valerie,

great idea! But can you also track which version "wins"?

Thanks

[Valerie_Whitin2]

Hi Natascia Almeida​ - yes, it will still let you know which one wins. You will get the normal email program dashboard results with the A/B test results broken out.

[Nithish_Bhat]

you can add a new recipient in the list, after the test completes. Winning mail goes to the new recipient.

[Michaela_Morale]

Yes it does, thank you Valerie!

[Josh_Hill13]

Sure.

Just set the Sample to 50/50 and there will be no winner sent. Do you mean something else?

Probably best to use Random Sample method.

[Michaela_Morale]

I guess I'm unclear how to do that since the slider options are Receives Test and Receives Winner