What is best practice to set up and conduct A/B testing on whole emails?

[Anonymous]

My team and I are trying to conduct A/B testing on whole emails and I am having a hard time finding anything definitive that lays out the steps and best practice to do this per program type in Marketo.  

We conducted a whole email test in an email program but when looking through the results it appears the challenger email was not sent. 

If anyone can provide some input or direct me to helpful resources, I would greatly appreciate it. 

Thank you. 

[Josh_Hill13]

Christine,

1. If the challenger wasn't sent, then you need to look at the settings of the Email Test group or contact support. Can you send us a screenshot?
2. For best practices, it is difficult to run a Whole Email test from a statistical standpoint. It is easy to run totally different emails, but it will be hard to draw conclusions about WHY one worked vs. the other. So I suggest to people that when running Whole Emails you only focus on ONE aspect of the email - CTA text, button color, one line, or an image. Otherwise it will be a pointless test.

[Dory_Viscoglio]

Hi Christine, if you're using a batch campaign as opposed to a triggered campaign, then champion/challenger doesn't work. Josh makes some valid points above as to why you might not want to do whole email tests, but assuming you want to do whole email tests then here's the easiest way I've found to do it:

1. Create both of your emails that you want to send -- not within a champion/challenger test.
2. Create the smart campaign to send, with your appropriate filters in the smart list.
3. Under the flow steps of the campaign, add a send email then add choice.
4. Use "If random sample" is 50%, send email version 1 otherwise send email version 2. (Your two emails that were created)

Hope this helps, good luck!

[Traci_Taylor]

Hi Dory. I'm currently testing a shorter and longer version of an email with your 50/50 test method described above. How will I be able to see the results of the two different emails? Specifically clicks and CTR.

Thanks.

[Dory_Viscoglio]

Hey Traci, you can use an email performance report to easily see the results of the emails side by side.