New Email A/B Test Feature

[Anonymous]

With the new feature... I just want to do one email blast and test all recipients (50/50%) but want to test the whole audience, not have the winning subject line email sent afterwards. (This is soley for testing purposes this run) Would I just drag the testing bar to 100% so they all receive the test and NONE the winner? And if so, would this evenly split the email 50/50%?

And the schedule would be the exact same for send test as send winner?

Thanks!

[Veronica_Holme4]

Hi Kim,

The new email program is not really built for that scenario.

For that scenario personally I'd be tempted to stick to using a default program and working with the old "random sample" flow action technique, as that basically just randomly sends email A to 50% and email B to the other 50% in a single smart campaign and allows you to compare the performance of a single send, with no follow up "winner" send.

[Justin_Cooperm2 · Accepted solution]

If you want to take advantage of the new dashboards, you have a few options:

1. Use the email program's A/B test functionality as you mention in your post and set the test type to be for "Whole Emails." Select the two emails that you want to test and then set the sample size to be 100%. Next, set the metric you are interested in observing (opens, for example) and set to declare the winner manually. You can then schedule when to run the test and when/where to receieve a results report. Approve your program and then wait to collect results. In the future, you can either declare a winner to use moving forward or discontinue the program altogether. 
2. You can use Champion/Challenger functionality to test any email you are sending within Marketo. If you already have a campaign that is sending an email, you can right-click on the email asset and select "New test." From there, set the test type to be "Whole Emails" and select the challenger email you'd like to test the selected email against. Clone that email asset into the test and then set the distribution to be 50%/50%. Next, set the metric you are interested in observing (opens, for example) and schedule when to receive a results report. Approve your test and then wait to collect results. In the future, you can either declare a champion or discard the test altogether. 

Based on your scenario, #2 is probably the best bet. You'll be able to view a nice results dashboard by clicking on the email you are testing!

Justin

[Anonymous]

I think you are describing Champion/Challenger, right? There is no "winner" in this case.

[Anonymous]

Thanks! Yea, I guess I was meaning to do the Champion/Challenger test. This is exciting, love it!

[Anonymous]

My report did not show that I clicked on one of the emails. The dashboard only shows the click when I hover my mouse over the bars of the upper right bar chart.
What have I missed?


Does the program have to be an 'Email' program or it can be a 'Default' program too?



  

[Anonymous]

Regarding Justin's post of 12/18/2013 - we are looking at the same scenario and wanting to use the Email Program's convenient A/B split capability to run a small test where 50% of our entire audience received Subj A and the other 50% received the other but were a little concerned as to whether setting the sample size to 100% would work as intended (since this isn't exactly what it was designed for) so I was glad to see your Option 1 description. However, I'm not sure it actually does work because the test I set up to try it is just sitting there "waiting to run" 15 minutes after it was scheduled to start. Has anyone had success with this?

Regarding Justin's Option 2, According to Marketo's December Customer Training PPT, you can only use champion/challenger in Triggered Campaigns and Engagement Programs. If used in a batch campaign the system only sends the champion (and a test I just ran confirmed this). So Option 2 would NOT work if using a batch campaign to send the email.