Date/time A/B testing question

[JC100]

Hi folks,

I'd like to send an email to 50% of the audience on Feb 23, and the rest 50% on Feb 24 to test out which date works better. 

So, there will be a "winner", but nobody will received it as it's a date rather than an email.

Will the following settings do that for me? Or, will it send the email out to 25% on Feb 23 and 25% on 24, and leave the rest 50% of audience receiving nothing but waiting for my mannual "winner"?

Thanks!

[Darshil_Shah1 · Accepted solution]

This setting will send test emails out to 50% of the qualified people. The different variations will send to equal portions of the chosen Test Sample Size (i.e., each of two the test variants will be sent out to nearly 25% of the total qualified people). In the end, post the test, 50% of the total qualified people would be left to receive the winning email (which will be sent once you declare the winner and schedule the winning email send).

[JC100]

Thanks Darshil,

We eventually used 2 campaigns - one on feb 23 and another on feb 24. We used the Random Sample filter to get 50% audience for 23 feb batch, then used "Not Was Sent Email" to screen the rest 50% audience. So this is an A/B test without using the built-in A/B test. 

Just curious: what if I set the test to 100% audience? 50% go on feb 23 and another 50% go on feb 24? This will match my test need but 100% of test seems not recommended

[Darshil_Shah1]

If you select the test sample size to 100%, Marketo would try to send both test versions (one on the 23rd and the other on the 24th) to equal portions of the chosen test sample size. Also, since you'd have run your test on 100% of the audience, you would not be left with anyone to send the winner email to, which is why Marketo recommends users to not select 100% in the test sample size.

[ChristineT]

When testing what time of the day the email performs best, is there a reason for not using samlpe size 100%. Is there another way you would recommend to thest what time of the day the email performs best?

Thank you in advance for your reply.

[Darshil_Shah1]

Marketo recommends users to not set the test sample size to 100% while performing the AB test as there's a high chance that your winning email will not go out to anyone once the test period is over. If you’re using a static list as the audience criteria, setting the sample size to 100% sends the test emails to everyone in the audience and the winner goes to no one. If you’re using a smart list, setting the sample size to 100% sends the email to everyone in the audience at that time. When the email program runs again at a later date (once the test period is completed), any new people who qualify for the smart list will also receive the email since they’re now included the the period is over. It is due to this uncertainty paired with not being able to send the tested and better-performing email out to the audience, Marketo (and many of its users) doesn't recommend setting the AB test sample size to 100%. You could also use the random sample choice step condition to send all versions of emails to equal portions of the audience instead of using the email program.