Understanding Date/Time A/B Testing

[Karthick77]

For date/time A/B testing, the process of determining the winner needs to be done manually, as there is no automatic option available. After analyzing the results, the winner must be declared, and the winning email sent accordingly.

However, if there is no intention to declare a winner and simply send emails at different times, utilizing a 100% sample size can achieve this. It ensures delivery at two different times without the need to set specific winner criteria.

This is Not recommended when we are doing for winner criteria.

Is my understanding correct?

 Anyone, please help me to understand the following concept:

Marketo recommends users to not set the test sample size to 100% while performing the AB test as there's a high chance that your winning email will not go out to anyone once the test period is over. If you’re using a smart list, setting the sample size to 100% sends the email to everyone in the audience at that time. When the email program runs again at a later date (once the test period is completed), any new people who qualify for the smart list will also receive the email since they’re now included the the period is over. It is due to this uncertainty paired with not being able to send the tested and better-performing email out to the audience, Marketo (and many of its users) doesn't recommend setting the AB test sample size to 100%. You could also use the random sample choice step condition to send all versions of emails to equal portions of the audience instead of using the email program.

Thank you

[uditmathur · Accepted solution]

Hi,

All Marketo AB test are designed to compare different version of your email/LP may it be the subject, Date & Time etc.

Date & Time AB test is meant to get the best time to send email campaign so that you get the maximum response for your campaign.

If you set 100% as the sample size your entire list will be sent with your mail at different time & you will not be able to get the best suited time which might impact your ROI.

Although it always suggested to have the best/winning option be used for your majority clients, but YES if your business use case does not require to declare winner & just want to send email at different times to random selection of your list you can try using 100% sample size.

[Karthick77]

Thank you @uditmathur