Re: Can I declare an A/B test winner in less than 4 hours?

Anonymous
Not applicable

Can I declare an A/B test winner in less than 4 hours?

I want to A/B test the subject lines on my weekly newsletters, but I don't want the test group and remainder group to receive the newsletters 4 hours apart. 

I understand that A/B test needs time to run in order to determine a winner.... but I've found the majority of my contacts open the email within the first hour anyway, can I force the program to declare a winner and send out the reaminder of my emails after just 2 hours? 
Tags (1)
5 REPLIES 5
Dory_Viscoglio
Level 10

Re: Can I declare an A/B test winner in less than 4 hours?

Hi Elicia, from what I've seen, you cannot force an email program to send the winner in less than a 4 hour window. Another option would be to use the champion/challenger test feature to send out a percentage of your emails, and then send the winner to anyone who didn't receive the first email. Because this requires both manual review (to select the winner), and a lot more setup since champion/challenger emails can only be done as triggered emails, I wouldn't suggest this option.

Would it be possible to send as a single batch and use your findings to improve your subject lines week over week?
Matt_Stone2
Level 9

Re: Can I declare an A/B test winner in less than 4 hours?

What I do is I set the test group to 100% and then immediately pick a winner after the four hour window. This almost entirely eliminates any people receiving the newsletter at a later time (though anyone who signs up in that four hour window will get it after the winner is chosen).
Dory_Viscoglio
Level 10

Re: Can I declare an A/B test winner in less than 4 hours?

Matt, I'm surprised that's worked for you. In the past when I've tried to do a 100% send, I've found that only about 75% sends, and Marketo reserves 25% to receive the winner. Granted, I only tried this once, but burned once is more than enough for me. 
Matt_Stone2
Level 9

Re: Can I declare an A/B test winner in less than 4 hours?

Seems to work fine for me--maybe there was a bug when you attempted it? It'd be great to have an employee verify this.
Dory_Viscoglio
Level 10

Re: Can I declare an A/B test winner in less than 4 hours?

Hi Matt, I'd also like a response from Marketo, but I'm not holding my breath. See this thread for more information about my gripes with email programs.