SOLVED

Re: How long should the test duration for an a/b test be?

Go to solution
Anonymous
Not applicable

How long should the test duration for an a/b test be?

I have a email going out tomo to 20K ppl with an a/b test on the subject line.

I really don't want to wait too long to send out the winner bc I want it to go out tomorrow, not another day (it's related to a launch).

How long does Marketo wait before it declares an email has not been opened? I suppose that 1 hour might not be enough?

Thus, is 1 hour enough to reach the sample size of 10%? 

Thanks!
 
Tags (1)
1 ACCEPTED SOLUTION

Accepted Solutions
Dory_Viscoglio
Level 10

Re: How long should the test duration for an a/b test be?

It might have changed, but I believe the minimum time for the automatic winner email program sends was 4 hours at one point. If all of your recipients are in the US then Kim's suggestion of waiting about 6 hours should be good. Otherwise, you're not necessarily going to have a very representative sample of opens, and your final send might not go at a very desirable time which could impact the overall outcome. At that point, I'd probably wait 24 hours. 

View solution in original post

7 REPLIES 7
Josh_Hill13
Level 10 - Champion Alumni

Re: How long should the test duration for an a/b test be?

I doubt 1 hour is enough. A good test would be 24-48 hours. I think marketo just waits the allotted time to declare a winner when you set it automatically. Whoever opens, opens.
Kim_Allen
Level 10

Re: How long should the test duration for an a/b test be?

I agree with Josh, 1 hour isn't enough time. However, I know I get a lot of pushback from the team if I don't have the sends all in one day. So I usually have the test send early in the morning and give it a good 6 hours or so before the automatic winner is sent.
Dory_Viscoglio
Level 10

Re: How long should the test duration for an a/b test be?

It might have changed, but I believe the minimum time for the automatic winner email program sends was 4 hours at one point. If all of your recipients are in the US then Kim's suggestion of waiting about 6 hours should be good. Otherwise, you're not necessarily going to have a very representative sample of opens, and your final send might not go at a very desirable time which could impact the overall outcome. At that point, I'd probably wait 24 hours. 
Anonymous
Not applicable

Re: How long should the test duration for an a/b test be?

Thanks everyone, helpful stuff!
Anonymous
Not applicable

Re: How long should the test duration for an a/b test be?

Hi, so there is no possible way to declare a winner after 30 minutes?

Dory_Viscoglio
Level 10

Re: How long should the test duration for an a/b test be?

Not if you're doing it through an email program. You can set it up manually and do it after 30 minutes, but I would advise against it. Just as food for thought, on Friday, 4 hours after an email send I saw a ~20% open rate (11am ET send). As of this morning it was 39%. Obviously this could have had some regionality associated with it, but 30 minutes won't always be very indicative of actual results.

Anonymous
Not applicable

Re: How long should the test duration for an a/b test be?

I agree with Dory - 30 minutes is definitely not enough.  My team usually likes the same day as well and our data shows our list opens our emails within a few hours. So I set our tests to run 4-6 hours.  Dory is also correct, Marketo doesn't let the test run for less than 4 hours.