How long should the test duration for an a/b test be?

[Anonymous]

I have a email going out tomo to 20K ppl with an a/b test on the subject line.

I really don't want to wait too long to send out the winner bc I want it to go out tomorrow, not another day (it's related to a launch).

How long does Marketo wait before it declares an email has not been opened? I suppose that 1 hour might not be enough?

Thus, is 1 hour enough to reach the sample size of 10%? 

Thanks!
 

[Josh_Hill13]

I doubt 1 hour is enough. A good test would be 24-48 hours. I think marketo just waits the allotted time to declare a winner when you set it automatically. Whoever opens, opens.

[Kim_Allen]

I agree with Josh, 1 hour isn't enough time. However, I know I get a lot of pushback from the team if I don't have the sends all in one day. So I usually have the test send early in the morning and give it a good 6 hours or so before the automatic winner is sent.

[Dory_Viscoglio · Accepted solution]

It might have changed, but I believe the minimum time for the automatic winner email program sends was 4 hours at one point. If all of your recipients are in the US then Kim's suggestion of waiting about 6 hours should be good. Otherwise, you're not necessarily going to have a very representative sample of opens, and your final send might not go at a very desirable time which could impact the overall outcome. At that point, I'd probably wait 24 hours. 

[Anonymous]

Thanks everyone, helpful stuff!

[Anonymous]

Hi, so there is no possible way to declare a winner after 30 minutes?

[Dory_Viscoglio]

Not if you're doing it through an email program. You can set it up manually and do it after 30 minutes, but I would advise against it. Just as food for thought, on Friday, 4 hours after an email send I saw a ~20% open rate (11am ET send). As of this morning it was 39%. Obviously this could have had some regionality associated with it, but 30 minutes won't always be very indicative of actual results.

[Anonymous]

I agree with Dory - 30 minutes is definitely not enough.  My team usually likes the same day as well and our data shows our list opens our emails within a few hours. So I set our tests to run 4-6 hours.  Dory is also correct, Marketo doesn't let the test run for less than 4 hours.