A/B testing in engagement program--choosing random sample

[Anonymous]

Hi all, 

We have discovered the A/B test option in an email within an engagement program. We understand that subject line can be tested; champion/challenger. But how do we chose the random sample of the members of the engagement program?

How would I determine the send date of the A/B test? The random sample of our audience?

Any advice would be great! Thank you for your thoughtful answers! 

[Anonymous]

Hi Jessica -
The engagement program handles all of that for you. It will send the A/B test on the day of the next cast, whichever date/time is determined by the engagement program. The random sample is additionally chosen by A/B test, so no need to worry about that aspect either. Just say how many get the champ & how many get the challenger, the engagement program handles the rest.

[Anonymous]

Thank you Adam! So I should trust the A/B test to choose the sample size.

Next question: if the test is sent the day of the next cast, does the engagement program gather the 'winner' data in the same day and then send out to the remaining list, the same day? 

It just seems like the test should run a day before the actual cast...

Thank you for your thoughtful response!

[Anonymous]

No worries. So the test doesn't work the same way as the blast test does. With a blast the test goes out, waits for several hours and then you can declare when to send the winner. The Engagement test is more of an ongoing knowledge share thing, where it shows you the results over time over multiple casts, continually testing until you declare the test over. So right now I'm typically testing subject lines in my nurturing emails for 2 months before declaring a winner and starting a new test.

[Anonymous]

 
Hi Adam,

You are brilliant; our team thinks you're great!

How did you learn so much about A/B testing for engagement programs? 

Is there some literature we can read? The first time we heard about this feature was at the 2014 Summit...

Jessica

[Anonymous]

Brilliant is a bit of a stretch. I just listen well to what others have to say. 🙂 As far as documentation, I'm not sure, do some searching in the community. How did I learn? I attended a webinar about these topics.