We regularly send out a "newsletter" to around 50.000 persons and we would like to start doing A/B testing on the Subject Line. As always what seems to be a simple task ends up taking a bit longer when planning the actual setup 🙂
Our need is to do a split test on around 5.000 persons. After around 24 hours a winner should be declared and we need to split our winner send to the remaining 45.000 persons into two equal parts and send them out with around 4 hours between them so our customer service doesn't get to many calls at once.
The Marketo A/B testing on Newsletters only supports the winner being sent to the remaining 45.000 at once.
We are looking into using the Champion/Challenger function instead but I'm not sure if that is a recommended solution since it only works with trigger based SC's. It's also not ideal since I would prefer the winner declaration being automatic but I'm afraid that will not be possible with our needs.
Our current plan is using a default program with 1 email, three static lists and 4 smart campaigns. The setup is:
...
EMAIL
Email with a Champion/Challenger subject line test
LISTS
a) A/B Test List
b) Winner List 1st half
c) Winner List 2nd half
SMART CAMPAIGNS
Smart Campaign 01
Smart List: Our Main Target List
Flow: Add to list using random sample 10% to "A/B Test List", 45% to "Winner List 1st half" and default choice "Winner List 2nd half"
Smart Campaign 02
Smart List: Member of List "A/B Test List"
Flow: Request Campaign "Smart Campaign 03"
Smart Campaign 03
Smart List: Campaign is Requested
Flow: Send Email "Newsletter"
Smart Campaign 04
Smart List: Member of Lists "Winner List 1st half" or "Winner List 2nd half"
Flow: Send Email If Member of List "Winner List 1st half" > Wait step 4 hours > Send Email If Member of List "Winner List 2nd half"
...
We will then need to declare the winner of the Champion/Challenger test manually after around 24 hours and run Smart Campaign 04 afterwards.
So my questions are:
1) Would this setup work?
2) Is running a batch of 5.000 persons through a Campaign is Requested trigger an issue in terms of system load and execution time?
3) Is there a better/simpler way to achieve what we need?
Any thoughts are appreciated - thanks
.png)