I have set up our new structure at my company for how to go about A/B Testing when we need to send to a large audience size – essentially, the only way we are able to A/B Test and follow the method for breaking out our total audience into different groups is via Random Sample A/B Testing in a smart campaign. Let me explain my thought process below:
Audience Size generated via smart list is 28,000 people and we find that 1,000 people have the same domain (e.g., @siemens.com) and we don't want to send to those people at the same time due to the possibility of receiving hundreds of soft bounces, which could then hurt our deliverability rates. Now, if we had 9,000 people, then that would be a different story as this is what I'm proposing:
Using an Email program template and this is what the structure would be (with details baked into my reasoning):
Now that I explained the structure of the program, let's see screenshots in order:
Step 1: Locate your program
Step 2: Generate your Smart List
Step 3: From here, export members into CSV from people’s tab, and then work CSV as such: keep only email addresses, go to Data > Remove Duplicates > leave Email Address checked > Run > Save CSV
<no screenshot to avoid privacy legal issues>
Step 4: Reimport CSV list into 01 Invite List
Step 5: Locate the 01a Populate Random Audience (10,000+) smart campaign
Step 6: Set up the 01a Populate Random Audience (10,000+) smart campaign
calling the 01 Invite List we just imported into
Set for total campaign exceeds (number greater than your generated audience that is planning to run through this smart campaign (e.g., set to 50,000 with audience of 26,047)
Step 7: Set up to send out of 01b Send Email 01 (External)
May be hard to see the Flow step, but essentially, to avoid soft bounce issues, and trying to do the "Random Sample is 50" A/B Test that Joe Reitz introduced on his YouTube channel, Marketo Fu, it essentially is saying Group 1 gets Test A, Wait 30 minutes, Group 2 gets Test B, Wait 30 minutes, Group 3 gets Test A, Wait 30 minutes, and Group 4 gets Test B.
Then you'd set up your Internal send for your seed list, and set up your triggered campaigns to capture non-success (program status change for Email > Invited in the flow step, but the smart list would listen to see if the email is delivered to them) and success (program status change form Email > Clicked CTA in the flow step, but the smart list would listen to see if the CTA within the email was clicked)
(Notice: And you can see how you'd want to filter out for ghost clicks within the success triggered smart campaign - I have another example where I talked about this here: https://nation.marketo.com/t5/product-discussions/shared-blog-i-want-to-believe-but-your-email-link-...)
And then, you would schedule the emails to go out and let Marketo do it's magic. Hope this was insightful, and could help A/B Test all while avoiding potential soft bounces, which would then in-turn help increase deliverability rates. Happy Marketo!
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.