Random Audience Splitting (with A/B Testing)

[dillonlee02]

I have set up our new structure at my company for how to go about A/B Testing when we need to send to a large audience size – essentially, the only way we are able to A/B Test and follow the method for breaking out our total audience into different groups is via Random Sample A/B Testing in a smart campaign. Let me explain my thought process below:

Audience Size generated via smart list is 28,000 people and we find that 1,000 people have the same domain (e.g., @siemens.com) and we don't want to send to those people at the same time due to the possibility of receiving hundreds of soft bounces, which could then hurt our deliverability rates. Now, if we had 9,000 people, then that would be a different story as this is what I'm proposing:

Using an Email program template and this is what the structure would be (with details baked into my reasoning):

Template (Structure):

1. Email program (Parent) – do Email Program (A/B Testing) for audience size of 9,999 people or less (smaller audience size).
1. Smart List – this will be used to generate the audience for 9,999 or less – please see processes below on how to set-up 10,000+ email sends a/b testing vs 9,999- email sends a/b testing
2. 01 Workflows folder – houses all workflows
1. 01a Populate Random Audience (10,000+) – this will only be used to split up the audience sizes – groups need to be in an even number to do proper A/B Testing (e.g., 4 groups, 6 groups, 8 groups, etc.)
2. 01b Send Email 01 (External) – do Random Sample (A/B Testing) for audience size of 10,000 people or more (larger audience size)
3. 01b Send Email 01 (Internal) – will be able to send to our internal stakeholders (marketing seed list such as myself and anyone else within my organization) – the reason we need to have this is because this will give our stakeholders the ability to see their emails goes out without having to worry about communication send limits based on Setup tab unchecking “Block non-operational emails” box. We have dozens of emails going out each week, so we don’t want internal stakeholders not getting the emails we send out based on them reaching their communication send limit, so it only makes sense to have a separate workflow for them. Please note, if doing an A/B Test, please send when winner is expected to be chosen (when doing Email Program A/B) or when final A/B Test runs (when doing Random Sample A/B). We should be able to know the time to send.
4. 01d Invited (Non-Success) – this will be able to mark their program status once the email gets delivered to the recipient. This is helpful to have, especially for when we are sending out of the Email program (Parent) since we can’t mark their program status at the parent-level alone.
5. 99 Clicked CTA (Success) – this will be able to mark their success for clicking the main CTAs within any said email and will check to see if they visited any of the said pages, and then will change their program status to Email > Clicked CTA within the flow step.
3. 02 Emails folder
1. Houses the email assets
4. 03 Lists folder
1. Houses the static list assets needed
2. Notes on each static list:

1.       00 Total Audience (Exported) – this will be used to generate the audience for 10,000 or more (smart list tab will have audience criteria, and people’s tab will have our members). From here, export members into CSV from people’s tab, and then work CSV as such: keep only email addresses, go to Data > Remove Duplicates > leave Email Address checked > Run > Save CSV
2.       01a Invite List – this will be where you import the CSV mentioned above and this will be used in the smart campaign 01a Populate Random Audience (10,000+)’s smart list tab to then break out our random audience groups within this smart campaign’s flow tab

•       02a through 02 etc.

1. This can depend on how big your list is – (e.g., Total Audience = 28,000 will break list into four groups listed from 02a Random Audience (Group 1) all the way to 02d Random Audience (Group 2)).
2. This is important to have these in Even Groups as mentioned earlier because if we plan to do A/B Testing (2 variables being tested) we need to have an even split, or a total group number that is divisible by 2 (e.g., 4 total groups = 2 groups get Test A, and 2 groups get Test B) and these will each have Wait steps of 30 mins in-between one another under the 01b Send Email 01 (External) workflow, found in this smart campaign’s flow step

Other folders:

1. 04 Reports folder – houses all reports needed
2. 99a Misc folder (archived) – houses any workflow fix programs that are out of the ordinary if we ran into any hiccups.
3. 99b NIU folder (archived) – houses any workflows, emails, lists, reports, etc. that are not-in-use any longer.

Now that I explained the structure of the program, let's see screenshots in order:

Step 1: Locate your program

Step 2: Generate your Smart List

Step 3: From here, export members into CSV from people’s tab, and then work CSV as such: keep only email addresses, go to Data > Remove Duplicates > leave Email Address checked > Run > Save CSV

<no screenshot to avoid privacy legal issues>

Step 4: Reimport CSV list into 01 Invite List

Step 5: Locate the 01a Populate Random Audience (10,000+) smart campaign

Step 6: Set up the 01a Populate Random Audience (10,000+) smart campaign

calling the 01 Invite List we just imported into

Set for total campaign exceeds (number greater than your generated audience that is planning to run through this smart campaign (e.g., set to 50,000 with audience of 26,047)

Step 7: Set up to send out of 01b Send Email 01 (External)

May be hard to see the Flow step, but essentially, to avoid soft bounce issues, and trying to do the "Random Sample is 50" A/B Test that Joe Reitz introduced on his YouTube channel, Marketo Fu, it essentially is saying Group 1 gets Test A, Wait 30 minutes, Group 2 gets Test B, Wait 30 minutes, Group 3 gets Test A, Wait 30 minutes, and Group 4 gets Test B.

Then you'd set up your Internal send for your seed list, and set up your triggered campaigns to capture non-success (program status change for Email > Invited in the flow step, but the smart list would listen to see if the email is delivered to them) and success (program status change form Email > Clicked CTA in the flow step, but the smart list would listen to see if the CTA within the email was clicked)

(Notice: And you can see how you'd want to filter out for ghost clicks within the success triggered smart campaign - I have another example where I talked about this here: https://nation.marketo.com/t5/product-discussions/shared-blog-i-want-to-believe-but-your-email-link-...)

And then, you would schedule the emails to go out and let Marketo do it's magic. Hope this was insightful, and could help A/B Test all while avoiding potential soft bounces, which would then in-turn help increase deliverability rates. Happy Marketo!

Best,
Dillon