A/B Testing

Anonymous
Not applicable
I am testing my campaign that will be deployed tomorrow morning. We are testing 8 different subject lines. I have set up the flow with 7 choices and the default. I added choice with random sample is 12 and 13 percent for 7 of the email, then the default.

I tested the campaign on my seed list and the first 8 emails went out correctly, sending all 8 versions, but after completing emails 9-15 were all the default email. I need the system to start again at the first email and continue through all 8 versions. 

Any suggestions?


Tags (1)
6 REPLIES 6
Anonymous
Not applicable
Drew's right, you can't specify who gets what, that defeats the purpose fo the sample being random. If instead you'd like it to be precise, then create 8 lists, add the people to get email one to list one, and so on. Then member of list=list one, send email one, etc. And not use random sample at all.

Otherwise it's anybody's game, anyone who qualifies has the ability to recieve any email in the sample.
Anonymous
Not applicable
Hi Kyle,

If you setup the flow I recommended with the 8 different emails as random samples and the default as "Do Nothing", it should properly distribute the 8 different emails over the entire distribution list. However, the Random Sample (as far as I'm aware) doesn't guarantee a linear process like you're imagining, where contact #9 receives email #1 and restarts the order. The Random Sample only guarantees that the emails will be distributed in the proportions you designate.

Drew
Anonymous
Not applicable
Hi Josh,

My campaign consistes for 8 different emails. I am testing the campaign on my seed list, which is 15 email addresses (this is where emails 9-15 come in).

What I had in mind when I set up the random sample choice in the flow step was that the campaign would send every email address a different email (from email #1- email #8). So, once it send the first 8 email addresses in the campaign email #1-email #8, then it would start again and the 9th email address in the campaign would receive email #1, then email address 10 would receive email #2, and so on.

When I originally set up my campaign with:

If Random Sample is 12 Send #1
If Random Sample is 13 Send #2
If Random Sample is 12 Send #3
If Random Sample is 13 Send #4
If Random Sample is 12 Send #5
If Random Sample is 13 Send #6
If Random Sample is 12 Send #7
Default: Send email #8

I thought that was the proper way to do it. And when I sent the campaign to my seed list, the campaign would properly send emails 1-8, but the rest of the email addresses afterwards would only receive the default email #8.

I am a little confused about your suggestion about 8 seperate randomly generated lists? Can you help explain
Anonymous
Not applicable

I'm confused, you said you had 8 emails and honestly either flow setup should work as described above.

Where are emails "9-15" coming into this?

You may want to consider generating 8 separate randomly generated lists from the main target group you plan to use. Then set those up as Send Email choice steps.

Anonymous
Not applicable
Hi Drew,

I set up the random sample as you suggested above. Unfortunately, the program only went though the 8 emails and then stopped. It didn't restart at Email #1 and continue to deploy the campaign. 

Kyle
Anonymous
Not applicable
Hi Kyle,

I think you need to change your Random Sample flow steps a bit. Your choices should all be Random Sample percentages totalling 100, and your default should be "Do Nothing". Something like this:

If Random Sample is 12 Send Email #1
If Random Sample is 13 Send Email #2
If Random Sample is 12 Send Email #3
If Random Sample is 13 Send Email #4
If Random Sample is 12 Send Email #5
If Random Sample is 13 Send Email #6
If Random Sample is 12 Send Email #7
If Random Sample is 13 Send Email #8
Default: Do Nothing