A/B Testing

[Anonymous]

I am testing my campaign that will be deployed tomorrow morning. We are testing 8 different subject lines. I have set up the flow with 7 choices and the default. I added choice with random sample is 12 and 13 percent for 7 of the email, then the default.

I tested the campaign on my seed list and the first 8 emails went out correctly, sending all 8 versions, but after completing emails 9-15 were all the default email. I need the system to start again at the first email and continue through all 8 versions. 

Any suggestions?

[Anonymous]

Hi Kyle,

I think you need to change your Random Sample flow steps a bit. Your choices should all be Random Sample percentages totalling 100, and your default should be "Do Nothing". Something like this:

If Random Sample is 12 Send Email #1
If Random Sample is 13 Send Email #2
If Random Sample is 12 Send Email #3
If Random Sample is 13 Send Email #4
If Random Sample is 12 Send Email #5
If Random Sample is 13 Send Email #6
If Random Sample is 12 Send Email #7
If Random Sample is 13 Send Email #8
Default: Do Nothing

[Anonymous]

Hi Drew,

I set up the random sample as you suggested above. Unfortunately, the program only went though the 8 emails and then stopped. It didn't restart at Email #1 and continue to deploy the campaign. 

Kyle

[Anonymous]

I'm confused, you said you had 8 emails and honestly either flow setup should work as described above.

Where are emails "9-15" coming into this?

You may want to consider generating 8 separate randomly generated lists from the main target group you plan to use. Then set those up as Send Email choice steps.

[Anonymous]

Hi Josh,

My campaign consistes for 8 different emails. I am testing the campaign on my seed list, which is 15 email addresses (this is where emails 9-15 come in).

What I had in mind when I set up the random sample choice in the flow step was that the campaign would send every email address a different email (from email #1- email #8). So, once it send the first 8 email addresses in the campaign email #1-email #8, then it would start again and the 9th email address in the campaign would receive email #1, then email address 10 would receive email #2, and so on.

When I originally set up my campaign with:

If Random Sample is 12 Send #1
If Random Sample is 13 Send #2
If Random Sample is 12 Send #3
If Random Sample is 13 Send #4
If Random Sample is 12 Send #5
If Random Sample is 13 Send #6
If Random Sample is 12 Send #7
Default: Send email #8

I thought that was the proper way to do it. And when I sent the campaign to my seed list, the campaign would properly send emails 1-8, but the rest of the email addresses afterwards would only receive the default email #8.

I am a little confused about your suggestion about 8 seperate randomly generated lists? Can you help explain

[Anonymous]

Hi Kyle,

If you setup the flow I recommended with the 8 different emails as random samples and the default as "Do Nothing", it should properly distribute the 8 different emails over the entire distribution list. However, the Random Sample (as far as I'm aware) doesn't guarantee a linear process like you're imagining, where contact #9 receives email #1 and restarts the order. The Random Sample only guarantees that the emails will be distributed in the proportions you designate.

Drew

[Anonymous]

Drew's right, you can't specify who gets what, that defeats the purpose fo the sample being random. If instead you'd like it to be precise, then create 8 lists, add the people to get email one to list one, and so on. Then member of list=list one, send email one, etc. And not use random sample at all.

Otherwise it's anybody's game, anyone who qualifies has the ability to recieve any email in the sample.