6 Replies Latest reply on Jan 30, 2013 12:35 AM by 6779

    A/B Testing

      I am testing my campaign that will be deployed tomorrow morning. We are testing 8 different subject lines. I have set up the flow with 7 choices and the default. I added choice with random sample is 12 and 13 percent for 7 of the email, then the default.

      I tested the campaign on my seed list and the first 8 emails went out correctly, sending all 8 versions, but after completing emails 9-15 were all the default email. I need the system to start again at the first email and continue through all 8 versions. 

      Any suggestions?


        • Re: A/B Testing
          Hi Kyle,

          I think you need to change your Random Sample flow steps a bit. Your choices should all be Random Sample percentages totalling 100, and your default should be "Do Nothing". Something like this:

          If Random Sample is 12 Send Email #1
          If Random Sample is 13 Send Email #2
          If Random Sample is 12 Send Email #3
          If Random Sample is 13 Send Email #4
          If Random Sample is 12 Send Email #5
          If Random Sample is 13 Send Email #6
          If Random Sample is 12 Send Email #7
          If Random Sample is 13 Send Email #8
          Default: Do Nothing
          • Re: A/B Testing
            Hi Drew,

            I set up the random sample as you suggested above. Unfortunately, the program only went though the 8 emails and then stopped. It didn't restart at Email #1 and continue to deploy the campaign. 

            Kyle
            • Re: A/B Testing

                   I'm confused, you said you had 8 emails and honestly either flow setup should work as described above.

                   Where are emails "9-15" coming into this?

                   You may want to consider generating 8 separate randomly generated lists from the main target group you plan to use. Then set those up as Send Email choice steps.

              • Re: A/B Testing
                Hi Josh,

                My campaign consistes for 8 different emails. I am testing the campaign on my seed list, which is 15 email addresses (this is where emails 9-15 come in).

                What I had in mind when I set up the random sample choice in the flow step was that the campaign would send every email address a different email (from email #1- email #8). So, once it send the first 8 email addresses in the campaign email #1-email #8, then it would start again and the 9th email address in the campaign would receive email #1, then email address 10 would receive email #2, and so on.

                When I originally set up my campaign with:

                If Random Sample is 12 Send #1
                If Random Sample is 13 Send #2
                If Random Sample is 12 Send #3
                If Random Sample is 13 Send #4
                If Random Sample is 12 Send #5
                If Random Sample is 13 Send #6
                If Random Sample is 12 Send #7
                Default: Send email #8

                I thought that was the proper way to do it. And when I sent the campaign to my seed list, the campaign would properly send emails 1-8, but the rest of the email addresses afterwards would only receive the default email #8.

                I am a little confused about your suggestion about 8 seperate randomly generated lists? Can you help explain
                • Re: A/B Testing
                  Hi Kyle,

                  If you setup the flow I recommended with the 8 different emails as random samples and the default as "Do Nothing", it should properly distribute the 8 different emails over the entire distribution list. However, the Random Sample (as far as I'm aware) doesn't guarantee a linear process like you're imagining, where contact #9 receives email #1 and restarts the order. The Random Sample only guarantees that the emails will be distributed in the proportions you designate.

                  Drew
                  • Re: A/B Testing
                    Drew's right, you can't specify who gets what, that defeats the purpose fo the sample being random. If instead you'd like it to be precise, then create 8 lists, add the people to get email one to list one, and so on. Then member of list=list one, send email one, etc. And not use random sample at all.

                    Otherwise it's anybody's game, anyone who qualifies has the ability to recieve any email in the sample.