7 Replies Latest reply on Sep 13, 2018 10:58 AM by Dylan Teo

    Email A/B Testing doesn't seem to work.

    Dylan Teo

      Hi all,

       

      I set up an email program and Subject Line A/B Test with a 50/50 weight. However, the results show that only the Champion has been getting clicks while the Challenger hasn't received any at all. It seems likely that none of the people in the smart list are receiving the challenger emails at all.

       

      Screen Shot 2018-08-31 at 1.53.36 PM.png

       

      The A/B Test was a Subject Line test with a 50/50 weight.

       

      Untitled 2.png

      For the email program flow, I used "EMAIL-Invitation 01" as "EMAIL-Invitation 01.Subject Line" was not available in the dropdown list.

       

      Untitled.png

       

      Does anyone have any idea why this is happening and what can be done to rectify it?

       

      I have referenced these two articles to no avail as the solutions provided don't offer much detail:

      A/B Testing vs. Champion/Challenger Emails

      A/B testing - am I doing something wrong?

       

      I would sincerely appreciate any help on this.

        • Re: Email A/B Testing doesn't seem to work.
          Ronen Wasserman

          Hi Dylan,

          Most of the times it is better to make A/B test through an Email program, from our experience the data that we get about A/B tests from programs that are not Email program are not accurate.

          You can check if it is not complicated to remake it in an Email program.

          2 of 2 people found this helpful
          • Re: Email A/B Testing doesn't seem to work.
            Veronica Holmes

            YOu are only showing the flow here. Can I confirm that the smart list you are using here is a trigger not a batch?

             

            Champion Challenger testing only works on triggered smart campaigns (that's it's primary purpose). If you are sending a batch campaign, I'd point you to the answer above, you'd need to use a nested email program with A/B testing to accomplish that.

             

            Veronica

            • Re: Email A/B Testing doesn't seem to work.
              Lillian Huang

              Hi Dylan Teo, It looks like in your screenshot that there's been only 1 Challenger set up. Marketo will not acknowledge the original subject line in the "Champion" email as a contender, and you'd need to list out 1. (Original Subject Line) and 2. (New Subject Line) , 3. (Other New...), etc.

               

              When you are using an Email Send (mailbox) program, it is most effective for when you want to set up a consecutive flow for an A/B Test:

              1. Run A/B test for a % of the population. So if you select 60%, 30% will receive test A and 30% will receive test B.

              2. Sending winner to remaining population, which in this case is 40% - which you can set based on an engagement metric (open, click, score, etc) or to manually determine the winner before the next scheduled date.

               

              To test 100% of the population, like others suggested, using a trigger campaign (in any program) would be most optimal.... and there's a few ways I like to do:

               

              [Request Campaign trigger] where you have two smart campaigns set up:

              1. The A/B Send campaign . (Trigger)

              - Smart List: Add a Campaign is Requested Trigger, with Marketo Flow Action as SourceScreen Shot 2018-09-04 at 7.15.06 PM.png

              - Flow: Set up Send Email step with the email that already has the A/B Test approved (like your screenshot).

              - Schedule: Then click Activate so the trigger is ready to fire!

               

              2. The Email Send Campaign    (Batch)

              - Smart List: Set up your batch as needed in the smart list. In Flow, set up "Request Campaign"

              - Flow: Use a flow step, Request Campaign, and then search for the "A/B Send" trigger campaign that you just set up. Remember it needs to be Activated in order to appear in the dropdown.

              Screen Shot 2018-09-04 at 7.16.03 PM.png

              - Schedule: When you are ready to have the A/B test rocket fire away, you can schedule this batch smart campaign. Before you do so, highly recommended to double check your email's A/B test, which you can do by right clicking on the email asset and "Send Sample Test".

              --------------------------------------------------------

              --------------------------------------------------------

              [Add to List trigger] by using a smart list, static list, and a trigger campaign.

              1. Create a Static List which you will use for the entire A/B test population.

              2. The A/B Email Send  

              Set up just like the first example's , except use a Add to List trigger instead of a Request a Campaign. Then in dropdown, select the Static List you just created.

              Remember to Activate again!

              3. Set the Trigger

              So there's a couple ways around this.... similar to

              OR you can create a Smart List to shape the email audience as you please

              ....then you go to the People tab and "Select All"

              ....then go to Person Actions > Marketing > Add to List               

              Screen Shot 2018-09-04 at 7.42.16 PM.png

              ....then you select the static list (from step 1) and confirm to run.  This will trigger the A/B test immediately!

               

              (Another creative method? use this method with a "Request a Campaign" trigger and then when reviewing a separate Smart List, you can also go to Person Actions > Special > Request a Campaign.  Then, you select the "A/B Send" trigger campaign that you just set up. Once you request this campaign, the trigger will immediately fire and send the A/B test. So again, be prepared with all your check points before doing this.)

              2 of 2 people found this helpful
              • Re: Email A/B Testing doesn't seem to work.
                Dylan Teo

                Hey all, following up on this, I emailed Stephanie, a Marketo architect and got a very detailed response from her. Sharing it here in hopes that it will help anyone who has the same confusion as I did:

                 

                "Hi Dylan,

                 

                Apologies - I just noticed your email now.

                 

                Setting up A/B testing directly on the email asset will only work for trigger campaign as you well aware for challenger email.

                Marketo sets this up and ensures that ultimately each version will be proportionately sent out to a percentage of people considering that it is a trigger campaign, which means that you do not know how many people will go through the program at the end of the day.

                 

                An A/B test can only work for Email Program. This does not work for any other channel.

                 

                Normally with A/B testing, you are looking at which email performs the best and from there you can start implementing your own best practice of an email.

                You also want to ensure that you can get the right type of reporting for the entire marketing program.

                This means the build of the program is important to provide you with the right ROI reporting.

                 

                The challenger email reduces the complexity for trigger campaigns, so you can focus on building a program for the reporting that you need.

                 

                Another way to do A/B testing is by using random sample flow step for batch campaign that is not an email program.

                 

                Happy to go through this with you in depth if needed."