0 Replies Latest reply on Sep 17, 2018 6:55 AM by Ronen Wasserman

    Send Email Program - Case Study

    Ronen Wasserman

      Recently a customer of ours tried to run an A/B test as part of a blast email he was sending. He contacted us, since, in the process, he ran into multiple issues, while trying to keep his old email blast program structure intact. The A/B tests he wanted to set were:

      • The email’s content (‘Whole Email’)
      • The email’s subject line (‘Subject Line’)
      • The email’s date and time (‘Date/Time’)

      As part of the A/B test, our customer wanted to use the ‘declare winner - automatic’ option as well. It was essential for him to keep all processes intact since the batch campaigns program templates worked very well for him until now; therefore he just wanted to add the A/B test functionality to the flow. As you may know, Marketo’s built-in A/B testing is only meant to be used in an Email Program. Using an Email Program easily allows you to create a test, to determine the date and time of the test and the date and time of the main blast. The A/B test dashboard is very useful. It has a step by step structure, and it ends with the last step where you need to approve the program.

       

       

      Send Email Program Dashboard

       

      Our customer's program template had nine smart campaigns. The first smart campaign was built for sending the email and the other eight were used for ‘program status change’ (for activities the lead performed after receiving the email – some of them were connected directly with the email statistics – such as ‘open’ and ‘click’, and some were related to activities in Salesforce).

      All the triggers and filters that were used in the eight of the smart campaigns (see below for more details about the send email smart campaign) used the original email name.

      As some of you may know, while using an email within an email program for A/b testing, Marketo will NOT recognize it anywhere else:

      Caution

      This could be a major issue for some, and for our customer it defiantly was. Because now all the old structure of the program became irrelevant, and it was possible that the whole tracking system of the program became irrelevant as well.

      As a matter of fact, even though it sounds like a disaster, the solution is quite simple.

      The first step is to approve the email program (the lower right rectangle). Make sure not to skip this step, or else it will be impossible for Marketo to recognize the email.

      The next step is to populate all the fields in all triggers and filters that require the email name with the following syntax:

      [parent program name].[email program name].[A/B test name] test

      For example:

      Parent program name: Black Friday

      Email program name: Send Email

      A/B test name: subject line

      Final syntax: Black Friday.Send Email.subject line test.

      Send email smart campaign: my customer is using a control group to check how an email effects on the population. In other words, he is using a "random sample" condition to take out 10% of the population, and send the email to the rest 90%.

      Using email program making such progress impossible, since it was built by Marketo as a smart campaign with one flow step – send an email.

      Our solution was to create a "preparing" smart campaign. Its smart list includes the population for the program, and the flow step separates it to control group and target group (via change program status).

      Change Program status

      The last thing to do is to set the smart list of the email program to include only the people who were marked as "Target Group".

      Just don't forget to run the preparation campaign before the A/B test time.

      Recently a customer of ours tried to run an A/B test as part of a blast email he was sending. He contacted us, since in the process, he ran into multiple issues, while trying to keep his old email blast program structure intact. The A/B tests he wanted to set were:

      • The email’s content (‘Whole Email’)
      • The email’s subject line (‘Subject Line’)
      • The email’s date and time (‘Date/Time’)

      As part of the A/B test, our customer wanted to use the ‘declare winner - automatic’ option as well. It was very important for him to keep all processes intact, since the batch campaigns program templates worked very well for him until now, therefore he just wanted to add the A/B test functionality to the flow. As you may know, Marketo’s built-in A/B testing is only meant to be used in an Email Program. Using an Email Program easily allows you to create a test, to determine the date and time of the test and the date and time of the main blast. The A/B test dashboard is very useful. It has a step by step structure, and it ends with the last step where you need to approve the program.