A/B testing timing for trigger campaign

[Elizabeth_Balle]

I am a little stumped on the best way to do this. I am setting up an abandoned cart campaign and want to test when to send the email, 3 days and 10 days following the date they abandon (I know these time frames sound long, but this is appropriate for our audience). I think I'm going to have to have two separate campaigns, but what's the best way to ensure that people only end up in one campaign or the other?

[John_Clark1]

Hi Elizabeth,

You could have a campaign with a 3 day wait step, then a Send Email flow step with a choice like "If Status is "purchased" > do nothing" and default Send Email.  Then you'd have another wait step of 7 days, and another Send Email flow step with the same choice.  That way there's only one campaign, and is their status changes during the wait, then they won't get the email.

John

[Josh_Hill13]

You may have to have two separate emails in this case, otherwise the report will show the same email and confuse the data.

[Elizabeth_Balle]

I think you're right. What would you suggest for making sure they are only in one campaign?

[John_Clark1]

Hi Elizabeth,

I think Josh was suggesting two different emails in one campaign.  One email for the 3 day send, and one for the 10 day send.  Then you'll be able to track more easily how many of each email is going out.  This can still be done using a single campaign and the flow I suggested earlier.

John

[Elizabeth_Balle]

Oh, ok I get it. But i don't want to send everyone both emails. I only want to send one email at 3 days to 50% of the people, and at 10 days ot the other 50%. But since it's based on a trigger, I'm not sure what I should be doing.

[Dory_Viscoglio]

Hey Elizabeth, one way to do this with a single campaign would be as outlined below. Basically everyone goes into the campaign and they're either sent the email or nothing happens, if they were sent the email they're removed, wait 7 days and then if they're still in the campaign they'll receive the 10 day email.

As Josh suggested, I would recommend two different emails so you can see performance of both. And you can review completed sales to see if the person received the first email or the second email.

[Anonymous]

if you don't want to use two e-mails, wouldn't it be a solution to send the email to 50% and send the other to a different campaign with a 7 days delay? that way you can use the same e-mail and look at the "campaign email performance" report, which I find more reliable than the regular "email performance"..

[Dory_Viscoglio]

Hey Jonas, yes, you could send to two different campaigns and then look at campaign email performance. I don't know if I would say that campaign email performance is any more reliable than a regular email performance report, though.

Basically you can either have two emails, or two smart campaigns... I think there are benefits to each option but I typically prefer to have fewer smart campaigns and more assets if I have to choose one.

[Dory_Viscoglio]

Also, you'd probably want another remove from flow step before both the 3 and 10 day sends in case they've purchased

[Anonymous]

Thanks for this! Super helpful!

[Elizabeth_Balle]

Yes, this is what I'm looking for. Thank you!

[Elizabeth_Balle]

Ok I set this up and now looking at it, I still have a concern. Since it's triggered, it'd just be one person at a time hitting the flow, right? So how would the 50% work?

Do I need to set up a campaign to dump people in to two different lists, then have those lists grabbed by the different wait period campaigns? Similar to what John suggested, but in his suggestion, I have the same concern about how do 50% of the people actually get moved over when it's a trigger...

[Dory_Viscoglio]

Hey Elizabeth, I don't know if I have any running at the moment, but I've done this before and never had a problem with triggered campaigns (even over the course of several weeks) receiving a 50/50 split. This is how we do a lot of our A/B testing since I hate that the results of Marketo's A/B tests disappear once you declare a winner. But, if you're concerned you could always run some tests with much shorter wait periods to sample how it would work.

One thing to note here, depending on how often you let someone flow through the smart campaign (qualification rules on the schedule page), someone could get into the flow and be sitting in a wait step and then get into the flow again. I would advise setting the frequency that they are eligible to once every 10 days at a minimum.

[Anonymous]

well... regarding the 50%.. I don't know how it works technically, yet. The campaign structure though would need to look something like below to make sense in my head.

campaign 50/50

     trigger = abandon cart

     random sample = 50

          add to list 3 days

     filter = is not in 3 days

in flow step: add to list 10 days

3 days = wait 3 days

10 days = wait 10 days

[John_Clark1]

Ahhh, ok.  So you're wanting to try sending emails to some leads after 3 days of having abandoned their cart, and some after 10 days to see which gets more people to return?  I understand now.  I though you wanted leads to get one at 3 and 10 as a sort of double reminder.

In this case, the normal A/B testing methods won't work because of the difference in timing.  What I would suggest instead is having one campaign that detects when they've abandoned their cart, and then 50/50 randomly calls either a campaign for a 3 day reminder or a campaign for a 10 day reminder.

Campaign 1

     Smart List

          (Triggers/Filters to identify leads you want to target)

     Flow

          Request Campaign flow step

               Add choice (If Random Sample is 50) > Request 3 Day Campaign

                    Default Choice > Request 10 day Campaign

3 Day Campaign

     Smart List

          Campaign Is Requested trigger

     Flow

          Wait 3 days

          Send Email

               (Add choice for Lead Status) If Status is "Abandoned" > Send 3 Day Email

                    Default > Do Nothing

10 Day Campaign

     Smart List

          Campaign Is Requested trigger

     Flow

          Wait 10 days

          Send Email

               (Add choice for Lead Status) If Status is "Abandoned" > Send 10 Day Email

                    Default > Do Nothing

[Elizabeth_Balle]

Ok, I like this. Question though, if this is spanning several weeks, how do we know that it's going to go 50% to each of the campaigns? If it's triggered, then only one person goes through the flow at a time, right?