Email A/B Testing doesn't seem to work.

[Dylan_Teo]

Hi all,

I set up an email program and Subject Line A/B Test with a 50/50 weight. However, the results show that only the Champion has been getting clicks while the Challenger hasn't received any at all. It seems likely that none of the people in the smart list are receiving the challenger emails at all.

The A/B Test was a Subject Line test with a 50/50 weight.

For the email program flow, I used "EMAIL-Invitation 01" as "EMAIL-Invitation 01.Subject Line" was not available in the dropdown list.

Does anyone have any idea why this is happening and what can be done to rectify it?

I have referenced these two articles to no avail as the solutions provided don't offer much detail:

https://nation.marketo.com/;

A/B testing - am I doing something wrong?

I would sincerely appreciate any help on this.

[Ronen-Was-SRpro]

Hi Dylan,

Most of the times it is better to make A/B test through an Email program, from our experience the data that we get about A/B tests from programs that are not Email program are not accurate.

You can check if it is not complicated to remake it in an Email program.

[Dylan_Teo]

Hi Ronen,

Thanks for your response! Do you have a link to a guide on how to create an A/B test through an Email program? I have been exploring the different options available in Marketo to no avail.

[Veronica_Holme4]

YOu are only showing the flow here. Can I confirm that the smart list you are using here is a trigger not a batch?

Champion Challenger testing only works on triggered smart campaigns (that's it's primary purpose). If you are sending a batch campaign, I'd point you to the answer above, you'd need to use a nested email program with A/B testing to accomplish that.

Veronica

[Dylan_Teo]

Hi Veronica,

You are right. I realised that the smart list did not contain triggers, which explains why the Champion/Challener did not work.

Do you happen to have a link to a guide on how to create a nested email program with A/B testing? I have been exploring the different options available in Marketo to no avail.

[Grace_Brebner3]

It's basically exactly the same process for creating a standard email program with A/B testing, you're just creating the program inside another program instead of inside a system folder. See: Email Test - A/B Test - Marketo Docs - Product Documentation​, and some notes about nesting email programs here: Understanding Email Programs - Marketo Docs - Product Documentation

[Lillian_Huang · Accepted solution]

Hi Dylan Teo​, It looks like in your screenshot that there's been only 1 Challenger set up. Marketo will not acknowledge the original subject line in the "Champion" email as a contender, and you'd need to list out 1. (Original Subject Line) and 2. (New Subject Line) , 3. (Other New...), etc.

When you are using an Email Send (mailbox) program, it is most effective for when you want to set up a consecutive flow for an A/B Test:

1. Run A/B test for a % of the population. So if you select 60%, 30% will receive test A and 30% will receive test B.

2. Sending winner to remaining population, which in this case is 40% - which you can set based on an engagement metric (open, click, score, etc) or to manually determine the winner before the next scheduled date.

To test 100% of the population, like others suggested, using a trigger campaign (in any program) would be most optimal.... and there's a few ways I like to do:

[Request Campaign trigger] where you have two smart campaigns set up:

1. The A/B Send campaign . (Trigger)

- Smart List: Add a Campaign is Requested Trigger, with Marketo Flow Action as Source

- Flow: Set up Send Email step with the email that already has the A/B Test approved (like your screenshot).

- Schedule: Then click Activate so the trigger is ready to fire!

2. The Email Send Campaign    (Batch)

- Smart List: Set up your batch as needed in the smart list. In Flow, set up "Request Campaign"

- Flow: Use a flow step, Request Campaign, and then search for the "A/B Send" trigger campaign that you just set up. Remember it needs to be Activated in order to appear in the dropdown.

- Schedule: When you are ready to have the A/B test rocket fire away, you can schedule this batch smart campaign. Before you do so, highly recommended to double check your email's A/B test, which you can do by right clicking on the email asset and "Send Sample Test".

--------------------------------------------------------

--------------------------------------------------------

[Add to List trigger] by using a smart list, static list, and a trigger campaign.

1. Create a Static List which you will use for the entire A/B test population.

2. The A/B Email Send  

Set up just like the first example's , except use a Add to List trigger instead of a Request a Campaign. Then in dropdown, select the Static List you just created.

Remember to Activate again!

3. Set the Trigger

So there's a couple ways around this.... similar to

OR you can create a Smart List to shape the email audience as you please

....then you go to the People tab and "Select All"

....then go to Person Actions > Marketing > Add to List               

....then you select the static list (from step 1) and confirm to run.  This will trigger the A/B test immediately!

(Another creative method? use this method with a "Request a Campaign" trigger and then when reviewing a separate Smart List, you can also go to Person Actions > Special > Request a Campaign.  Then, you select the "A/B Send" trigger campaign that you just set up. Once you request this campaign, the trigger will immediately fire and send the A/B test. So again, be prepared with all your check points before doing this.)

[Dylan_Teo]

Hey all, following up on this, I emailed Stephanie, a Marketo architect and got a very detailed response from her. Sharing it here in hopes that it will help anyone who has the same confusion as I did:

"Hi Dylan,

Apologies - I just noticed your email now.

Setting up A/B testing directly on the email asset will only work for trigger campaign as you well aware for challenger email.

Marketo sets this up and ensures that ultimately each version will be proportionately sent out to a percentage of people considering that it is a trigger campaign, which means that you do not know how many people will go through the program at the end of the day.

An A/B test can only work for Email Program. This does not work for any other channel.

Normally with A/B testing, you are looking at which email performs the best and from there you can start implementing your own best practice of an email.

You also want to ensure that you can get the right type of reporting for the entire marketing program.

This means the build of the program is important to provide you with the right ROI reporting.

The challenger email reduces the complexity for trigger campaigns, so you can focus on building a program for the reporting that you need.

Another way to do A/B testing is by using random sample flow step for batch campaign that is not an email program.

Happy to go through this with you in depth if needed."