Problems A/B Testing in a Smart Campaign

[Anonymous]

Can someone please help? I'm new to using Marketo and we set up two of our emails (both within separate smart campaigns) to have A/B subject line testing. We set the distribution to 20%/80% but only one subject line was sent. This happened for both smart campaigns.

These are the test results I see on my dashboard:

Here is what my campaign set up looks like (there are triggers set up in the flow, which aren't shown here):

Here is what I see when I set up an A/B test in my smart campaign email:

I'm not sure why there is a "Champion" already defined on this screen, but it's defaulting to the subject line that I used when I created the email. Should "Champion" be left blank and I define two Challengers?

Thank you for any help you can provide!

[Jenn_DiMaria2]

It looks like you've created a Champion/Challenger (within an already-existing Program) as opposed to an A/B Test (within an Email Send Program). They function slightly differently, so the setup also looks slightly different.

The dashboard you provided only shows opens (not sends), which means it's possible people received the Challenger email, but didn't open it. Can you run a report?

[Anonymous]

Thank you for responding, Jennifer. Where would I go to run a report manually? It looks like we have the report set up to be sent to us later today, but if I can pull it manually I'll look at it now.

[Anonymous]

In reading through the help articles, I see that we're supposed to declare a winner​, but I'm not fully understanding that process. The article states we select the winner of our choice, but shouldn't the winner be declared automatically based on the 20%/80% scale we set?

We thought that Marketo would send the subject line test to 20% of our audience and automatically declare the winner based on the number of opens (which was the criteria we selected) then the remaining 80% of the audience would receive the subject line that had the most opens. Are we incorrect in that assumption?

[Jenn_DiMaria2]

It looks like it's only automatic when using an A/B test within an Email Send Program (batch campaign). Champion/Challenger should be used only for trigger/engagement campaigns, which is what you've got going for you (from what I'm reading), so the use case is correct; however, because it's a trigger campaign, you will need to select the measurement criteria and manually declare a winner based on the results.

As far as your percentage audience, 20% of your audience would receive the test and the rest would receive the winner when the test is complete. So yes, your assumption is correct. I'd run an Email Performance Report to see how many emails were sent vs. opened, since the dashboard is only displaying opens.

[Anonymous]

I just received the email from Marketo about the subject line test:

Here is the sent information. Should this report show the email test, as well?

Our concern is that is looks like no one received the challenger subject line and we're not sure how to verify that? And if we can verify that no one received the challenger subject line, how can we fix that going forward?

Thanks!
Mandy

[Jenn_DiMaria2]

Do you have an example of how the triggers are set up in your campaign? If the campaign you're testing from is the "Pre-Registration" campaign, then that's a batch campaign and the test won't work.

[Anonymous]

Bummer. I think that's the case! We'll work on setting this up so it's not a batch and try that. Thank you so much for your help! I am really excited to learn more about how to use Marketo and being a part of this great community.

[Jenn_DiMaria2]

No problem, Mandy! I actually learned something myself while looking into this There's a lot to learn, but it's fun.