Champion/Challenger A/B Testing for Smart Campaigns

[Anonymous]

We had recently been using a "pipeline splitter" smart campaign to A/B test an activity-based smart campaign pipeline of emails versus a time-based pipeline. I found out from a Marketo support rep that the logic we had been using to drop 50% of new leads into campaign A and the remaining 50% into campaign B would not work correctly because it was designed for batch campaigns rather than triggered campaigns.

She suggested I try the Champion/Challenger A/B tester because it worked with triggered campaigns. However, I soon realized this didn't really solve my problem since the champion/challenger a/b tester could only test a single emails as opposed to entire smart campaigns.

So the only way we can really test an activity-based versus time-based campaign is to have 100% of new leads go into an activity-based campaign for a set period of time, and then switch to 100% of new leads in a time-based campaign. It's disappointing that Marketo is unable to A/B test two smart campaigns from a triggered campaign source simultaneously.

[Anonymous]

I am currently having problems with A/B testing. In our case it is based off of testing through batch campaigns. You are no longer able to A/B test with batch campaigns it must be set as a trigger. The rep mentioned the only reason that batch campaign testing worked was due to a hole in the system that they recently patched. Now, if you would like to A/B test you are only allowed to do it with triggered campaigns. 

I'm wondering why features offered for A/B testing cannot be used for both batch and triggered campaigns. I'm sure other users are also having the same problems. 

[Anonymous]

That's a good question, Christina. It sounds like we're in the same boat.

[Anonymous]

There are two differnet types of Email AB testing we support:

1) Email Program supports Batch campaign AB testing
2) Champion Challenger supports trigger campaign testing as well as testing inside an engagement program

We had a bug in the Champion CHallenger test that allowed the user to make it work in a Batch Campaign, but the results were not consistent.  We have since fixed this issue, and if you select a Champ/Challenger in a batch campaign, it will only send out the default.

Hope this helps!

[Anonymous]

Hi Cheryl, 

Thank you for responding. My main concern- We were allowed to use testing features (due to the bug) that were helpful to us as users, but without warning they were removed. In some cases, the tests gave us valid reporting and it filled a need for our testing processes. Are there plans to reinstate the testing that was available with the proper functionality? 

Thank you!
Christina

[Anonymous]

Hi Christina,

The email program AB test does what you are looking for.  is there a reason you cannot use this?  Keep in mind it can be nested inside default and event programs.    Let me know.

Thanks,
Cheryl

[Anonymous]

Thanks Cheryl. I guess I'm a bit confused. I explained my issue in detail to our rep and I was told that this isn't an option that Marketo offers and that there isn't a current solution. If someone could share the process in detail I would greatly appreciate it. We find the testing features very useful. Please let me know if it would be beneficial to share the information from my Support Case ticket. 

Thanks in advance!
Christina

[Anonymous]

Have you tried to use the Email Program with AB test? Or were you not aware that we have it? It's a standard split test on a batch campaign and it can be nested inside both the Default and the Event Programs.

https://community.marketo.com/MarketoArticle?id=kA050000000LGX7CAO

[Anonymous]

Hi Cheryl,

I wish Marketo included an A/B champion/challenger test for more than just subject lines or emails. A way to A/B test two campaigns/progressions of emails from a triggered lead source would be extremely helpful.

I'm aware that you can do do this with a batch list, but that doesn't work when trying to A/B split new leads into two different campaigns for an onboarding pipeline.

Thanks,
Alan

[Anonymous]

Alan - take a look at the random sample filter...this might help you achieve your goal.  You can find it under Special Filters.

[Anonymous]

Thanks for the suggestion, but we've already tried the random sample filter. It only works for batch campaigns, not triggered campaigns.