Champion/Challenger A/B Testing for Smart Campaigns

[Anonymous]

We had recently been using a "pipeline splitter" smart campaign to A/B test an activity-based smart campaign pipeline of emails versus a time-based pipeline. I found out from a Marketo support rep that the logic we had been using to drop 50% of new leads into campaign A and the remaining 50% into campaign B would not work correctly because it was designed for batch campaigns rather than triggered campaigns.

She suggested I try the Champion/Challenger A/B tester because it worked with triggered campaigns. However, I soon realized this didn't really solve my problem since the champion/challenger a/b tester could only test a single emails as opposed to entire smart campaigns.

So the only way we can really test an activity-based versus time-based campaign is to have 100% of new leads go into an activity-based campaign for a set period of time, and then switch to 100% of new leads in a time-based campaign. It's disappointing that Marketo is unable to A/B test two smart campaigns from a triggered campaign source simultaneously.

[Anonymous]

Cheryl - Thanks for responding. We have had this same issue and have switched to using email programs when we need to AB test, but do not like to do this because there aren't the safeguards in place for Email Programs that there are for smart campaigns. Namely, the abort funtionality and the ability to automatically stop a campaign affecting more than a designated number of users that work with smart campaigns sending an email. As a result, we have had an accidental mailing to the entire database, and are still sorting out the resulting deliverability issues. It is my feeling that if we are forced to used email programs they need to be properly supported. We try to be very careful with our mailings, but human error is bound to creep in sometime and these safeguards are important.

[Anonymous]

Hi, Cheryl Chavez and team!

We were hit by this one square in the face when trying to launch a program this week, and the reason we needed this specific program type is that we were staggering sends by time zone, and the "standard" program type seemed to be the only way to let that happen. We wanted all recipients to receive the email within what we have determined to be the "ideal" time window, and since the Email program type doesn't allow for staggered sends, this was what we decided to use.

We did end up being able to implement that triggered workaround, but it's a pain. I see by the flag on this idea that it's a "maybe one day", but please will you consider moving this up the roadmap a little bit? Being able to use champion/challenger testing on batch sends seems like a fairly obvious feature to include as a standard, and the Email program type is not always the best solution when there are other factors that need to go into a program.

Thanks for considering it. 😉

I hope you're well! Miss you guys!

Rebekkah