Provide the ability to perform automated A/B testing on batch emails within Marketo programs

[Dan_Stevens_]

I just finished a product roadmap call for a competitive marketing automation platform.  Like most other ESPs and marketing automation platforms, this platform has automated A/B testing (similar to the A/B testing that exists with Marketo Email Programs).  I realize this already exists for Email Programs.  What I'm asking for is the ability to do the same automated A/B testing within regular Marketo programs.  The champion/challenger method requires too much manual effort (especially for global organization with multiple workspaces).  The goal here is to automate the entire A/B testing of our emails within regular Marketo programs.

[Anonymous]

Dan,

I think I get what you mean, but can you be more specific about the vendor and ideal workflow?

The AB testing here is more of an add on, than embedded into the native workflow. Champ Challenger is trigger only.

There are some other ideas on improving AB testing overall, so I hope they can rebuild it to be more streamlined and provide better stats.

[Dan_Stevens_]

Josh - what I'm looking for is really basic:

• Select a test type or a combination of more than one (e.g., subject line, preheaders, from name, email content/design).
• Select the size of the test group (usually done by dragging an interactive slider) - e.g., 10% get version A; 10% get version B.
• Determine "winner" criteria (e.g., open rate, unique clicks, clicks on selected links)
• Define duration of test (e.g, 5 hours)
• AUTOMATICALLY send the remaining emails to the audience based on the "winner" (this is really the key point - to have the winning email sent automatically)

As for the competing vendors, almost all of them provide this within their product (links link to each product's A/B test capabilities): CampaignMonitor, Eloqua, Adobe, Act-On, Hubspot, etc.  The one I was referring to specifically was Sitecore (Sitecore is our website CMS and is significantly enhancing their offering to also compete with the top tier marketing automation vendors like Marketo, Eloqua and Adobe.
 

[Anonymous]

We offer everything you described in the Email Program.  What I am not clear on is why you need this on a regular program.  You can nest email programs within regular programs to achieve the same result.

You hightlighted AUTOMATICALLY.  We do have an automated send of the winner.

[Dan_Stevens_]

Yes, I understand that this is possible by nesting an Email Program within a regular program.  Not only does this add extra overhead within an existing program, but there are some limitations of an Email Program, such as:

• the inability to send multiple waves of that email.  Sure, you can clone it, but again, that's extra overhead and added bulk to our program structure/approach
• the email must be local to the program - for example, you cannot use Design Studio emails in an Email Program.  In fact, I think all activity (smart lists, smart campaigns) must be local to the Email Program.
• we've experienced significant delays when sending emails inside an Email Program
• inability to pause/kill an A/B test that's already in progress
• inability to send according to the recipients' local time zone (this is a limitation throughout Marketo)

Others have found these same limitations with the Email Program - here's a thread from last year.

But most importantly for us - Email Programs cannot be nested in Engagement Programs.  Almost all of our targeted, campaign-focused email sends are done within a stream of an Engagement Program.
 

[Michelle_Tizian]

I agree with Dan, it would be nice to nest the Email Programs in an Engagement program.  Also, the email within an email program doesn't go out until 15 minutes later, and you have to wait 4 hours for the results before the winning email is sent to the rest of the leads.  

[Dan_Stevens_]

Can someone from the Marketo Products team chime in here and let us know if something like this is part of the 2016 roadmap? 

[Jamie-Carey]

Really frustrated that the docs don't say that you can't use champion/challenger in a program in an engagement stream. Tired of having to do workarounds after discovering something doesn't work as expected.