A/B Testing Functionality for Emails

[Anonymous]

I think it should be possible to run A/B testing on emails in the same way that the current landing page A/B testing works. Basically, create the ability to convert an email asset to a test group, and when sent by a smart campaign, send version A to a fraction of the list, version B to a fraction, and so on.

I know AB testing is possible using the Random Sample function in a smart campaign, or by manually segmenting the list through smart lists, but this isn't terribly user friendly, and can be time consuming. As a user who likes tests everything, the addition of this feature would be a huge timesaver for me, and would likely make it a whole lot easier to track results later.

[Anonymous]

It'd be nice to see some easy A/B functionality in the Engagement Program!

[Anonymous]

Great idea Emily! It's been about a year since I originally submitted this idea. Would love to hear from Marketo if anything for easily A/B testing emails is coming down the pipe in the future.

The need for this has actually grown more important with the new engagement programs, as you cannot do the random sample approach at all in these programs, and must link to other programs/smart campaigns to accomplish it. It's still possible and workable, but in this situation, the engagement program has actually made A/B email testing more complicated by adding this second step. 

I think an elegant and dead simple approach for A/B testing emails (aka just like the way the landing page testing currently works) would be a huge win. 

[Patsy_Dowling3]

We are currently working on AB Testing of emails hope to deliver the feature in early Fall 2013.

[Anonymous]

Hi Patsy,

Thanks for the heads up, and great to hear!

[Anonymous]

Hi Patsy D,

When developing this, can you make sure this works well with engagements streams?  

Here is a use case and two requirements:

• Use Case:  We send a A/B email test group to most of our database.  We then add that same A/B test group to an engagment stream.
• Requirement 1:  Engagement stream prevents people who previously recieved either email in test group from receiving this email as part of a stream
• Requirement 2:  We're able to choose to use the winning email only in the engagement stream (i.e. in the test group, assign one email as the winner so future enagement streams only send the winner. 

Right now, if i'm sending a great email that i want to also use in engagement streams, I either need to forfiet the A/B test or risk that someone will later get the same email twice as part of a stream (and right now i still A/B and take the risk). 

Thanks!
-Lee

[Anonymous]

Hi Patsy D, 

Here is another idea for this development.  Can there be a special A/B Email test report that allows: 

• Pull "From Name" into the report
• Pull "Subject Line" into the report
• A field at the A/B test group where we can describe the testing strategy
• A field at the A/B test group to make notes on what was learned from the test

Use case:  Every week we review what we learned from the previous week's tests.  Every quarter we look back at our email tracking sheet to review the tests where we learned something so we can incorporate these learnings into the future.  Having these learnings summarized makes it easy.

Thanks!
-Lee 

[Anonymous]

Delivered in our December 2013 release