Global A/B Testing Woes?

[Anonymous]

I seem to be having a number of challenges doing A/B testing for a global enterprise company. How is everyone else managing A/B tests when there are multiple time zones, languages, and platform limitations?

• Engagement Programs
  • To enable the most sophisticated functionality, we are using "program-in-a-program" that sends the emails using smart campaigns that are added to our engagement stream. However, it appears that the A/B testing in Engagement Programs ONLY works when a plain email is dragged into the stream. Our company prefers program-in-a-program so that we can have individual SFDC campaigns, multi-touch attribution, and the ability to exclude people from receiving promoted assets that they've already downloaded on our website. So the standard functionality isn't a great option here.
  • So then that leads us to two options - we either have to set up a champion/challenger within the program which basically will use the smart campaign in the stream to call a trigger campaign (which sends the email and qualifies the send for a champion/challenger). However, if we add a large list of recipients into our nurture program (i.e. 20,000) then the trigger-based campaign takes a long time to execute (around 1 second per recipient) and results in slow system performance. The end result is that we might be sending emails at 8pm at night instead of 9AM in the morning - plus it involves extra setup steps to "jerry rig" this type of setup.
  • The other option is to simply to a random sample conditional flow step in the smart campaign's "Send Email" step. This works OK and has been our preferred method for engagement programs. It involves some manual analysis to determine which one worked better, but it's OK. However, this doesn't work so well in other examples (listed below)
• Email Programs
  • Email Programs sound like a good option for email A/B tests for email blasts. They don't require doing random sample filters, creating unnecessary trigger campaigns to do champion/challenger, etc. However, there are a few limitations for global companies:
    • The Subject line and From Name/Reply-to tests don't seem to take into account Dynamic Content. So if you have a multilingual email being sent out in 15 languages, there doesn't seem to be an option to set the subject line test for only the North American English segment.
    • The Email Programs do not currently support global time zones. It's meant to be a simple way to send an email blast, but if your audience is spread all over the globe I would have to either schedule it three times (for EMEA, APAC, and NA) or create three separate programs.
• General Programs
  • So it's my assumption that most large companies will end up creating a general program using wait steps that send the email by the end-user's time zone. This works for for a standard send (If Time Zone = APAC => Send Email, Wait 8 Hours, If Time Zone = EMEA => Send Email, Wait 8 Hours, If Time Zone = NA => Send Email). However, there's no way to combine multiple filters into conditional flow steps (like "If Time Zone = APAC and Random Sample = 50%") which means that most global campaigns using an A/B tests will probably be split into three separate batch campaigns that do the same thing (but are scheduled at different times). This is probably our best option but it will require memorizing the specific times of day that each time zone should receive these and require scheduling 3x as many campaigns as we really should.
  • I suppose the second way of doing this would be to build a batch campaign with <<Request Trigger-based Email Send with Champion Challenger>> wait 8 hours <<Request Trigger-based Email Send>> Wait 8 Hours <<Request Trigger-based Email Send>>. But as we all know, Marketo doesn't automatically deduplicate by email address when emails are sent through Trigger campaigns (they are seen as unique leads), so a recipient would receive both versions of the email if they're a duplicate in the database. The fix would be to export your list out of Marketo, dedupe in Excel, and import back into Marketo as a static list and then run your batch campaign based off of that. And of course hope the queue doesn't get too backed up if you had a large send. So this option is not great because it really involves setting up multiple campaigns, list imports, and a lot of campaign setup work just to do an A/B test for an email that's sent to multiple time zones.

SHORT STORY – there are lots of ways to A/B test in Marketo but it’s hard for big global companies with big databases, multiple languages, and time zone considerations. As Marketo moves upward in the market to compete with enterprise-grade solutions, is this something that can be prioritized to make simpler/faster/better?

Or if someone else knows a better way than all of the above, I am all ears and would love to hear what you are doing!

[Colin_Mann]

We are also using the ‘program-in-a-program’ approach using smart campaigns but we are doing A/B testing at the most foundational level right now i.e. testing subject lines and email layout. We’ve not gone deeper into the other variables that we could test. At this level the champion/challenger approach works OK for us. The slowness is generally a challenge I find when processing larger data sets across the board in Marketo and it’s something I hope they can improve.

The random sample conditional flow step is good idea.

Personally I don’t use or fine Email Programs helpful in this case. We generally prefer the flexibility of the ‘program-in-a-program’ plus smart campaigns approach because it lends itself much better to tracking the engagement moments we are most interested in and passing the right information about those over to the CRM.

We only use the General Programs for scenarios when we are passing data over quickly and are perhaps only sending 1 email to keep our brand on the desk of the prospects before our lead generation agents make a call. But again we also use the ‘program-in-a-program’ plus smart campaigns approach here anyway.

At the moment we create separate engagement programs for each country/region and this is how we handle the time zone challenge. We don’t use the ‘wait step’ as we try to avoid one-off emails but tend to use engagement programs. We simply clone the program and then update the smart campaigns. However, this would be a challenge when sending 1 email to a region.

We are currently rebuilding our nurture tracks to take into account factors such as streams that use the same emails and landing pages i.e. so that we can minimise our email/landing page creation work.

I am talking with your company at the moment as it happens about potential campaign support.

I agree with your last comment. Once you get into multiple scenarios it’s easy for campaigns to become over complex and that’s when mistakes can creep in.