When Email Programs won't cut it: how to test your emails

Anonymous
Not applicable

This post outlines how to test emails when you want to:

  • test for statistical significance
  • test an email in a nurture
  • have clear reporting when using an email performance report
  • track what variation a lead received

Email Programs are great to use if you are just getting started with A/B Testing. Email programs allow you to test a variety of different elements in your email, and then declare a winner after a send.

pastedImage_17.png

But what about when you want to A/B test an email in your nurture program? Email Programs come with their limitations:

pastedImage_16.png

I'm going to show you how I A/B test my email blasts and email nurtures. See this as a different way to test if you have one of the challenges above.

An A/B test for a simple email is easy to set up, and usually your objective will be opens and/or clicks. As pre-work, I strongly recommend outlining your hypothesis, goals and variable for your A/B test. You can use the framework I've outlined in my other post as a reference. Here's an example of an email A/B test outline.

In this example, we are going to

  • test an email used in a nurture program
  • create a 50/50 split test
  • test the subject line as a variable

Start from a default program in Marketo

First, create a program in Marketo that contains an email. If you are going to test an existing email that's already live, skip this part and jump to the next section: Emails.

pastedImage_0.png

Emails

Create your control email. This is the email that you're going to try and beat out with your variation email.

You may be testing an existing email, or you may be creating a new email. You want your control email to be the content that you want to beat. So if your hypothesis is that "a funny subject line will be more effective than a serious one," your control email will be the serious one.

Once your control email is completely reviewed, tested and approved, simply clone that email and append v2 to the new variation:

pastedImage_12.png

Smart Campaigns

OK! Now you’re going to set up campaigns that will automate the testing for you.

Campaign: Send Email

Depending on whether this is a blast or an email nurture, the Smart List will vary. Everything else is the same. In this example, I'll show how you set it up as a nurture email.

Smart List

Member of any Engagement Program (the filter lies in the stream itself, when you drag the entire program into the stream).

pastedImage_7.png

Flow

pastedImage_8.png

Schedule

Do nothing

For Nuture

Add to Engagement Stream

pastedImage_9.png

Activate the campaign

pastedImage_10.png

Reports

Let’s create a basic report so you can check your test at a moment’s notice!

Report: Email Performance

Smart List: none

Setup:

Sent Date: All Time

Marketing Activities Emails: Email .v1, Email .v2

pastedImage_4.png

That's it! So simple! Now when you look at your email performance report, you will easily see which one is performing better:

pastedImage_11.png

3341
2
2 Comments
Tara_Wright
Level 1

Hi, you mentioned used this for both email blasts and nurture programs.

If using for email blasts, after you have created your smart list, in the flow would would you have 2 choices for Send Email for your your A and B emails and as the default leave as Do Nothing? After deciding on a winner, creating a second smart campaign to send the winner, excluding those that have already been sent both variations of the email?

Eg.

Random Sample = 10, send Email A

Random Sample = 10, send Email B

Default = Do Nothing

I prefer to use Default Programs over Email Programs so we have the ability to re send emails if needed. 

Grace_Brebner3
Level 10

Hey Tara, 

The discussion you're replying to is over three years old & the OP is now deactivated. You might want to start a new question thread to get your question answered