Random Sample with 2 email copy

[Anonymous]

Hi

Please help me set up the random sample. I know about A/B testing but random sample is new that I want to use because we are using default program type.

So we have a list of around 1500 members and two emails with the different subject line. how can we determine which copy has the better open rate?

I am doing the batch campaign for these members so 1st batch has around 500 members.

Thanks,

Vipula

[Grace_Brebner3]

Hey Vipula,

When using random sample in the flow steps to test email variants against each other, you need to review an email performance report to understand which of your two emails is performing best, and then update your "winner" send steps accordingly.

[Anonymous]

Thanks Grace

What is the suggested sample size and how to set up the samples?

[Grace_Brebner3]

Sample size should be large enough for you to be confident that the result is statistically significant. How big that is will be somewhat dependent on your typical open/click rates. There's plenty of resources online that discuss this.

To set this up you'd just add two choices to the send email flow step, enter your sample size (e.g. 50%, 20% etc), and choose the email variants for each, then put do nothing as the default choice. You'd check the report after a long enough period had elapsed, and then send the "winner" to all those remaining who didn't get either test variant.

[Eric_Salamon1]

Vipula,

First I want to understand: Are you trying not to do it as an AB test? Because that would take a random sample.

If the answer is yes, then I would export the list then sort it by your the Marketo ID this will randomize it. After which you cut the list into sizes that would work for you. Usually a good rule of thumb that Marketo follows is a bare minimum of 5% to one list and 5% to the other. I think that it should very on the size of your list. Personally I would feel comfortable sending it to 1% or 2% given 1500 people. But truly it is up to you and the level of error you feel comfortable with.

Below the for statistical relevance.

The formula is s = √∑((x_i – µ)²/(N – 1)).

• s is the standard deviation.
• ∑ indicates that you will sum all of the sample values collected.
• x_i represents each individual value from your data.
• µ is the average (or mean) of your data for each group.
• N is the total sample number.

[SanfordWhiteman]

sort it by the Marketo ID this will randomize it

That's pretty darn deterministic, as leads added earlier in your instance will be favored in your tests.

Using the Marketo Unique Code would provide acceptable randomness.

But if you're in Excel anyway, might as well add a RAND() column and sort on that.

[Eric_Salamon1]

Nice call and true, but I was thinking in simplistic ways again it depends on how random and the degree of error.

[SanfordWhiteman]

Yeah, most people don't even know what they're asking for when they say "random" anyway.

[Veronica_Holme4]

Honestly - with all due respect to the creative answers above - aren't we overcomplicating things here? You say you want to do a batch blast within a default program and check the best open rate - without having any more details about what else you're trying to do, this seems like a textbook case for using the Email Program as a nested local asset to do the email send and subject line A/B test setting the sample size to 100% if you just want to blast everyone rather than using the "choose the winner" functionality, and then if you need to use a smart campaign to update statuses in the parent Default program. That gives you the default program, and the email program dashboard. Am I missing something here?

I find random sample very useful when I am trying to create a randomised static lists, but it isn't going to help you to do a good A/B test. Just my 2 cents

[Anonymous]

Hey Veronica, quick question - the solution you are proposing would only work for a 1-time batch right? I.e. It will not work if I need to add people to the Email Send program continously, i.e. anytime somene is added a list, add them to an Email Send program and send them an AB test. Seems like the Email program can only be configured for a batch send. Many thanks for confirming if you know a solution for this other than using a random sample in a default program.