A/B Testing & Statistical Significance

[Anonymous]

I'm a relatively new user, so be gentle!

I'm wondering if there is a way to do A/B testing in Marketo such that the system automatically sends the winning email the moment testing identifies a winner with a predetermined level of statistical significance.  In other words, the standard method has us identify a pre-set % of our target list to send the email.  Say we have a list of 10,000...we can select a winner by sending the variations randomly to 20%, or 2,000.  However, this could be a wasted opportunity.

Imagine an extreme (and extremely simplified) scenario where 100% open version A and 0% open version B.  In that case (without doing any math), let's imagine that we can be 95% confident after seeing the results from only 500 emails (250 for A and 250 for B), that A is superior.  At that point, if we continue sending out B for the purposes of confirming what we already know, we're effectively wasting 750 opportunities(half the remaining 1,500 sample).

So my question is this: is there a way for marketo to choose a winner based on statistical significance if that can be established prior to the pre-determined sample size?

Thanks you in advance,
Dave Gimpel

[Anonymous]

Email Programs.

Add one: https://community.marketo.com/MarketoArticle?id=kA050000000LC8FCAW
Schedule one: https://community.marketo.com/MarketoArticle?id=kA050000000LCSVCA4
Define one: https://community.marketo.com/MarketoArticle?id=kA050000000LCMXCA4

If you do a search for Email Programs A/B Testing, you should be able to pull up all of the other references as there are many that you'll want to go through in order to gain an understanding of how it all comes together.  

[Anonymous]

Respectfully, that didn't answer my question at all.  I do know how to set up a simple A/B test (including scheduling and defining the winner criteria), but that wasn't the point of my original question.  At your convenience, please take a second look at my previous email and if you require clarification, please let me know where I'm being unclear.

Thanks,
Dave

[Anonymous · Accepted solution]

I'll be gentle: No, there is not.

But you should suggest it as an idea in the community.

[Anonymous]

That is a bummer, and I think I will.

Thanks!

[Anonymous]

Hi David / All,

Athough you cannot detersome a statistically significant sample size in Marketo, if it helps anyone, here are some online tools and calculations I am using to determine sample size (audience %) to put into Marketo.

I am going to use an external sample size calculator. Here is one: http://www.surveysystem.com/sscalc.htm#one. 
An email A/B testing article suggested using a confidence interval (margin of error) of 5%, and the 95% confidence level. After you get the sample size by the total number of emailable contacts in your list. If you use these numbers, and 60% of your sample opens Email A, you can be 95% sure that between 55% (60 minus 5) and 65% (60 plus 5) would have also opened Email A.

Hope this helps someone! 

 

[Phillip_Wild]

Hi David

As others have mentioned, there is no way to do this within Marketo - and I haven't seen it in other marketing automation software either. I love that you are into the subject since A/B testing is one thing, A/B testing with proper science behind it is completely another.

One thing Marketo lacks that is much simpler to implement is a p value calculator for the results. We always see "A is the winner since it had 1100 opens, while B only had 1050 opens". But in reality, as you probably know, that result might be completely meaningless if the sample size wasn't large enough, and might be purely random variation.

Here's a calculator I use to determine whether my results are statistically significant after the test is run: A/B Test Calculator: MeasuringU

Estimating the sample size, as Rhianne Makovec​ mentioned, can be done with online calculators. But - this requires estimating the proportions ahead of time, which is a bit of educated guesswork. So it's always worthwhile analysing the results for statistical significance after the fact to make sure the effect you see is likely the result of the test.

Phil