Select A/B Test Winner by Statistical Significance

[Anonymous]

I propose an A/B testing winner selection method in Marketo such that the system automatically sends the winning email the moment testing identifies a winner with a predetermined level of statistical significance.  In other words, the standard method has us identify a pre-set % of our target list to send the email.  Say we have a list of 10,000...we can select a winner by sending the variations randomly tosay, 20%, or 2,000.

However, this could be a wasted opportunity.

Imagine an extreme (and extremely simplified) scenario where 100% open version A and 0% open version B.  In that case (without doing any math), let's imagine that we can be 95% confident after seeing the results from only 500 emails (250 for A and 250 for B), that A is superior.  At that point, if we continue sending out B for the purposes of confirming what we already know, we're effectively wasting 750 opportunities (specifically, the inferior B emails from the remaining 1,500 sample).

Assuming a minimum sample size is reached, and the confidence level is achieved, the AB test is cut short, and the winning email is sent to the full remainder of the list.  There are issues that would need to be resovled with this method, but I would be interested in working them out with community members in philosophical allignment with me.

Cheers,
Dave

[Phillip_Wild]

Hi Dave

I like your approach, but I'd like a small step in between.

When I evaluate A/B test results in the Email Program Dashboard view, I would love to see the level of statistical significance. At the moment, I use http://www.measuringu.com/ab-calc.php, and manually plug in the opens/clicks and delivered to get my stats.

I think it's potentially super misleading for those not familiar with statistics (most of us) to show a "winner" and a "loser" based purely on a raw number. We do a lot of smaller sends, to under a thousand users, and these won't show statistical significance unless you really change something radical. But if you were going by the winner, then you would often decide that one option is "better" than another prematurely, and sometimes incorrectly.

To build this in Marketo, all you could need is a small calculation in the background. Should be easy functionality, no?

Phil