A/B testing question regarding sample size

[Diane_Schmid]

I recently set up an A/B email test where I thought 50% of my list was getting the A/B test, and then the remaining 50% would get the winner, but it appears that only 50% of my entire list received the test and winner.  My smart list size was 78K, so I was under the assumption that 50% of that would be getting the A/B test, and the remaining would get the winner.  However it only appears that 31K names were delivered in total.  So with A/B testing, when you choose a sample size, does that really mean that's the total % that will receive both the test and the winner?  If that's the case, if I wanted to do an A/B test to the entire list, would I set the sample to 100%? And if it's set to 100%, does that mean 50K will receive the A/B test and after the testing period 50K will receive the winner? Or does it just mean 50K will receive email A and 50K will receive email B?

Thanks!

[Josh_Hill13]

Diane,

Sample size means that 50% of the list is selected to receive BOTH versions. So 25% randomly get A, the other 25% get B.

AFTER the winner is determined in the sample, the remaining 50% will receive the winner.

This is in the docs.

[Diane_Schmid]

Thanks Josh.  So just to confirm, if my full list size was 78K and I chose a sample size of 50%, then 39K should have received the A/B split, and then the remaining 39K should have received the winner, correct?  Because if that's the case, my test did not go out correctly, as my total delivered quantity was only 31K.  Thanks again for the quick response.

[Roxann_McGlump1]

Hello Diane,

When you set the Sample size, that is the percentage of the list that receives the test, not the percentage that receives the whole thing.  So, of your 78K leads, 50% should have received the test and the remaining 50% should have received the winner.  If only 31K of your 78K leads received any email at all from the A/B test, test or winner, you will probably need to contact support for help determining why.

[Anonymous]

Josh is correct in how it will work.  The only thing I can think of is that a lot of your names may be blocked do to opt outs, bounced email addresses?  Or the test is not yet complete.

[Diane_Schmid]

Turns out there was a communications limit set on the campaign that I was unaware of, and this is why the remainder of the leads were not sent to.  Thanks for your help!

[Anonymous]

Oh those communication limits! The answer to most of my low send lists