- Marketing Nation
- :
- Products
- :
- Product Discussions
- :
- A/B testing question regarding sample size

Your Achievements

Next /

Sign inSign in to Community to gain points, level up, and earn exciting badges like the new Applaud 5 BadgeLearn more!

View All BadgesSign in to view all badges
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

06-14-2016
07:34 AM

06-14-2016
07:34 AM

A/B testing question regarding sample size

I recently set up an A/B email test where I thought 50% of my list was getting the A/B test, and then the remaining 50% would get the winner, but it appears that only 50% of my entire list received the test and winner. My smart list size was 78K, so I was under the assumption that 50% of that would be getting the A/B test, and the remaining would get the winner. However it only appears that 31K names were delivered in total. So with A/B testing, when you choose a sample size, does that really mean that's the total % that will receive both the test and the winner? If that's the case, if I wanted to do an A/B test to the entire list, would I set the sample to 100%? And if it's set to 100%, does that mean 50K will receive the A/B test and after the testing period 50K will receive the winner? Or does it just mean 50K will receive email A and 50K will receive email B?

Thanks!

6 REPLIES 6

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

06-14-2016
08:12 AM

06-14-2016
08:12 AM

Re: A/B testing question regarding sample size

Diane,

Sample size means that 50% of the list is selected to receive BOTH versions. So 25% randomly get A, the other 25% get B.

AFTER the winner is determined in the sample, the *remaining* 50% will receive the winner.

This is in the docs.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

06-14-2016
08:20 AM

06-14-2016
08:20 AM

Re: A/B testing question regarding sample size

Thanks Josh. So just to confirm, if my full list size was 78K and I chose a sample size of 50%, then 39K should have received the A/B split, and then the remaining 39K should have received the winner, correct? Because if that's the case, my test did not go out correctly, as my total delivered quantity was only 31K. Thanks again for the quick response.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

06-14-2016
08:15 AM

06-14-2016
08:15 AM

Re: A/B testing question regarding sample size

Hello Diane,

When you set the Sample size, that is the percentage of the list that receives the test, not the percentage that receives the whole thing. So, of your 78K leads, 50% should have received the test and the remaining 50% should have received the winner. If only 31K of your 78K leads received any email at all from the A/B test, test or winner, you will probably need to contact support for help determining why.

Anonymous

Not applicable

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

06-14-2016
09:00 AM

06-14-2016
09:00 AM

Re: A/B testing question regarding sample size

Josh is correct in how it will work. The only thing I can think of is that a lot of your names may be blocked do to opt outs, bounced email addresses? Or the test is not yet complete.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

06-14-2016
09:09 AM

06-14-2016
09:09 AM

Re: A/B testing question regarding sample size

Turns out there was a communications limit set on the campaign that I was unaware of, and this is why the remainder of the leads were not sent to. Thanks for your help!

Anonymous

Not applicable

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content

06-14-2016
11:52 AM

06-14-2016
11:52 AM

Re: A/B testing question regarding sample size

Oh those communication limits! The answer to most of my low send lists