SOLVED

Re: Best practice for test sample size on A/B email test

Go to solution
Anonymous
Not applicable

Best practice for test sample size on A/B email test

What is a good guideline for size of the test group? My smart list is approximately 80k people. Is 5% (4,000) too low for a test group? Would appreciate any benchmarks or best practices that people have used. Thank you.
Tags (1)
1 ACCEPTED SOLUTION

Accepted Solutions
Josh_Hill13
Level 10 - Champion Alumni

Re: Best practice for test sample size on A/B email test

Alisa,
Typically statistics will provide the correct sample size. A minimum of 30 people is a rule of thumb, but for larger populations like yours, you should calculate the proper sample size.
http://www.surveysystem.com/sscalc.htm

yields 382 at 95% confidence at +/- 5%
if we narrow it to +/- 1% then you need 8575

Generally Marketo's default of 10% should work for most situations. Keep in mind that if you choose 5%, only 2.5% will receive A or B. Thus, you might want to keep the smallest sample at 800-1000.

View solution in original post

2 REPLIES 2
Josh_Hill13
Level 10 - Champion Alumni

Re: Best practice for test sample size on A/B email test

Alisa,
Typically statistics will provide the correct sample size. A minimum of 30 people is a rule of thumb, but for larger populations like yours, you should calculate the proper sample size.
http://www.surveysystem.com/sscalc.htm

yields 382 at 95% confidence at +/- 5%
if we narrow it to +/- 1% then you need 8575

Generally Marketo's default of 10% should work for most situations. Keep in mind that if you choose 5%, only 2.5% will receive A or B. Thus, you might want to keep the smallest sample at 800-1000.
Anonymous
Not applicable

Re: Best practice for test sample size on A/B email test

Thanks Josh - very helpful!