Best practice for test sample size on A/B email test

[Anonymous]

What is a good guideline for size of the test group? My smart list is approximately 80k people. Is 5% (4,000) too low for a test group? Would appreciate any benchmarks or best practices that people have used. Thank you.

[Josh_Hill13 · Accepted solution]

Alisa,
Typically statistics will provide the correct sample size. A minimum of 30 people is a rule of thumb, but for larger populations like yours, you should calculate the proper sample size.
http://www.surveysystem.com/sscalc.htm

yields 382 at 95% confidence at +/- 5%
if we narrow it to +/- 1% then you need 8575

Generally Marketo's default of 10% should work for most situations. Keep in mind that if you choose 5%, only 2.5% will receive A or B. Thus, you might want to keep the smallest sample at 800-1000.

[Anonymous]

Thanks Josh - very helpful!