100% Test Sample Size in Email A/B Testing

[Chelsey_Davis]

Howdy!

I have a quick question...

I have an email test with 2 subject lines and 2 tweaks to copy ( a 2 x 2 test) with a total population of 160,000 emails. I want to set the test up like this (whole email test):

Email 1 (Subject Line A, Copy A)

Email 2 (Subject Line B, Copy A)

Email 3 (Subject Line B, Copy A)

Email 4 (Subject Line B, Copy B)

If I'm just looking for data and not necessarily a 'winner' and will be uploading a static list for this test, can I set the test sample size to 100% without any issues?

Thanks!

[Devraj_Grewal · Accepted solution]

Chelsey,

The easiest way to ensure this is done correctly is to just duplicate the emails so there are 4 in total, each with their own subject line/copy combination. Then create a smart campaign and have a flow step like this:

1. Send Email

     Choice 1: if random sample is 25%: send Email 1

     Choice 2: if random sample is 25%: send Email 2

     Choice 3: if random sample is 25%: send Email 3

     Choice 4: if random sample is 25%: send Email 4

     Default Choice: do nothing

This ensures every person in the list is only sent one variation of the email and each email is sent to about the same size of the list (25% of 160,000).

[Chelsey_Davis]

Thanks, Devraj. I was trying to keep it in an email program, but your route seems like a cleaner option.

[Devraj_Grewal]

It's much easier for reporting purposes as well just to have each email variation created as a separate email then you don't need to mess with the test variant constraints.

[Chelsey_Davis]

That's very true. Thanks, Devraj!