Validation of AB Test Through Email Program

[Anonymous]

Hello,

I am looking to do an A/B test through an Email Program utilizing nine different variants. I have this setup, but want to ensure the test sample size is setup correctly. This will be a one time send, and I want ALL templates to get an equal share of the contacts. With my current understanding of how this should work, I want to set the "Test Sample Size" to 100%. I'm just looking for validation that this is the correct way to set this up.

[Devraj_Grewal · Accepted solution]

David,

What you have set up now means that all leads in the email program receive one of the nine variants, however no one will receive the winning email once that is determined (ex. most clicks). Per design, if you change the test sample size to 33%, that fraction will receive the variants and then once a winning email is determined, the other 66% will only get the winning variant.

Since you want all variants to be sent evenly. I would instead just use a smart campaign and set the flow of send email: random sample (% of whatever 1/9 is) and select all nine variants for a total of nine choice conditions. Since you will not be sending the winning email anyway and just want each email to be sent evenly. You can then create an email performance report that compares those nine emails together.

[Dory_Viscoglio]

Seconding Devraj's answer here! The setup above will send 100% of your audience your champion email, and none will receive the challenger. I also would do a random sample split, 11% for 8 of the groups, and the final group will be 12%.

[Anonymous]

Thanks, Devraj! This is the route I took in setting this up.