SOLVED

Re: Validation of AB Test Through Email Program

Go to solution
Anonymous
Not applicable

Validation of AB Test Through Email Program

Hello,

I am looking to do an A/B test through an Email Program utilizing nine different variants. I have this setup, but want to ensure the test sample size is setup correctly. This will be a one time send, and I want ALL templates to get an equal share of the contacts. With my current understanding of how this should work, I want to set the "Test Sample Size" to 100%. I'm just looking for validation that this is the correct way to set this up.

AB Test Review.PNG

Tags (2)
1 ACCEPTED SOLUTION

Accepted Solutions
Devraj_Grewal
Level 10 - Champion Alumni

Re: Validation of AB Test Through Email Program

David,

What you have set up now means that all leads in the email program receive one of the nine variants, however no one will receive the winning email once that is determined (ex. most clicks). Per design, if you change the test sample size to 33%, that fraction will receive the variants and then once a winning email is determined, the other 66% will only get the winning variant.

Since you want all variants to be sent evenly. I would instead just use a smart campaign and set the flow of send email: random sample (% of whatever 1/9 is) and select all nine variants for a total of nine choice conditions. Since you will not be sending the winning email anyway and just want each email to be sent evenly. You can then create an email performance report that compares those nine emails together.

View solution in original post

3 REPLIES 3
Devraj_Grewal
Level 10 - Champion Alumni

Re: Validation of AB Test Through Email Program

David,

What you have set up now means that all leads in the email program receive one of the nine variants, however no one will receive the winning email once that is determined (ex. most clicks). Per design, if you change the test sample size to 33%, that fraction will receive the variants and then once a winning email is determined, the other 66% will only get the winning variant.

Since you want all variants to be sent evenly. I would instead just use a smart campaign and set the flow of send email: random sample (% of whatever 1/9 is) and select all nine variants for a total of nine choice conditions. Since you will not be sending the winning email anyway and just want each email to be sent evenly. You can then create an email performance report that compares those nine emails together.

Dory_Viscoglio
Level 10

Re: Validation of AB Test Through Email Program

Seconding Devraj's answer here! The setup above will send 100% of your audience your champion email, and none will receive the challenger. I also would do a random sample split, 11% for 8 of the groups, and the final group will be 12%.

Anonymous
Not applicable

Re: Validation of AB Test Through Email Program

Thanks, Devraj! This is the route I took in setting this up.