A/B Test Limitations

[Craig_Wu]

Hi,

We wish to conduct an A/B Whole Email test with 16 versions. Is there any limitations on the number of versions that can be sent in any given test? What are some considerations to factor prior to sending this test?

Thanks!

[Dan_Stevens_]

Do you have an extremely large sample size?  I only ask because it may be difficult to evaluate the most successful email with such a large number of test variants.  Even with a large sample size, I suspect it will still be difficult to understand what elements/content resulted in better conversions.  16 is a lot.

[Craig_Wu]

Hi Dan,

Thanks for the quick reply. The size is not particularly large: 46,600. I understand that 16 versions is quite a lot. I didn't plan this test, just trying to make sure that it will work as requested and to mitigate any technical problems that we may encounter before we send.

Thanks again for any additional advice on technical concerns.

Craig

[Dan_Stevens_]

Craig - I believe you're limited to only two emails when testing "whole emails".  Unlike other tests - like "subject line" testing - there's not "+" icon to add more test variants.

[Craig_Wu]

Hi Dan,

I actually went ahead and tried to attach all 16 email versions to the A/B test. When doing the Whole Email test, you need to click "Add" to attach the whole email before it opens another Variation field to add assets. I reached the limit once I added 10 versions.

10 is the limit to A/B Whole Email testing.

Craig

[Dan_Stevens_]

Ah, good to know.  Apparently I didn't test thoroughly enough.