A/B Test Limitations

Craig_Wu
Level 1

A/B Test Limitations

Hi,

We wish to conduct an A/B Whole Email test with 16 versions. Is there any limitations on the number of versions that can be sent in any given test? What are some considerations to factor prior to sending this test?

Thanks!

5 REPLIES 5
Dan_Stevens_
Level 10 - Champion Alumni

Re: A/B Test Limitations

Do you have an extremely large sample size?  I only ask because it may be difficult to evaluate the most successful email with such a large number of test variants.  Even with a large sample size, I suspect it will still be difficult to understand what elements/content resulted in better conversions.  16 is a lot.

Craig_Wu
Level 1

Re: A/B Test Limitations

Hi Dan,

Thanks for the quick reply. The size is not particularly large: 46,600. I understand that 16 versions is quite a lot. I didn't plan this test, just trying to make sure that it will work as requested and to mitigate any technical problems that we may encounter before we send.

Thanks again for any additional advice on technical concerns.

Craig

Dan_Stevens_
Level 10 - Champion Alumni

Re: A/B Test Limitations

Craig - I believe you're limited to only two emails when testing "whole emails".  Unlike other tests - like "subject line" testing - there's not "+" icon to add more test variants.

pastedImage_0.png

pastedImage_1.png

Craig_Wu
Level 1

Re: A/B Test Limitations

Hi Dan,

I actually went ahead and tried to attach all 16 email versions to the A/B test. When doing the Whole Email test, you need to click "Add" to attach the whole email before it opens another Variation field to add assets. I reached the limit once I added 10 versions.

10 is the limit to A/B Whole Email testing.

Craig

2018-02-06 12_09_55-TR1_FY18Q3 Winter Travel.Whole Emails Test.png

Dan_Stevens_
Level 10 - Champion Alumni

Re: A/B Test Limitations

Ah, good to know.  Apparently I didn't test thoroughly enough.