Guidance on Launching A/B Test

[Anonymous]

Hello,

I am currently working to launch an A/B test with nine different email variations. I want to use the out of the box functionality Marketo has for this and believe I know how to set this up. HOWEVER, my contacts are contained within a Smart List. How do I set the A/B test up to to pull from the Smart List I have?

[Grégoire_Miche2 · Accepted solution]

Hi David,

Are you using an email program or a champion challenger A/B test ? If it's anemail program, just run it on your smart list. If a champion challenger, create a  batch smart campaign with the smart list as a target and use an send email flow step, using the test group as the email.

In any case, just send the email as usual, and let Marketo randomly assign each lead to a variant.

Now I would be a little suspicious about the relevance of a test with 9 variants. If you end up sending each variant to less than a few thousands of people, the statistical relevance of the result will be questionable.

Also, avoid testing multiple criteria on the same time, you'll have a harder time making sens of the results.

-Greg

[Anonymous]

Thanks, Greg. I'm working to set this up as a champion challenger A/B test (The option used by right clicking on the email and selecting add test). So once the A/B test is setup, and the variants defined, I just send the contacts the email in a Smart Campaign and Marketo will do the allocation? Just want to make sure I have this straight.

[Grégoire_Miche2]

Hi again David,

yes this is exactly it

-Greg

[Grégoire_Miche2]

and please do some quick math on how many leads will receive each variants and make sure the samples are large enough to get some valid info.

Ex: Let's say you have a database of 18000, and you send the test to 10% of it. This means the test is sent to 1800 leads and each variant is sent to 200 people. Imagine the average open rate is 2%, it means an average of 4 opens per variants. But if the standard deviation is not good, it may mean that some variants will have 0 opens, and that would be just the result of your lead sample, not of the tested email. In this case, this would prevent you to even test the clicks. This does not mean the variant is having bad results, it just mean your sample is too small for you to know

Of course, if you are sending the test on the half of a 10 000 000 records DB, all I wrote above is irrelevant

-Greg

[Josh_Hill13]

Please remember that Champ Challenger is for triggered campaigns or Engagements only.

[Anonymous]

Why can't it be used in other types of email sends?

[Grégoire_Miche2]

Hi Hanna,

This is a product limitation, clearly documented. See Add an Email Champion/Challenger - Marketo Docs - Product Docs

-Greg