Statistical significance for a A/B test

[Nitesh_Kumar]

Does it make sense to try out email A/B testing with smaller audience sizes, our email sends are usually towards small focused targets and we have been trying email experiments from last 2-3 months, the variation in results so far is very small to determine any significant winner, do we have a way where we can identify the statistical significance of an A/B test  based on the target audience size?

[Abhishek_Chand2]

Hi Nitesh,

A/B testing is actually more tricky than we think it is, and when the random sample is very small then the results we get not always very helpful in making a decision.

You can watch one of the most wonderful session by Jessica kao, during the Adobe summit. The session is called "Making the most out of A/B testing in MArketo". The session includes things we do not consider while doing the A/B testing in Marketo and if we implement them then the insights proves to be very helpful for the future email campaigns.

Please find the link below:

Adobe Summit 2019—The Digital Experience Conference | March 24–28, 2019 

[Saurabh_Tyagi1]

Hi Nitesh Kumar‌ - As mentioned in the question, I would support looking at the statistical significance of each, and all tests. This could have some bias but you would still be better informed. For example, this tool can help you calculate the p-value for tests.

Hope this helps!

[Carey_Picklesi7]

If you aren't able to increase the audience size, which it doesn't sound like you can, you may wish to consider the difference between the A/B variants. Are they dramatic enough to draw different results? Sometimes people use tests that are too similar so its hard to get different results. I would suggest using stronger variations when you have a smaller sample size, such as:

• Plain Text looking vs HTML
• Subject lines that focus on different topics - If you have an email that has a single topic, find 2 different angles to presenting it (Such as (a) Save money by doing these 3 things and (b)Read this case study to understand how improve your processes)