6 Replies Latest reply on Sep 2, 2014 2:10 PM by Grant Booth

# Is it a 50/50 split when A/B testing landing pages?

I am running an A/B landing page test for the first time and wondering if the testing turns out to be about 50/50 on the two different test pages being displayed.

In initial testing, it seems to favor the page I first converted. Just trying to get a sense for what I can expect.

• ###### Re: Is it a 50/50 split when A/B testing landing pages?
My understanding was that it should randomly load each page in the test group - so for a 2 page test group it should be 50/50. How big is your sample size (how many times have you loaded the page) and how skewed is it (60/40, 70/30, etc.)?
• ###### Re: Is it a 50/50 split when A/B testing landing pages?
It was a really small test pool (like 20) I was just seeing it skew (70/30) during testing so I was just concerned I'd see a huge skew later on.
• ###### Re: Is it a 50/50 split when A/B testing landing pages?
Using a binomial distribution, that's a 3.7% probability. I recommend using a larger sample size and then opening a support case if you're still seeing strange results.
• ###### Re: Is it a 50/50 split when A/B testing landing pages?
I plan on using a larger sample side when the test goes live, just was starting with some internal testing.
• ###### Re: Is it a 50/50 split when A/B testing landing pages?
How large of a sample size would one need for a true 50/50 split?
• ###### Re: Is it a 50/50 split when A/B testing landing pages?
There's not really a certain size to guarantee that it splits 50/50. It could be a 50/50 split with a sample size of 10 - it's just that a large skew becomes less probable with increasingly large sample sizes.

The rule of thumb in statistics is to use a sample size of at least 30 - otherwise a lot of statistical methods aren't very reliable. However, this is fairly arbitrary. It's technical possible that with a sample size of 100 you could see 70/30. It's just very unlikely.
1 of 1 people found this helpful