Is it a 50/50 split when A/B testing landing pages?

[Anonymous]

I am running an A/B landing page test for the first time and wondering if the testing turns out to be about 50/50 on the two different test pages being displayed. 

In initial testing, it seems to favor the page I first converted. Just trying to get a sense for what I can expect. 

Thanks in advance! 
 

[Anonymous]

My understanding was that it should randomly load each page in the test group - so for a 2 page test group it should be 50/50. How big is your sample size (how many times have you loaded the page) and how skewed is it (60/40, 70/30, etc.)?

[Anonymous]

It was a really small test pool (like 20) I was just seeing it skew (70/30) during testing so I was just concerned I'd see a huge skew later on. 

[Anonymous]

Using a binomial distribution, that's a 3.7% probability. I recommend using a larger sample size and then opening a support case if you're still seeing strange results.

[Anonymous]

I plan on using a larger sample side when the test goes live, just was starting with some internal testing. 

[Anonymous]

How large of a sample size would one need for a true 50/50 split?

[Grant_Booth]

There's not really a certain size to guarantee that it splits 50/50. It could be a 50/50 split with a sample size of 10 - it's just that a large skew becomes less probable with increasingly large sample sizes.

The rule of thumb in statistics is to use a sample size of at least 30 - otherwise a lot of statistical methods aren't very reliable. However, this is fairly arbitrary. It's technical possible that with a sample size of 100 you could see 70/30. It's just very unlikely.