Is it a 50/50 split when A/B testing landing pages?

Anonymous
Not applicable

Is it a 50/50 split when A/B testing landing pages?

I am running an A/B landing page test for the first time and wondering if the testing turns out to be about 50/50 on the two different test pages being displayed. 

In initial testing, it seems to favor the page I first converted. Just trying to get a sense for what I can expect. 

Thanks in advance! 
 

Tags (1)
6 REPLIES 6
Anonymous
Not applicable

Re: Is it a 50/50 split when A/B testing landing pages?

My understanding was that it should randomly load each page in the test group - so for a 2 page test group it should be 50/50. How big is your sample size (how many times have you loaded the page) and how skewed is it (60/40, 70/30, etc.)?
Anonymous
Not applicable

Re: Is it a 50/50 split when A/B testing landing pages?

It was a really small test pool (like 20) I was just seeing it skew (70/30) during testing so I was just concerned I'd see a huge skew later on. 
Anonymous
Not applicable

Re: Is it a 50/50 split when A/B testing landing pages?

Using a binomial distribution, that's a 3.7% probability. I recommend using a larger sample size and then opening a support case if you're still seeing strange results.
Anonymous
Not applicable

Re: Is it a 50/50 split when A/B testing landing pages?

I plan on using a larger sample side when the test goes live, just was starting with some internal testing. 
Anonymous
Not applicable

Re: Is it a 50/50 split when A/B testing landing pages?

How large of a sample size would one need for a true 50/50 split?
Grant_Booth
Level 10

Re: Is it a 50/50 split when A/B testing landing pages?

There's not really a certain size to guarantee that it splits 50/50. It could be a 50/50 split with a sample size of 10 - it's just that a large skew becomes less probable with increasingly large sample sizes.

The rule of thumb in statistics is to use a sample size of at least 30 - otherwise a lot of statistical methods aren't very reliable. However, this is fairly arbitrary. It's technical possible that with a sample size of 100 you could see 70/30. It's just very unlikely.