I am running an A/B landing page test for the first time and wondering if the testing turns out to be about 50/50 on the two different test pages being displayed.
In initial testing, it seems to favor the page I first converted. Just trying to get a sense for what I can expect.
There's not really a certain size to guarantee that it splits 50/50. It could be a 50/50 split with a sample size of 10 - it's just that a large skew becomes less probable with increasingly large sample sizes.
The rule of thumb in statistics is to use a sample size of at least 30 - otherwise a lot of statistical methods aren't very reliable. However, this is fairly arbitrary. It's technical possible that with a sample size of 100 you could see 70/30. It's just very unlikely.
Using a binomial distribution, that's a 3.7% probability. I recommend using a larger sample size and then opening a support case if you're still seeing strange results.
My understanding was that it should randomly load each page in the test group - so for a 2 page test group it should be 50/50. How big is your sample size (how many times have you loaded the page) and how skewed is it (60/40, 70/30, etc.)?
Adchoices