I'm going to start by putting it out there that I really dislike email programs, but in order to give them a fair shot we ran an A/B test earlier today and I had some questions that maybe someone with more experience on these can answer...
1. Our total audience size is 153,000 and we set the test for a 10% test and 90% to receive the champion. Now, I would have expected this to mean that 15,300 emails would be sent and then the winner from that bunch would be sent to the remaining 137,700 people. Why, then, have over 20,000 emails been delivered??
1a. Marketo, are we even able to test to 100% of our list? I've tried this previously and seen about 75% sent out and 25% saved for the winner... If there's a limit here, please tell us this somewhere, and don't let us put the slider all the way up to 100%.
2. If this is an A/B test, why isn't it sent as a 50/50 split?
3. I'm noticing that the opens are an absolute number, which is leading me to believe that the champion is not going to be decided by the percentage of opens, but instead will be decided by the absolute number of opens of each email. Someone please tell me that I'm incorrect, and the winner will be chosen by the higher PERCENTAGE of opens. This relates to the point above, if the winner is chosen on an absolute number, then this test was screwed from the start when one subject line was sent to 2/3 of my test and only 1/3 received the other subject line.
Really disappointed...
1. Our total audience size is 153,000 and we set the test for a 10% test and 90% to receive the champion. Now, I would have expected this to mean that 15,300 emails would be sent and then the winner from that bunch would be sent to the remaining 137,700 people. Why, then, have over 20,000 emails been delivered??
1a. Marketo, are we even able to test to 100% of our list? I've tried this previously and seen about 75% sent out and 25% saved for the winner... If there's a limit here, please tell us this somewhere, and don't let us put the slider all the way up to 100%.
2. If this is an A/B test, why isn't it sent as a 50/50 split?
3. I'm noticing that the opens are an absolute number, which is leading me to believe that the champion is not going to be decided by the percentage of opens, but instead will be decided by the absolute number of opens of each email. Someone please tell me that I'm incorrect, and the winner will be chosen by the higher PERCENTAGE of opens. This relates to the point above, if the winner is chosen on an absolute number, then this test was screwed from the start when one subject line was sent to 2/3 of my test and only 1/3 received the other subject line.
Really disappointed...
.png)