Re: Email Programs - Someone please set me straight...

Dory_Viscoglio
Level 10

Email Programs - Someone please set me straight...

I'm going to start by putting it out there that I really dislike email programs, but in order to give them a fair shot we ran an A/B test earlier today and I had some questions that maybe someone with more experience on these can answer... 

1. Our total audience size is 153,000 and we set the test for a 10% test and 90% to receive the champion. Now, I would have expected this to mean that 15,300 emails would be sent and then the winner from that bunch would be sent to the remaining 137,700 people. Why, then, have over 20,000 emails been delivered??

1a. Marketo, are we even able to test to 100% of our list? I've tried this previously and seen about 75% sent out and 25% saved for the winner... If there's a limit here, please tell us this somewhere, and don't let us put the slider all the way up to 100%.

2. If this is an A/B test, why isn't it sent as a 50/50 split? 

3. I'm noticing that the opens are an absolute number, which is leading me to believe that the champion is not going to be decided by the percentage of opens, but instead will be decided by the absolute number of opens of each email. Someone please tell me that I'm incorrect, and the winner will be chosen by the higher PERCENTAGE of opens. This relates to the point above, if the winner is chosen on an absolute number, then this test was screwed from the start when one subject line was sent to 2/3 of my test and only 1/3 received the other subject line. 


0EM50000000SLKJ.jpg

Really disappointed... 
Tags (1)
4 REPLIES 4
Anonymous
Not applicable

Re: Email Programs - Someone please set me straight...

Hi Dory,

Can you send me a link to the email program with these results.  Something doesn't seem right.

Thanks,
Cheryl
Dory_Viscoglio
Level 10

Re: Email Programs - Someone please set me straight...

Hi Cheryl, this is the link: https://app-sj05.marketo.com/#EBP1387A1 

The test is still currently running, and the winner is scheduled to be sent out at 3pm Eastern today. The results I grabbed were from right after most of the emails had been delivered yesterday, but the send distribution hasn't changed. 

Any insight that you can provide would be greatly appreciated... You're the reason I've tried to give the email programs a second chance!
Dory_Viscoglio
Level 10

Re: Email Programs - Someone please set me straight...

Well the winner was sent based on the percentage, not the absolute number, of opens which is good... But the skew of the sends didn't help my case with our team. 
Josh_Hill13
Level 10 - Champion Alumni

Re: Email Programs - Someone please set me straight...

subscribing. I've been seeing several related questions to this and would also like an answer.