A/B Testing - Different Send Volumes

[Zack_Bakht]

Hello,

We have had a few different email programs send a different amount of emails to our A variant and our B variant, and was curious to see if anyone knows why that might be happening. For example, a recent date/time test we ran had 67,000 emails delivered to our A variant, and 41,000 emails delivered to our B variant. This one specifically was a morning vs. afternoon send; this set of leads received their most recent email before this all at the same and day, leading me to believe our communication limits are not the issue. Any ideas would help.

Thanks!

[Priyank_Joshi3]

Hi Zack,

Can you please share the A/B test setup that you have used on your email programs? Per my understanding Date/Time A/B test might have been setup in a way that 60% records receives email version-A while rest of the 40% receives email version-B on defined time. For your reference please refer product doc - Use "Date/Time" A/B Testing - Marketo Docs - Product Documentation

Thanks!

Priyank

[Zack_Bakht]

Hi Priyank,

Thanks for the reply. The test was setup as follows:

Variant A: Morning Send - May 1, 2019, 8:00 AM PDT

Variant B: Afternoon Send - May 1, 2019, 3:00 PM PDT

Test Sample Size: 50%

Winner Criteria: Opens

The Morning Send received 67k emails, Afternoon Send 41k. My understanding is that 50% of the available leads should have received the test, with each variant getting half of the available leads. My confusion is to why it is skewed in favor of the Morning Send; I figured if comm. limits were the issue, the Afternoon Send would be the higher number, due to more leads becoming available as the day passed.

Thanks,

-Zack

[Kevin_McMahon1]

Was there a separate email that went out between the morning and evening that could have affected part of this audience's communication limits?

[Zack_Bakht]

There were no other emails to that audience that day or that week.

[SanfordWhiteman]

Are you checking Delivered or Sent?

[Zack_Bakht]

Delivered, which is what the email send program shows. But I just checked the total numbers and the delivery rate was 95.4%, so even if all of the bounced emails were to one variant, it wouldn't make up the difference.

[SanfordWhiteman]

OK, so what does the Smart List look like exactly?

Since Marketo calculated 67K as the morning send, is it possible that 67K was the expected size of the afternoon send at the time of the morning send -- in other words, there were ~268K qualified leads total in the morning -- but other SCs deleted or invalidated 26K of those slated for the afternoon send by the time the afternoon send time rolled around? Just trying to cover some edges here.

[Zack_Bakht]

Sure Sanford, thanks for the help.

Smart List: Advanced Filters - 1 and 2 and 3 and 4 and (5 or (6 and 7))

1) Email Address is not empty

2) Lead Removal Segments (our typical lead removals, competitors, clients, etc.)

3) Unsubscribed is false

4) Global Segments is not Quarantine (something we do to remove false clicks)

5) Lifecycle Status is MCL

6) Lifecycle Status is MEL (Upper Funnel)

7) Job Title not contains Owner

Let me know if any of that causes confusion, I'm happy to elaborate.