A/B test reports are not making sense

[ThaísMacedo]

Hello, Marketo Community!

I'm having trouble figuring out how many people received an email that includes an A/B test. 

To give context, this triggered email went out for a few weeks without a challenger, then I added an AB test 50/50 around 1 month ago. 

If I pull out an email performance report, I get (sent):

Total: 6,523 

Version A: 14
Version B: 24

Now, that's when things start to get weird: if I create a smart list with the filter "was sent email", I get:

Total: 3,943

Version A: 981

Version B: 97

Do you have any idea why these numbers are so different? Is this a bug or am I doing something wrong?

[Christiane_Rode]

How long has your email been going out? The Send Email activity in a person's record is only retained for 90 days, so, depending on how long your email has been going out, you may see a discrepancy between the report and your smart list. You'll still see data in the email performance report, but "Was sent email" data over 90 days can't be pulled with a smart list.

[ThaísMacedo]

 Thank you for the answer. I think it solves part of the mystery (why there are more people on the smart list than on the report when we look at total numbers).

But still, there are more people in the smart list than on the report for each variant, which goes against this logic.

[Darshil_Shah1]

Hi @ThaísMacedo, 

Apologies if I'm missing out on anything here, but how are you splitting the email metrics by the test variants in a single email performance report? AFAIK the email metrics for all the test variants roll-up to the email asset itself (and are not broken by the test variants). There's also an idea on the community to add the ability to split the email performance data by the test variants.

[ThaísMacedo]

I'm using the Email Performance report + smart list (inside the report) with the filter A/B sample. 

[Darshil_Shah1]

A/B Sample filter? Did you mean Was Sent Email filter with constaint on individual Test Variant filter? Can you please share the filter snapshot, if possible?