Reporting on A/B testing through Email Programs

[Jmhayes]

Hey there, 

I'm planning some extensive A/B testing in 2020 and considering the benefits of setting up AB tests in an Email Program vs. a Smart Campaign. 

We are leaning towards setting up AB tests through Email Programs, as the functionality/flow is a bit more user-friendly. 

However: I was wondering if there is an easy way to pull vanity & performance metrics (ie. Open Rate, CTR, Bounce Rate, etc) for emails that are run through an AB test through Email Programs -- for each email/factor we're testing. The dashboard for the post mortem AB test from an Email Program is extremely helpful, but when you export the "raw data", it's quite confusing and hard to interpret (see sample below): 

For the purposes of ongoing reporting to other teams, it's important for us to be able to share out these core metrics (like open rates, CTR, etc.).

I know that you can pull performance by setting up an Email Performance Report, but this does not break out the performance of each email or factor we're testing (and treats the email as "one").

Does anyone have any suggestions for how to get these performance metrics when running AB tests through email programs?

[Grace_Brebner3 · Accepted solution]

Hey there,

Interestingly enough, you actually can do this with an email performance report.

If you go into the smart list component of the report and add a filter like "was delivered email", you'll see an option in the constraint list called "Test Variant". 

This will result in a report that only shows one of the versions sent. You can then create another report within the program that constrains to the other version(s).

Hope that helps 🙂

[Jmhayes]

This is great!! Thank you for your help on this, Grace.