I love the new A/B test feature -- it's great for A/B testing emails that will end up in engagement streams, while still a single person from recieving the same email twice.
However, the downside of these emails not showing up in reports is a real bummer.
We export all A/B email results into an excel sheet and make notes on what we learned from each test. Whenever developing an email, we look back at similar campaigns in the past to see what we learned and this guides our strategy.
Yes it is a bummer. A real blocker to what is otherwise a great new feature.
I'm told that this is not just limited to individual email tests results but the entire email. If an email is used in an A/B test it cannot be returned in a traditional email performance report. That is a huge oversight that needs to be fixed. Without the ability to compare different emails side by side we will have to consider going back to the old clunky method of splitting groups in a flow step.
Totally agree, Jack. Good news though -- Marketo's support team is second to none. I reached out and here's what i heard back:
"My name is James and I am happy to help.
This is a known issue we are currently working to resolve. There is an open engineering case, and I am monitoring the progress. It looks like from last update they are running some test fixes and hope to have a working model in February.
I do not have a more exact time frame at this time. We appreciate the patience while we work to resolve this issue, and apologize for any inconvenience at this time. "
So it looks like we can keep using in and hopefully we'll have a fix in late february.
It would be great to have a summary report of A/B tests in one place. I just tookover Marketo at my company and it would be nice to look back in time and see what tests they have executed, what the results were, etc.
.png)