Subject Line - A/B Test

[Anonymous]

How do you determine the winner if the criteria is Open Email. If it's based on the count of open email during the testing, can someone explain why on our testing the winner was Subject B but Subject A performed better.

These are the result:

Subject A - Open Email: 303

Subject B - Open Email : 299

[Valerie_Whitin2]

It is based on the open rate. Which is calculated by # of opens/# delivered. So, it is possible for the B test group to win if it had a smaller delivery group. Hope this helps!

[Anonymous]

Hi Valerie,

I'm having a similar issue. Though even with a smaller delivery group, my B subject line performed worse than A and was still declared the winner. Please see below:

Subject Line A - 29 opens /// 586 delivered = 4.95% open rate

Subject Line B - 24 opens /// 584 delivered = 4.11% open rate

And if it somehow matters, A had 5 clicks while B had 4. It looks like in every way, B performed worse.

Also, the Marketo A/B test winner notification shows B as winning with some absurdly vague "Opens" number:

This notification is the only "solid" reasoning I've found for B winning the test, but I can't figure out in the slightest how this number is calculated -- and I've come up blank in combing the forums thus far.. Can you provide any insight on this?

Thank you!

PJW

[Ronen-Was-SRpro]

Hi,

You can choose what will be the winner criteria when you are making the A/B test.

Define the A/B Test Winner Criteria - Marketo Docs - Product Documentation

[Anonymous]

Hi Ronen,

Thank you but I am well aware of that, and that does little in regards to answering my inquiry above. What I'm not sure of is how Marketo declared B the winner, when A clearly had more opens and a higher open rate - with the established A/B Winning Criteria to be opens.

And I'm curious how Marketo calculates this arbitrary "opens" number that it reports in the A/B Test Winner Notification. I've recently had the same exact problem in another deployment, where the "loser" of the test (aka less opens) is declared the winner.

[Valerie_Whitin2]

Hi PJ Walerysiak​

That does sound like odd behavior. Have you spoken to support yet? I've recently had some issues in our A/B reporting as well which they said they are working on fixing...not sure if it is related or not though!