Subject Line - A/B Test

Anonymous
Not applicable

Subject Line - A/B Test

How do you determine the winner if the criteria is Open Email. If it's based on the count of open email during the testing, can someone explain why on our testing the winner was Subject B but Subject A performed better.

These are the result:

Subject A - Open Email: 303

Subject B - Open Email : 299

5 REPLIES 5
Valerie_Whitin2
Level 5

Re: Subject Line - A/B Test

It is based on the open rate. Which is calculated by # of opens/# delivered. So, it is possible for the B test group to win if it had a smaller delivery group. Hope this helps!

Anonymous
Not applicable

Re: Subject Line - A/B Test

Hi Valerie,

I'm having a similar issue. Though even with a smaller delivery group, my B subject line performed worse than A and was still declared the winner. Please see below:

Subject Line A - 29 opens /// 586 delivered = 4.95% open rate

Subject Line B - 24 opens /// 584 delivered = 4.11% open rate

And if it somehow matters, A had 5 clicks while B had 4. It looks like in every way, B performed worse.

Also, the Marketo A/B test winner notification shows B as winning with some absurdly vague "Opens" number:

pastedImage_0.png

This notification is the only "solid" reasoning I've found for B winning the test, but I can't figure out in the slightest how this number is calculated -- and I've come up blank in combing the forums thus far.. Can you provide any insight on this?

Thank you!

PJW

Ronen-Was-SRpro
Level 7

Re: Subject Line - A/B Test

Hi,

You can choose what will be the winner criteria when you are making the A/B test.

pastedImage_1.png

Define the A/B Test Winner Criteria - Marketo Docs - Product Documentation

Ronen Wasserman
Anonymous
Not applicable

Re: Subject Line - A/B Test

Hi Ronen,

Thank you but I am well aware of that, and that does little in regards to answering my inquiry above. What I'm not sure of is how Marketo declared B the winner, when A clearly had more opens and a higher open rate - with the established A/B Winning Criteria to be opens.

And I'm curious how Marketo calculates this arbitrary "opens" number that it reports in the A/B Test Winner Notification. I've recently had the same exact problem in another deployment, where the "loser" of the test (aka less opens) is declared the winner.

Valerie_Whitin2
Level 5

Re: Subject Line - A/B Test

Hi PJ Walerysiak

That does sound like odd behavior. Have you spoken to support yet? I've recently had some issues in our A/B reporting as well which they said they are working on fixing...not sure if it is related or not though!