5 Replies Latest reply on Oct 9, 2018 9:02 AM by Valerie Whiting

    Subject Line - A/B Test

    Ad Ops

      How do you determine the winner if the criteria is Open Email. If it's based on the count of open email during the testing, can someone explain why on our testing the winner was Subject B but Subject A performed better.


      These are the result:

      Subject A - Open Email: 303

      Subject B - Open Email : 299

        • Re: Subject Line - A/B Test
          Valerie Whiting

          It is based on the open rate. Which is calculated by # of opens/# delivered. So, it is possible for the B test group to win if it had a smaller delivery group. Hope this helps!

            • Re: Subject Line - A/B Test
              PJ Walerysiak

              Hi Valerie,


              I'm having a similar issue. Though even with a smaller delivery group, my B subject line performed worse than A and was still declared the winner. Please see below:


              Subject Line A - 29 opens /// 586 delivered = 4.95% open rate

              Subject Line B - 24 opens /// 584 delivered = 4.11% open rate


              And if it somehow matters, A had 5 clicks while B had 4. It looks like in every way, B performed worse.


              Also, the Marketo A/B test winner notification shows B as winning with some absurdly vague "Opens" number:

              This notification is the only "solid" reasoning I've found for B winning the test, but I can't figure out in the slightest how this number is calculated -- and I've come up blank in combing the forums thus far.. Can you provide any insight on this?


              Thank you!