Champion Challenger Sample Size

[Anonymous]

RE: An email champion/challenger test in a default program

During the setup, we identify which percentage will receive the Champion vs. the Challenger/s. This led me to believe that the entire pool of recipients identified in my Smart List would be split based upon these percentages. Until you "Declare a Winner" - both the Champion and Challenger will continue to send at the proportion you have chosen. My question is: if there are still records that have not been sent either version when I declare a winner-how did Marketo determine who to send my test to in the first place? 24 hours after my test, it had sent to approx 8,000 of 11,000 records.

[Josh_Hill13]

Are you doing this in a Trigger smart campaign, an Engagement, or using the Email Send?

What is the expected volume or actual volume through that email?

A screenshot or two would be helpful. Your description isn't clear about which steps occurred and why you are concerned about the proportion vs. volume.

[Anonymous]

Thank you. It was a Default Program with a batch send assisted with the Campaign is Requested.  I set it up according to the Product Docs and what Dan described below. Expected volume was 11,000. I am not sure what screenshot would help to clarify?

My concern was that 24hr after the send, right before I declared the winner, Marketo had only sent to approx 8,000 of 11,000 records. Since there is no "test phase" with C/C, I fully expected that 24 hours later Marketo would have sent either the Champion or Challenger to all 11,000 based on the distribution chosen during set up.

[Dan_Stevens_]

The way Champion/Challenger works for default programs (unlike separate Email Programs, which have more automated A/B testing; and support batch sends (C/C only support trigger sends)) - which you indicated here - is your current email is considered the Champion, until proven otherwise.  It's important to remember that you have to manually declare the winner (again, unlike Email Programs).

Here's an example A/B test based on subject line:

Let's say 11,000 leads qualified for the trigger campaign.  40% of those leads will have received the email with the subject line "Subject Line Test B" (4,400), while the remaining leads (6,600) will have received the one with "Existing Subject Line".  Since the Champion criteria is % based, it's fine that the test variants aren't equal.  Although, when using C/C, I tend to go with a more even distribution between the two test groups like I show above.  This distribution will continue until you declare a winner - at which time, 100% of the leads will now receive the Champion.

I think where some people get confused with C/C in a default Program (vs. A/B testing in an Email Program) is DISTRIBUTION vs. TEST SAMPLE SIZE:

• A/B testing using C/C in a default program isn't doing any pre-testing against a small sample size.  Everyone that qualifies is receiving one of the emails (either the existing one (the current champion) or the challenger.  The % defined isn't a sample size, but rather how the email will be distributed to any lead that qualifies in your trigger campaign send.

    

• A/B testing in an Email Program, on the other hand, only sends out 50/50 tests (if you're doing a test with only two variants) to the small sample size defined below.  And then it will send out the entire batch using the winning email on the date/time pre-defined.  Automatically.

    

Now to your specific question:

if there are still records that have not been sent either version when I declare a winner-how did Marketo determine who to send my test to in the first place? 24 hours after my test, it had sent to approx 8,000 of 11,000 records.

It sounds like you're trying to do some sort of batch sending of emails.  C/C doesn't work for batch sends - only trigger sends.  By chance are you using a trigger campaign - with a "Campaign is Requested" trigger - in conjunction with a batch campaign to work around this? 

Before you declared a winner, all (100%) of your leads were receiving a version of the email (either champion or challenger) - there is no "test" phase where only a small percentage of your audience receives one or the other (which is what happens using the A/B testing in Email Programs).

[Anonymous]

Thanks Dan. Yes, a Campaign is Requested/Batch was used to get around the limitation on the send. My concern was that 24hr after the send, right before I declared the winner, Marketo had only sent to approx 8,000 of 11,000 records. Since there is no "test phase" with C/C, I fully expected that 24 hours later Marketo would have sent either the Champion or Challenger to all 11,000 based on the distribution chosen during set up.