An A/B test sent the "winner" email to some leads even though the test sample size was 100%

No ratings

Issue Description
You created a batch A/B test with a sample size of 100%.  Even though all the leads should have been sent the test email, some leads still receive the winner.

 

Issue Resolution
Email Batch Programs are built in such a way that the test is sent with an background 'exploration' Smart Campaign, and the winner is sent with a background 'exploitation' campaign.  The exploitation campaign will send the "winner" email to the following Smart List:

  • Member of Program
  • Not Sent Test Email

 

There are two situation where the exploration campaign may not send to all the leads in the program, even when the sample size for the test is 100%.

 

  • If there are duplicate leads in the program, the program will not send the email to the same address twice.  It will send the test email to the first lead and skip all following leads with the same email address.  Those skipped leads will then qualify for the "winner"/exploitation campaign and will receive the winner.
  • If there are leads in the program who hit their Communication Limits while the "test"/exploration is running, they will be skipped.  When the "winner"/exploitation campaign runs, if those leads are not at their Communication Limits at that time, they will be sent the winner email because they were not sent the test email.

Is this article helpful ?

YesNo


Labels (1)
Comments

Does this mean that, if we have duplicate leads in the program and we run a test with a 100% sample size, the duplicate emails will always get both test and winner?

Yes, that could happen.

Is there a way to determine the way that the duplicates are selected? in terms of order. For example, I want to send an email, and we have a known duplicate issue (being addressed in SFDC) Which duplicate will be the first sent to?

It will be the first one pulled through the Smart List, which you should be able to see by viewing the audience for the program.