An A/B test sent the winner email to some leads even though the test sample size was 100%

Issue


You created a batch A/B test with a sample size of 100%.  Even though all the leads should have been sent the test email, some leads still receive the winner.

 

 


Solution


Email Batch Programs are built in such a way that the test is sent with an background 'exploration' Smart Campaign, and the winner is sent with a background 'exploitation' campaign.  The exploitation campaign will send the "winner" email to the following Smart List:
  • Member of Program
  • Not Sent Test Email

There are two situation where the exploration campaign may not send to all the leads in the program, even when the sample size for the test is 100%.

  • If there are duplicate leads in the program, the program will not send the email to the same address twice.  It will send the test email to the first lead and skip all following leads with the same email address.  Those skipped leads will then qualify for the "winner"/exploitation campaign and will receive the winner.
  • If there are leads in the program who hit their Communication Limits while the "test"/exploration is running, they will be skipped.  When the "winner"/exploitation campaign runs, if those leads are not at their Communication Limits at that time, they will be sent the winner email because they were not sent the test email.

 


Labels (1)