An A/B test sent the "winner" email to some leads even though the test sample size was 100%

A document by 1507095077414__vishals@grazitti.com on behalf of Miles Ybaben on Jul 5, 2017Last modified by Vishal Sharma on Jun 26, 2018
Version 4Show Document
  • View in full screen mode

Issue Description
You created a batch A/B test with a sample size of 100%.  Even though all the leads should have been sent the test email, some leads still receive the winner.

 

Issue Resolution
Email Batch Programs are built in such a way that the test is sent with an background 'exploration' Smart Campaign, and the winner is sent with a background 'exploitation' campaign.  The exploitation campaign will send the "winner" email to the following Smart List:

  • Member of Program
  • Not Sent Test Email

 

There are two situation where the exploration campaign may not send to all the leads in the program, even when the sample size for the test is 100%.

 

  • If there are duplicate leads in the program, the program will not send the email to the same address twice.  It will send the test email to the first lead and skip all following leads with the same email address.  Those skipped leads will then qualify for the "winner"/exploitation campaign and will receive the winner.
  • If there are leads in the program who hit their Communication Limits while the "test"/exploration is running, they will be skipped.  When the "winner"/exploitation campaign runs, if those leads are not at their Communication Limits at that time, they will be sent the winner email because they were not sent the test email.

Is this article helpful ?

YesNo


Attachments

    Outcomes