A/B Testing and Dynamic Emails

[Anonymous]

Does anyone know about how A/B testing works with regard to dynamic emails?  More specifically, if you set up an email to have a random sample of 50, does each email get sent to 50% of each segment or 50% of the total list of recipients regardless of segment?

[Josh_Hill13]

Shauna,

It depends how you set this up from the start and how many segments are involved.

Smart List: Segmentation A with Segments 1 and 2

Email has content for Segment 1 and Segment 2

If you did

Flow: Send Email IF Random Sample = 50%, then Send Email A, Default: Send Email B

and both emails had just Default Segment, then a random group of the entire Segmentation will receive Email A OR Email B.


But if Email A and Email B have segment specific content, then you'll encounter an odd situation. If we use the standard Random Sample Choice method above, you'll have 50% of each segment receiving Email A and 50% receive Email B. Each Segment will see Email A's segment content just for them and the same with Email B.

Essentially you'll have 4 different emails being sent out depending first on Random Sample, then on each Email's dynamic content Segment.

So each email will be sent to 50% of the entire Segmentation and 50% of each Segment. If you instead chose a specific Segment to send from the start of the smart campaign, then you'd have 50% of Segment 1 receive Email A and 50% of Segment 1 receive Email B.

But that defeats the purpose of the Segmentation to some extent.