Issue with AB testing using an engagement stream

[ashah123]

HI Team ,

We wanted to conduct A/B testing for our batch-and-blast campaign using the engagement program. We have four emails to send every Tuesday with different content to the same audience. One of our previous consultants recommended leveraging the engagement program, allowing us to set the availability of the four emails for the desired send day. This way, newly added records won't receive previous emails.

I read a document stating that A/B testing can be used in the engagement program stream or for a trigger using a champion-challenger approach, so I attempted to implement that.

I selected a sample size of 10 for this test, allocating 40% for A/B testing and 60% for the winner. I set up the engagement stream with the A/B testing email I created using the champion-challenger approach. However, all 10 emails were sent together, and no A/B test was performed.

Can someone guide me on how to approach A/B testing with the champion-challenger method for nurturing or a batch-and-blast campaign as mentioned above?

Let me know if you need additional details.

Regards,

Akshat

[uditmathur]

Hi @ashah123 ,

Seems you have misunderstood the concept of C/C test.

In C/C test the slider which you select implies that out of your total sample size (in your case 10) 4 person will receive the challenger subject line & remaining 6 will receive the champion subject line (original subject line is treated as champion). Unlike A/B test where the slider defines out of the total members how many will receive the test mails in equal proportion.

Once your test is concluded you will have to manually declare the winner based on the performance of each subject line.

Suppose in your case the challenger subject line gives better performance, than once you declare the winner all future mails will be sent with the challenger subject line.   

You can further read here.

Hope this helps 😊

[ashah123]

Hi @uditmathur ,

To your point , ideally 4 people should receive the challenger email right and rest the champion email , but it didnt happened tht way all 10 received the champion subject line upfront 

Also is there a way to use this for batch and blast type of campaign leverage engagement program where we can decide AB test date and winner date to send emails ? Or it wont be possible ? is it something  only possible for trigger based emails where it runs everyday and based on the C/C and the metrics we then have to decide the champion subject line out of both but can decide the Ab test and winner dates ?

Can you explain me if possible 

[Michael_Florin]

I'd say that Email Programs are not supposed to be used within Engagement Programs. Email Programs are for A/B testing in Batch & Blast scenarios.

That doesn't mean you can't test in Engagement Programs. You can 

A) Use the Random Sample Flow Step to split audiences to run into different streams. What you're testing then is not one specific email's subject line but a certain order of emails or a flow of different content or probably different schedules of email sends.

B) Use the Random Sample Flow Step to send different versions of emails. To do that, you will have to use Smart Campaigns as content of your streams instead of emails directly.