Issue with AB testing using an engagement stream

[ashah123]

HI Team ,

We wanted to conduct A/B testing for our batch-and-blast campaign using the engagement program. We have four emails to send every Tuesday with different content to the same audience. One of our previous consultants recommended leveraging the engagement program, allowing us to set the availability of the four emails for the desired send day. This way, newly added records won't receive previous emails.

I read a document stating that A/B testing can be used in the engagement program stream or for a trigger using a champion-challenger approach, so I attempted to implement that.

I selected a sample size of 10 for this test, allocating 40% for A/B testing and 60% for the winner. I set up the engagement stream with the A/B testing email I created using the champion-challenger approach. However, all 10 emails were sent together, and no A/B test was performed.

Can someone guide me on how to approach A/B testing with the champion-challenger method for nurturing or a batch-and-blast campaign as mentioned above?

Let me know if you need additional details.

Regards,

Akshat

[uditmathur]

Hi @ashah123 ,

Seems you have misunderstood the concept of C/C test.

In C/C test the slider which you select implies that out of your total sample size (in your case 10) 4 person will receive the challenger subject line & remaining 6 will receive the champion subject line (original subject line is treated as champion). Unlike A/B test where the slider defines out of the total members how many will receive the test mails in equal proportion.

Once your test is concluded you will have to manually declare the winner based on the performance of each subject line.

Suppose in your case the challenger subject line gives better performance, than once you declare the winner all future mails will be sent with the challenger subject line.   

You can further read here.

Hope this helps 😊

[ashah123]

Hi @uditmathur ,

To your point , ideally 4 people should receive the challenger email right and rest the champion email , but it didnt happened tht way all 10 received the champion subject line upfront 

Also is there a way to use this for batch and blast type of campaign leverage engagement program where we can decide AB test date and winner date to send emails ? Or it wont be possible ? is it something  only possible for trigger based emails where it runs everyday and based on the C/C and the metrics we then have to decide the champion subject line out of both but can decide the Ab test and winner dates ?

Can you explain me if possible 

[Michael_Florin]

I'd say that Email Programs are not supposed to be used within Engagement Programs. Email Programs are for A/B testing in Batch & Blast scenarios.

That doesn't mean you can't test in Engagement Programs. You can 

A) Use the Random Sample Flow Step to split audiences to run into different streams. What you're testing then is not one specific email's subject line but a certain order of emails or a flow of different content or probably different schedules of email sends.

B) Use the Random Sample Flow Step to send different versions of emails. To do that, you will have to use Smart Campaigns as content of your streams instead of emails directly.

[ashah123]

@Michael_Florin 

Do you have any documents related to A/B testing that include some explained scenarios?

1. I have attempted A/B testing within the email program, which is quite straightforward.

2. I faced challenges using the champion/challenger approach for both nurturing and batch-and-blast emails within the engagement program.

3. My main challenge is figuring out if we can set an A/B test and a winner date within the engagement program.

4. I also tried conducting A/B testing using smart campaigns, creating a winning version through a different smart campaign. However, this process is too manual, requiring us to check manually every time.

5. Therefore, I am looking to explore A/B testing further with engagement programs for both nurturing and batch-and-blast emails. Since we send multiple emails within the same program, we thought we could leverage the engagement program for A/B testing. This is currently where my challenge lies.

Regards,

Akshat

[uditmathur]

Hi @ashah123 ,

Ideally this should not happen. Can you retry with a different set of test data.

With C/C test you can not define winner automatically. you will have to manually declare winner.

[ashah123]

@uditmathur  Sure i can give a try again ..thats not an issue

I have a question regarding the engagement stream: Can we set specific dates for A/B testing and determine a winner date? Or is this feature only applicable to nurturing emails that utilize the champion-challenger method? For instance, if six individuals qualify for a particular day based on the defined champion-challenger size, do they receive those emails, and must we manually select the winner?

Additionally, could you please provide some documents that explain A/B testing with relevant use cases for nurturing programs or for batch and blast emails using the engagement stream?

Regards,

Akshat

[uditmathur]

Hi @ashah123 ,

No. for C/C test we can not set specific dates for test run as this is governed through the stream cast. You can definitely design your cast to suit your test dates. Also since C/C test is an ongoing test you will have to manually declare the winner when you have gathered sufficient data to support your decision of winner.

YES. 

A recent blog post from @Balkar_Singh explain A/B test in depth you can read the blog here.

Hope this Helps 😊

[ashah123]

Thanks @uditmathur , let me check the blog article and learn more about AB testing