A/B testing in Stream, Is BOT activity driving statistically unremarkable results?

hlawrence
Level 2

A/B testing in Stream, Is BOT activity driving statistically unremarkable results?

This question references my previous question on BOT activity

 

We are engaging in some A/B testing of our marketing emails that are in an engagement stream, as one should.  I choose "new test" 50/50 with champion and challenger.  Up to now we have only done subject line tests.  I have tried 'Click to open %' and 'Custom Conversion Ratio' (clicking on one specific link in that email), but all results are within a couple clicks of 50/50. 

 

This week we tried whole emails, and STILL the results are 50/50

 

hlawrence_0-1708457891253.png

I am starting to get the feeling that BOT activity is driving these A/B test results.  In the previous question linked above, we had found out that a large amount of our clicks were coming from the first link in the TEXT ONLY version, suspecting BOT activity.  A recent email:

hlawrence_1-1708458211901.png

Do you have any ideas on how to filter out BOT clicks for A/B testing?

Thanks.

 

6 REPLIES 6
Vinay_Kumar
Level 10 - Community Advisor

Re: A/B testing in Stream, Is BOT activity driving statistically unremarkable results?

Have you enabled the Bot Activity Identification in your instance?

Vinay_Kumar_0-1708524053060.png

If it is enabled then you use a smart list to check the bot clicks.

Vinay_Kumar_1-1708524157066.png

 

 

hlawrence
Level 2

Re: A/B testing in Stream, Is BOT activity driving statistically unremarkable results?

This is how our BOT activity Identification is set.

hlawrence_0-1708535518955.png

 

Zoe_Forman
Level 9 - Community Advisor + Adobe Champion

Re: A/B testing in Stream, Is BOT activity driving statistically unremarkable results?

As @Vinay_Kumar says this looks to be set up correctly,

 

@hlawrence have you created the Smart lIsts to identify BOT activity?

hlawrence
Level 2

Re: A/B testing in Stream, Is BOT activity driving statistically unremarkable results?

This is an example of the Current Link Performance Report: 

hlawrence_0-1708547548482.png

I must be doing this incorrectly, as these constraints do not affect the report click or percentages at all.      

Darshil_Shah1
Level 10 - Community Advisor + Adobe Champion

Re: A/B testing in Stream, Is BOT activity driving statistically unremarkable results?

@hlawrence, Your filtering criteria look good! This should filter all the Click activities in the selected email with Bot Activity marked as False and pattern doesn't match the ones selected (you can even remove the Bot Activity Pattern constraint from the filter if you don't need it, i.e., you're just looking to filter out people with any activities that are identified by Marketo as bot irrespective of the activity pattern).

hlawrence
Level 2

Re: A/B testing in Stream, Is BOT activity driving statistically unremarkable results?

@Darshil_Shah1

Happy to hear we are set up correctly!  It still remains that I have a huge number of clicks coming from one (first) link in the Text version of the emails that is not consistent with human behavior and not able to be filtered out of the reporting. 

What is your recommendation on how to ensure that the A/B tests are based on human clicks.  Are there any tips or tricks with the 'Custom Conversion Ratio', maybe?

Thanks!