A/B testing in Stream, Is BOT activity driving statistically unremarkable results?

[hlawrence]

This question references my previous question on BOT activity

We are engaging in some A/B testing of our marketing emails that are in an engagement stream, as one should.  I choose "new test" 50/50 with champion and challenger.  Up to now we have only done subject line tests.  I have tried 'Click to open %' and 'Custom Conversion Ratio' (clicking on one specific link in that email), but all results are within a couple clicks of 50/50. 

This week we tried whole emails, and STILL the results are 50/50

I am starting to get the feeling that BOT activity is driving these A/B test results.  In the previous question linked above, we had found out that a large amount of our clicks were coming from the first link in the TEXT ONLY version, suspecting BOT activity.  A recent email:

Do you have any ideas on how to filter out BOT clicks for A/B testing?

Thanks.

[Vinay_Kumar]

Have you enabled the Bot Activity Identification in your instance?

If it is enabled then you use a smart list to check the bot clicks.

[hlawrence]

This is how our BOT activity Identification is set.

[Zoe_Forman]

As @Vinay_Kumar says this looks to be set up correctly,

@hlawrence have you created the Smart lIsts to identify BOT activity?

[hlawrence]

This is an example of the Current Link Performance Report: 

I must be doing this incorrectly, as these constraints do not affect the report click or percentages at all.      

[Darshil_Shah1]

@hlawrence, Your filtering criteria look good! This should filter all the Click activities in the selected email with Bot Activity marked as False and pattern doesn't match the ones selected (you can even remove the Bot Activity Pattern constraint from the filter if you don't need it, i.e., you're just looking to filter out people with any activities that are identified by Marketo as bot irrespective of the activity pattern).

[hlawrence]

@Darshil_Shah1

Happy to hear we are set up correctly!  It still remains that I have a huge number of clicks coming from one (first) link in the Text version of the emails that is not consistent with human behavior and not able to be filtered out of the reporting. 

What is your recommendation on how to ensure that the A/B tests are based on human clicks.  Are there any tips or tricks with the 'Custom Conversion Ratio', maybe?

Thanks!