Does anyone run A/B tests without relying on Marketo A/B test native randomization capabilities?

[Andy_N]

Hey everyone,

Working now with our data team on the best way for us to split the audience for the email A/B test campaigns. Until now we’ve been randomizing people for that purpose in our CRM using signup time and/or native A/B testing capabilities of Marketo when sending emails.

We’ve noticed though that it might be not ideal as some segments of our clients were overrepresented in one group vs another and this skewed success metrics since different segments are treated differently by sales. In general, different segments use our product differently so if they aren't represented roughly equally in A and B then we'll have an issue.

I there anyone here who has experience spliting their audiences into A and B themselves and how do you do that? Which formulas, algorithms do you use?

Any insights are appreciated.

Thank you!

Andy

[SanfordWhiteman]

Native A/B is a uniform distribution, so not sure what kind of skew you're concerned about.

However if you want to shuffle leads in a specific way, so there is a skew based on certain characteristics, you could use a webhook-powered service. You can also use a  webhook for strict round-robin distribution, which of course will be non-random and just based on order of execution.

[Andy_N]

The concern with our previous experiments was the skews to our KPIs because different segments of our clients weren't represented equally in both groups.

What do you mean by webhook-powered services that can randomize on certain characteristics?

Can you give an example of a webhook that's used for strict round-robin distribution?

[SanfordWhiteman]

---

The concern with our previous experiments was the skews to our KPIs because different segments of our clients weren't represented equally in both groups.

To be clear: true randomization could never ensure that any particular characteristic is always equally distributed across 2 groups, regardless of the number of items. It's a clear sign of a broken/deceptive (P)RNG is if some arbitrary value appeared to be exactly round-robined across the groups!

---

What do you mean by webhook-powered services that can randomize on certain characteristics? Can you give an example of a webhook that's used for strict round-robin distribution?

---

Any service that uses a shared counter can distribute people into a fixed number of buckets, A-B-A-B-A-B. If you want to sample across the entire database then you'd need to run people through the webhook twice (once to preload or prime the database, then again to sample from it).

[Andy_N]

Can you give examples of "any service that uses a shared counter"?

Also, what do you mean by "a shared counter"?

[SanfordWhiteman]

---

Also, what do you mean by "a shared counter"?

---

A counter (or database sequence, same idea) that is accessible and incrementable/decrementable across all calls to a webhook.

---

Can you give examples of "any service that uses a shared counter"?

Unfortunately, due to Community rules I can't name specific names.