Random sample flow step not very random

[Phillip_Wild]

Hi everyone

A question about using Random Sample as a flow step. I had a flow step that looks like this:

Add to List

Choice 1: Random Sample is 50 - add to List A

Default: Add to List B.

We then used those lists to report on bookings, as an A/B test.

So here's the weird thing. We have a custom field in Salesforce called "Agency Secondary". I grouped the people in each list by this, and here's a sample of what I see:

Agency Secondary	List A	List B
Agency 1	0	650
Agency 2	1055	0
Agency 3	246	0
Agency 4	0	116

This does not look very random to me! My guess is that Marketo is taking large chunks of my total list and then allocating then to lists in, say,  1,000 person chunks. Not cool.

What should I be seeing here, and is there any way I can make this closer to random? I know that "random" is an interesting topic in computing so it's never going to be truly random, but the above seems a bit ridiculous to me. That's fine if you have consumers where different people in a given chunk of 1,000 shouldn't show similar behaviour, but at a particular travel agency (which is what I'm dealing with) there would be specific booking demographics of clients that could easily sway any A/B test I'm looking to run.

Any ideas?

[SanfordWhiteman]

The random sample method is (at least this is the received wisdom) some kind of hash-bucket algorithm. So it's not using a PRNG, that's true. The trend you're seeing here indicates that the hash bucket keys are dependent in some way on that field. But the relationship may be more complex, for example if people were added with those agency values in a particular order?

In any case, you can get a truly random distribution by using a webhook.

[Phillip_Wild]

Thanks Sanford....I don't think they would have been added in a particular order, but it's certainly possible.

How would you do it using a webhook? Push out to a service which allocates a value of 0 or 1, randomly?

[SanfordWhiteman]

One of two ways:

• Random allocation like you said. No problem with this except it tends toward a 50/50 split but is not guaranteed to be exact.
• Round-robin allocation, which guarantees a strict 50/50 split (for an even number of leads!) but if in some strange reason the input is already interleaved in the same way, would not change that distribution (i.e. if leads, or widgets in general, are already being round-robined between two stacks before being sent in, then the output would simply mirror that distribution, if that makes sense).