Some Tests on Random Sample Behavior

Anonymous
Not applicable

Sometimes I wonder about random things. Like the exact behavior of random sample. In case you are also curious about these kinds of things, I thought I'd share some results of a few tests I've done with random sample.

Does random sample work for trigger campaigns not just batch?

I tested this out because I wanted to understand whether or not I could essentially split a set of incoming leads in even groups even if they were going through a trigger campaign. For example, I might use this if upon hitting MQL, I wanted to ‘round robin’ leads and send each of X number of salespeople an even number of leads, except all leads would hit MQL at different times so a batch campaign would not be appropriate.

I ran a test with a trigger that, upon lead creation, added a random sample of 50% of leads to List 1 and the rest to List 2. I did one test list import with ~180 leads and then did a second test list import with ~180 different leads.

Total list counts after completing the two imports:

  • List 1: 183
  • List 2: 182

The results tab of the smart campaign in question looked basically like this:

It was clear that even though it was a trigger campaign, Marketo had alternated adding the records to the lists. This shows the transition between list 1 and list 2 (based on Yahoo versus Hotmail email addresses) and you can clearly see that even though the Hotmail lead came in on a different list a few minutes later, it remembered where it had left off.

I then created several Gmail test leads manually and several AOL leads through form fill to confirm the behavior was the same, which it was.

So yes, random sample works exactly as you might hope for a trigger campaign.

How does random sample work when your lists have an uneven count and does that vary with and without using the Default Choice?

I ran some tests to figure out what happens when you’re running random sample against a list of uneven lead counts. For example, if you have 3 leads and you run a random sample of 50% to each of two lists, what happens to Lead #3?

Initially, this stemmed from wondering whether you might leave off leads if you set your random sample up like this:

Although these add up to 100%, what happens if you had 101 leads on the list? Do you get the same results as if you set up your random sample like this?

These were tested as a batch campaign run.

Count of Original Leads

Random Sample Setup

Results

3

Choice 1: 50% to List A

Choice 2: 50% to List B

Default: Do Nothing

As you can see, two leads were added to list A and one was added to list B.

3

Choice 1: 50% to List A

Default: to List B

As you can see, two leads were added to list A and one was added to list B.

11

Choice 1: 33% to List A

Choice 2: 33% to List B

Choice 3: 34% to List C

Default: Do Nothing

As you can see, four leads were added to list A, four to list B, and three to list C.

11

Choice 1: 33% to List A

Choice 2: 33% to List B

Default: to List C

As you can see, four leads were added to list A, four to list B, and three to list C.

My conclusion is that it does not matter which way you set up the random sample (using default choice or not) from a functionality standpoint. Marketo always runs through and evaluates each lead on the list and never leaves one off. (However, I would still choose to use the Default Choice rather than three choices in this scenario.)

As an interesting aside, I ran a quick pivot table to see how many times in my five runs through the 33% example a lead got added to which list to see whether it was truly randomly selecting individuals, since the results log were always adding in order. You can see that there’s a large imbalance here - people would almost always end up on the same list in each run, but not always.

Of course, my tests are not statistically significant or anything, but I feel reasonably confident that I could base my campaign design off this info. If anyone else has suggestions or test results on random sample, feel free to share. Inquiring minds want to know!

5285
7
7 Comments
Jason_Hamilton1
Level 8 - Champion Alumni

Great post! I could have used it two weeks ago, I ran a similar experiment to add new leads to a nurtue and a nurture test group, and received similar results as you.

Keep up the posts Kristen, they are always very relevant and useful!

Thanks,

Jason

Alexandra_Grass
Level 1

Thanks a million for this post! It saved me a great deal of testing / hand-wrangling when designing my latest program, whilst confirming some things that I'd intuited based on prior experiences with the random function.

Cheers!

Alexandra.

Anonymous
Not applicable

Interestingly, we were just discussing this behavior again last night and I got more clarification on how this works, officially, and why I saw the results I did.

Random sample in trigger campaigns is not truly random. It's sequential based on how you set up the campaign. The campaign keeps track of how many leads have taken a given step in terms of percentage relative to the other buckets and will move on to the next bucket once the bucket percentage is greater than the condition's percentage. We do this is to get nicely distributed buckets, even for a small sample size (otherwise the 90% bucket could actually be 100% for a small sample size). Batch campaigns, however, are truly a random sample.

Edward_Unthank_
Level 10

Hey, that was from my support case!

Also, batch campaigns are not truly random. If you run the same amount of leads through batch campaigns with the same flow steps, you'll consistently get the same results. They "randomness" is math-based instead of randomly generated numbers.

If you run 100 leads through 10 separate cloned smart campaigns with the same flow steps, each to assign a number 1 through 10 randomly based on a random sample choice flow step, the same results will be replicated. Even small variations in sizes of leads in batch campaigns leads to startlingly similar resulting numbers.

Cheers,

Edward Unthank | Founder, Etumos

Jessica_Cross1
Level 6

wow.  Support quotes you without credit. Edward Unthank (Etumos)

SanfordWhiteman
Level 10 - Community Moderator

Long-delayed P.S... That's 'cuz it's a statically seeded random number generator. This is a good thing for reproducibility.  Like if you're running a simulation in life sciences or whatever, you need a sequence of numbers that's non-repeating and acceptably random across a long range, but you also need to run through same sequence over and over again, with different variables, to measure effects. In martech, too, that's probably what we want more often than not.

FlowBoost can generate dynamically seeded random sequences that can't be predicted, if that's what you want:

var distro = FBRandom.getSequence().autoSeed(),

    random = FBRandom.generateInt(1,10)(distro);

Arvind_Rangaraj
Level 2

Hi,

I set-up a batch campaign for all the leads that come through marketo. I need to assign Contact Person ID to the leads that come into Marketo in Round Robin. There are a total of 10 Contact persons, hence am using "Change Data Value" in the Flow of the Smart Campaign with 10 "Random Sample" in that. I have given the percentages for each Random sample to be 10(so that total comes to 100%). I kept the Default to be one of the 10 Contact person ID. I needed to test so I ran some test campaigns with same "Flow" conditions and here are the Random sample assigning:

When Default condition is not given:

When campaign had 51 Leads

All 10 Random sample values(Contact Person id) & no default Person ID was given

5 leads were assigned to Each Person ID

1 Lead was skipped

When Default condition is given:

Contact Person ID: 7 is also used in Default condition

61 Leads

6 leads were assigned to Each of 10 Contact Person ID

1 Lead was added to Default Person ID

---------------

116 Leads

12 leads were assigned to Each of 9 Contact Person ID -

8 Leads was added to Default Person ID(Contact Person ID: 7)

---------------

30 Leads

3 leads were assigned to Each of 10 Contact Person ID

-----------------

18 Leads

2 leads were assigned to Each of 9 Person ID

0 Leads was added to Default Person (IDContact Person ID: 7)

--------------

43 Leads

4 leads were assigned to Each of 9 Person ID

7 Leads was added to Default Person (IDContact Person ID: 7)

-------------

47 Leads

5 leads were assigned to Each of 9 Person ID

2 Leads was added to Default Person (IDContact Person ID: 7)

You can see that the Round Robin assignment is done only when the total count of the batch campaign is Divisible by 10.

Can anyone please let me know how can I use Round Robin for any count of leads in a Smart Campaign that runs everyday for the leads that come in the Smart Campaign.

Thanks,

Arvind. R