Sometimes I wonder about random things. Like the exact behavior of random sample. In case you are also curious about these kinds of things, I thought I'd share some results of a few tests I've done with random sample.
Does random sample work for trigger campaigns not just batch?
I tested this out because I wanted to understand whether or not I could essentially split a set of incoming leads in even groups even if they were going through a trigger campaign. For example, I might use this if upon hitting MQL, I wanted to ‘round robin’ leads and send each of X number of salespeople an even number of leads, except all leads would hit MQL at different times so a batch campaign would not be appropriate.
I ran a test with a trigger that, upon lead creation, added a random sample of 50% of leads to List 1 and the rest to List 2. I did one test list import with ~180 leads and then did a second test list import with ~180 different leads.
Total list counts after completing the two imports:
- List 1: 183
- List 2: 182
The results tab of the smart campaign in question looked basically like this:
It was clear that even though it was a trigger campaign, Marketo had alternated adding the records to the lists. This shows the transition between list 1 and list 2 (based on Yahoo versus Hotmail email addresses) and you can clearly see that even though the Hotmail lead came in on a different list a few minutes later, it remembered where it had left off.
I then created several Gmail test leads manually and several AOL leads through form fill to confirm the behavior was the same, which it was.
So yes, random sample works exactly as you might hope for a trigger campaign.
How does random sample work when your lists have an uneven count and does that vary with and without using the Default Choice?
I ran some tests to figure out what happens when you’re running random sample against a list of uneven lead counts. For example, if you have 3 leads and you run a random sample of 50% to each of two lists, what happens to Lead #3?
Initially, this stemmed from wondering whether you might leave off leads if you set your random sample up like this:
Although these add up to 100%, what happens if you had 101 leads on the list? Do you get the same results as if you set up your random sample like this?
These were tested as a batch campaign run.
Count of Original Leads | Random Sample Setup | Results |
3 | Choice 1: 50% to List A Choice 2: 50% to List B Default: Do Nothing | As you can see, two leads were added to list A and one was added to list B. |
3 | Choice 1: 50% to List A Default: to List B | As you can see, two leads were added to list A and one was added to list B. |
11 | Choice 1: 33% to List A Choice 2: 33% to List B Choice 3: 34% to List C Default: Do Nothing | As you can see, four leads were added to list A, four to list B, and three to list C. |
11 | Choice 1: 33% to List A Choice 2: 33% to List B Default: to List C | As you can see, four leads were added to list A, four to list B, and three to list C. |
My conclusion is that it does not matter which way you set up the random sample (using default choice or not) from a functionality standpoint. Marketo always runs through and evaluates each lead on the list and never leaves one off. (However, I would still choose to use the Default Choice rather than three choices in this scenario.)
As an interesting aside, I ran a quick pivot table to see how many times in my five runs through the 33% example a lead got added to which list to see whether it was truly randomly selecting individuals, since the results log were always adding in order. You can see that there’s a large imbalance here - people would almost always end up on the same list in each run, but not always.
Of course, my tests are not statistically significant or anything, but I feel reasonably confident that I could base my campaign design off this info. If anyone else has suggestions or test results on random sample, feel free to share. Inquiring minds want to know!
7 Comments
.png)