How does "Random Sample" work?

[Phillip_Wild]

I've seen "Random Sample" in Flow steps when you use "Add Choice". But...is this self contained random samples, or is it reducing the amount of leads flowing through as it goes?

As an example:

Choice 1: If Random Sample is 33%, send email A

Choice 2: If Random Sample is 33%, send email B

Default: Send Email C

Would this work? Does the second random sample start afresh from the full amount of leads, or is it smart enough to remove anyone who qualified for Choice 1?

My theory is that it will remove as it goes, which means you would need to adjust the percentages. So if you started with 100 people:

* 33 people qualify for Email A and get that send. 67 remain.

* 33% * 67 = approx 23 people then qualify for Email B and get that send.

* The remaining 100 - (23 + 33) = 44 people will get Email C.

So if you wanted to get three splits of 33% each, the second choice should say 50%.

This is basically just a more difficult way to do an A/B test through that functionality, so I wouldn't necessarily use it like that - but I'd love to know 100% how the functionality works so I can think about how to use it.

Thanks, Phil

[Marisa_Rybar]

Hey everyone,

I have a couple questions on this. I'm going to use this "random sample" for a different use case so I will need to keep this outside of an email flow step.

Essentially what I want to happen is that there are 7 states which are going to comprise a new shared sales territory that will have a completely different assignment rule than the rest of our territories. This is what needs to happen:

• Sales Team 1 should receive 32% of incoming leads in this territory
• Sales Team 2 should receive 26% of incoming leads in this territory
• Sales Team 3 should receive 22% of incoming leads in this territory
• Sales Team 4 should receive 21% of incoming leads in this territory

The way I'm thinking this needs to be set up is that I have a smart campaign with a trigger of "person is created" and the states defined as filters, then the flow steps set up with the random sample as established above.

Or does the random sample flow step require a static list? In which case I would imagine the way it would need to be set up would be that I'd have a smart campaign listening for when new people are created in this shared sales territory, with a flow step to add them to a static list. Then, I'd have a second smart campaign set up with a trigger "Is Added to List" and the flow steps set up with the random sample as established above.

My other question though is this. Will this random sample work as a triggered campaign on a static list as it grows, or can it only work once on a static list through a batch campaign? (In other words, will the random sample flow step recognize the changes in list size and adjust the %'s accordingly?)

Josh Hill​?

[Anonymous]

The trouble I have found (and Marketo Support confirmed this), is that if you add in choices other than the random steps, it can throw the randomness off.

For instance, this was the flow step that prevented the random steps from working properly.

[Rachel_Egan]

Greg, can you please elaborate on why this didn't work properly? When I look at your screen shot, this is what I assume would happen:

1. All active clients will NOT have anything happen
2. All BDR reps will NOT have anything happen
3. 33% of the remaining leads will receive the BDR Rep value ending with UZIA0
4. 33% of the remaining leads will receive the BDR Rep value ending with GXiIAM
5. Anybody left over (34%) will receive the BDR Rep value ending with TIAQ

Please correct me if I'm wrong or if it doesn't work that way. I would like to know what actually happens so that I can make sure my flow steps work properly.

Thanks!

[Michaela_Iery2]

Hi! It's important to remember that in flow actions, Marketo looks for the first choice that the lead will qualify for and a lead can qualify for only one choice in any single flow action. If it does not qualify for Choice 1, it will move on to Choice 2, etc. If the lead qualifies for Choice 2, Marketo will not look at the remaining choices for that lead. It has qualified for a choice and Marketo moves the lead on. So in the example above, if the lead is an Active Client, the flow step will stop there as the lead has qualified for the first choice. It would likely be better to move these "qualifying" choices (Active Client, no rep) into the smart list part of the campaign.

[Anonymous]

Random works best when there are no other choices to deter it. So by having choices above it (1 and 2) that are empty, throws off the randomness of the flowstep. This is what Marketo support told me.

[Justin_Norris1]

Phillip Wild​ My experience is the same as Anna Bruning​ in that you can use random sample directly in the "Send Email" flow step. Christina Reynolds , where did you encounter that restriction?

The random sample divisions are based on the original total as Christina Reynolds described. Each percentage is a percentage of the original 100% and the "else" option sends to the balance of the sample after the previous percentages are removed.

[Josh_Hill13]

Guys,

Your choice step should be

Random Sample=33% to List A

Random Sample=33% to List B

Default to List C <==this will send the remaining 34% to List C.

I would be very careful about doing this any other way.

Yes, you can use this tool in most flow steps. You do not need to do this in an Email Send AB Test.

[Anonymous]

Hi Josh,

We are going to use a random sample in a email flow step to A/B test our two onboarding emails. As you mentioned above, is it best if I remove the choice 2 step and just set choice 1 = 50% with email 1 and then the default choice with email 2, or will this not make a difference? Thanks!

[Rachel_Egan]

I agree with Chris Saporito​. I wouldn't have the 2nd choice in there as it's redundant to choice 2.

Another thing to think about is that if you're doing something more complicated with the logic (say putting 15% of people in one bucket, 25% in another bucket, etc.) the default choice would catch everybody left over, just incase you calculated wrong and missed 1% or something.

[Anonymous]

Okay great thank you both, that helps!

[Chris_Saporito]

Personally, when I set it up I would not have the Choice 2 in there. But it will not make a difference.

[Anonymous]

Hi Josh,

Can we get individual analytics for each sample send?

For example, I've been using the A/B testing method but I was told that I cannot get the individual stats (opens, clicks, bounces, unsubscribes, CTRs...) for each email within the test. I need to find a way to test different whole emails and be able to pull these stats.

Thanks!

[Phillip_Wild]

Sarah, you definitely can get these numbers...but it's not intuitive. Use a regular email report, but pull the "Was Sent Email" filter into the Smart List. Select the email you are looking to investigate, and then add a constraint "Variant". You can then select variant A or B. The report will then only show stats for that variant.

Of course, the program summary screen will show high level stats, depending on what test you are using, but you can get more depth using that method above.

[Anonymous]

I'll try it -- Thanks Phillip!

[Anonymous]

Thanks Justin Norris​ & Anna Bruning​ for pointing that out! We first started using "random sample" last year for A/B testing in default programs, and we weren't able to add it as a choice in the email flow step for a default program. I had never gone back to look at it again (stuck in an old habit I guess). I saw your comments this morning, so I tested it out and IT WORKS! Hooray! Thank you for the comments.

Phillip Wild​ - Skip the add to list flow step and send straight from the send email flow step!

Send Email

Choice 1: If Random Sample is 33, Email A

Choice 2: If Random Sample is 33, Email B

Choice 3: If Random Sample is 34, Email C

[Phillip_Wild]

That makes sense! It seemed weird that the option was there but I couldn't use it.

Thanks all.

Phil

[Anna_Bruning1]

Christina Reynolds​, are you talking about email (mailbox) programs? I know you can't use random samples in the smart lists for those programs, but I have used random samples in flow steps of regular programs to send emails so I can confirm that it works.

[Anonymous · Accepted solution]

Hi Phillip Wild​!

I struggled with this at first too. You have the right idea though.

With the random sample flow step, you'll need to be sure that all choices equal 100. In your example above, Choice A would get 33%, Choice B would get 33% and Choice C would have to get 34%. Also, you will need to add the various sample choices to a static list instead. You are unable to directly send the email in the random sample flow step. Once you add the members of each sample to a list, you create the email send flow step and add the constraint "member of list".

You may want to add a small wait step depending on the size of the send. I typically like to add a 2 minute wait step.

Hope that helps! Let me know if you have questions.

[Phillip_Wild]

Hi Christina

Interesting. So despite the fact you can add “Random Sample” to a choice step in a “Send Email” step in a flow, you still can’t actually send an email in that fashion?

I think I know what you mean - but just to clarify, you would need to do the following in a flow series:

• Add to List

-Choice 1 - If Random Sample is 33%, then add to Static List A

-Choice 2 - If Random Sample is 33%, then add to Static List B

-Choice 3 - If Random Sample is 34%, then add to Static List C

Wait 2 mins

Send Email

-Choice 1 - If Member of List Static List A, then send Email A

-Choice 2 - If Member of List Static List B, then send Email B

-Choice 1 - If Member of List Static List C, then send Email C

Is that right?

Thanks, Phil

[Anonymous]

Exactly right! That will work.

Side note: Under the "send email" flow step listed above the last choice should be choice 3 - probably just a typo.