How does "Random Sample" work?

[Phillip_Wild]

Sarah, you definitely can get these numbers...but it's not intuitive. Use a regular email report, but pull the "Was Sent Email" filter into the Smart List. Select the email you are looking to investigate, and then add a constraint "Variant". You can then select variant A or B. The report will then only show stats for that variant.

Of course, the program summary screen will show high level stats, depending on what test you are using, but you can get more depth using that method above.

[Anonymous]

I'll try it -- Thanks Phillip!

[Anonymous]

Hi Josh,

We are going to use a random sample in a email flow step to A/B test our two onboarding emails. As you mentioned above, is it best if I remove the choice 2 step and just set choice 1 = 50% with email 1 and then the default choice with email 2, or will this not make a difference? Thanks!

[Chris_Saporito]

Personally, when I set it up I would not have the Choice 2 in there. But it will not make a difference.

[Rachel_Egan]

I agree with Chris Saporito​. I wouldn't have the 2nd choice in there as it's redundant to choice 2.

Another thing to think about is that if you're doing something more complicated with the logic (say putting 15% of people in one bucket, 25% in another bucket, etc.) the default choice would catch everybody left over, just incase you calculated wrong and missed 1% or something.

[Anonymous]

Okay great thank you both, that helps!

[Anonymous]

The trouble I have found (and Marketo Support confirmed this), is that if you add in choices other than the random steps, it can throw the randomness off.

For instance, this was the flow step that prevented the random steps from working properly.

[Rachel_Egan]

Greg, can you please elaborate on why this didn't work properly? When I look at your screen shot, this is what I assume would happen:

1. All active clients will NOT have anything happen
2. All BDR reps will NOT have anything happen
3. 33% of the remaining leads will receive the BDR Rep value ending with UZIA0
4. 33% of the remaining leads will receive the BDR Rep value ending with GXiIAM
5. Anybody left over (34%) will receive the BDR Rep value ending with TIAQ

Please correct me if I'm wrong or if it doesn't work that way. I would like to know what actually happens so that I can make sure my flow steps work properly.

Thanks!

[Anonymous]

Random works best when there are no other choices to deter it. So by having choices above it (1 and 2) that are empty, throws off the randomness of the flowstep. This is what Marketo support told me.

[Michaela_Iery2]

Hi! It's important to remember that in flow actions, Marketo looks for the first choice that the lead will qualify for and a lead can qualify for only one choice in any single flow action. If it does not qualify for Choice 1, it will move on to Choice 2, etc. If the lead qualifies for Choice 2, Marketo will not look at the remaining choices for that lead. It has qualified for a choice and Marketo moves the lead on. So in the example above, if the lead is an Active Client, the flow step will stop there as the lead has qualified for the first choice. It would likely be better to move these "qualifying" choices (Active Client, no rep) into the smart list part of the campaign.