Random Sample in Engagement Programs

[Anonymous]

I'm running an engagement program that I only want to send Email 1 to a certain amount of people in the pool every week day until it's exhausted.  I currently have this as the flow:

The first day it ran perfectly and since then won't send emails.  Does anyone know what is going on?  Or have a suggestion on what to do?  It should be sending just under 1,000 emails a day and I'm not sure what is going on.

[Devraj_Grewal]

Sari,

Is this a batch or trigger campaign? Can you please attach screenshots of the smart list for the smart campaign? I am thinking you only allowed those leads to qualify once for the smart campaign. I would also add the filter of "Not was sent email: Email 1" to make sure people aren't sent the email twice if they run through the smart campaign twice.

You have it set up so that of all the people running through the flow, only 5 percent will be sent the email and all others will skip that step and wait 6 days before being added to the engagement program.

[Anonymous]

Hi Devraj -

This is a Batch campaign.  Here is the Smart list

Also here is the schedule:

I didn't think of the "Not was sent email" and I'll add that. Where would I put that step in the flow? But since I don't have that in there now shouldn't it still be sending to the 5% everyday even if it's duplicated?  On the schedule do I need to run it so that they run through it every time? 

[Devraj_Grewal]

So you have the smart campaign set up such that every day, people meeting the criteria of the smart list will enter the flow and only 5% of them will receive the email but 6 days later all of them will enter the engagement program. Thus, the majority of these people will not receive the email before being added to the engagement program.

The qualification rules you've set up for the smart campaign only allow people to run through once, so every day only those who have not entered the smart campaign will run through, and of which only 5% will receive the email and eventually all will enter the engagement program.

Just to make sure I have your intention correct, you want all those meeting the smart list's criteria to enter the engagement program, but only 5% of them overall to receive the email? Since all those who have gone through the flow are now members of the program, it wouldn't be necessary to add them to the smart campaign again just to have them added to the engagement program again.

[Anonymous]

Our intention is that the overall list of about 20,000 only 5% of them receive email 1 every week day.  And then the 5% that got the email 1 wait 6 days until they under the engagement program.  Until the 20,000 people have been exhausted through being sent email 1.  We only want a limited amount of people to go through the program at once.  How do I make that happen?

[Devraj_Grewal]

Okay that clears a bit up. I would first check to see if all those individuals who ran through the smart campaign are members of that engagement program and only 5% have been sent the email. You can decide if you would like to send the email to those individuals since they are already part of the engagement program.

In the future, the smart campaign should be set up like this:

Smart List:

also include filter: "not was sent email: Email 1"

also include filter: "member of engagement program: false: Marissa"

Flow:

1. Send Email:

     Choice 1:

          random sample: 5% send email: Email 1

     Default: do nothing

2. Remove from flow: this campaign

     Choice 1: not was sent email: Email 1

         Default: Do nothing

3. Wait 6 days

4. Add to Engagement Program

Setup:

Each person can run through this flow every time

This allows every person to run through the smart campaign every day until they have already been sent the email or added to the engagement program since they need to have not been sent the email or already added to the engagement program to qualify for the smart campaign. Once in the flow, 5% of them will receive the email and the others will not. Those 95% not sent the email will be removed from the flow until the next day where they qualify again to be sent the email. Those 5% that are sent the email will then wait the 6 days and be added to the engagement program.

[SanfordWhiteman]

Bear in mind that if you send to 5% of the remaining leads every day this isn't the same as 5% of the overall list -- it's mathematically impossible.

That is, you won't be sending to 1000 people for 20 days. Rather, the daily percentage of the overall list will steadily go down every day. On day 1, you'll send to 1000. On day 2, 950. Day 3, 902. Day 4, 857.  By Day 30 you'll only be sending to 250 people a day, or ~1% of the original list. 30 days later you'll be under 50 per day.  Exhausting the entire list will take over 100 days, at the end of which you'll be trickling out a couple per day. I don't see that as a useful exercise unless that's a deliberate plan.

If you want 1000 leads x 20 days, you're going about this the wrong way. You should split the list from the beginning into 20 random samples (20 lists or perhaps program statuses or whatever you want, just do it all at once). As has been well-documented here, Random Sample should be used within a single Flow step to avoid surprising results.

[Anonymous]

Thank you Sanford for the answer.  That is really interesting about the Random Sample as this is what our Marketo consultant told us to do in order to get the result we are looking for.

What would be the easiest way to go about this?  I want to use the Engagement Program for this so that the people will go through each stream and receive emails over time and not have to think about it in a way. But we need it to be a limited amount of the main list a day, every weekday until exhausted.

[SanfordWhiteman]

... this is what our Marketo consultant told us to do in order to get the result we are looking for.

Guess they didn't test it.

Random Sample is quite an elusive beast: lots of people think they get it but don't. (For another example, did you know that it's possible for a campaign w/Random Sample to never qualify anybody, if the percentage is too low?)

What I would do is run the main list through a single Random Sample (batch) campaign that assigns them to one of 20 lists in a single Flow step. That's one Add to List with 19 Choices, 5%, 5%, 5%... and then a Default Choice. Then those lists become your working groups.

[Anonymous]

Thanks Sanford!  I really appreciate the help on this!