Wait Step Processing and Impact on System Performance - Scalability of Marketo

[Rich_Miller2]

Hello,

Does anyone have any architecture documentation, etc. for wait steps: How they are processed, do they have an impact on Marketo performance, general best practices for global/scaled deployments, etc.

I have never had an issue with them, meaning they have never slowed the delivery of mails to leads later in the smartcampaign (after the wait step) for campaigns with <50K leads in them.  But I am getting ready for a large deployment. (500K records in an engagement program). The send will be from a program (with smart campaign) inside an engagement stream.

These leads are in different timezones, so the idea is:

• Trigger the first mail
• Wait 7 hours
• Trigger the second send
• Wait 8 Hrs
• Trigger the third send

I would love to do it in one smartcampaign, but only if it will not slow the performance of the mail sends. They still need to go out at the right time. e.g. I cannot have the third mail be delivered 20 hrs after the first trigger, when it should have been 15 (8+7).

Thanks,
RM

[Josh_Hill13]

Well, the Wait Step itself will have nothing to do with the processing time. After 8 hours, the lead comes out.

If I am reading this right, you have a Program dropped into an Engagement Stream that you intend to use with 500k records. Right?

If you send your campaign this way, each record will get 3 emails at the end of the time. I am hopeful you are splitting this so it says if in Member of Smart List IN "Asia" send email, etc.

Questions:

1. Do you have other programs in the stream?

2. Is this a batch/one off in reality? (Please do not use engagement for this).

3. Why could you not use 3 batch campaigns to schedule this properly?

[Rich_Miller2]

Thanks for the response. Here is some additional info.

• Yes. It is a program that I dropped into the stream.
• There are conditions in the smartcampaign that limit the geos.

e.g.  if asia, send this mail.

Wait 7 hrs

If Europe, send this mail.

Wait 8 hrs

If US or blank region, send this mail.

• There are other programs in the stream that would work the same way. Different emails/content, but the same general flow.

• It is going to be an ongoing effort. i.e. we will continue to add content (programs) to the stream.

With your #3, are you thinking of three programs in the stream. One for each geo. Their smartcampaigns would still need the delays in them, since they all have to go off the stream cadence.

But it sounds like you are not aware of any general best practices for using wait steps with large sends. i.e. will the number and length of wait steps will not cause delays with larger numbers of leads in the stream.

Do they do something like this:

Filter entire qualified list, process entire list (but really only send what is applicable), wait, filter whole list again and process whole list, etc. If it was doing that, I could see how it could slow things down, especially with large lists as it may have one send piled up on another. But I am not an app architect.

Because if is was a simple filter, process and send only to applicable, wait, filter, process and send only to applicable, etc. it should not matter how many wait steps you have, how long they are, or how many leads you are processing.

Thanks! I know we are going deep in this one.

[Casey_Grimes · Accepted solution]

Hi Rich,

So, you actually hit the nail on the head: the big issue for Marketo in terms of lag for email sends is actually in list qualification. As such, I'm tempted to do a very cheap workaround to ensure the list counting can be queued separately: have your flow step send to USA leads immediately and actually have the other regions compile into static lists. Set up timed campaigns that correspond to the deployment times that the records in that region and have them email anyone on the static list, then have a small cleanup campaign that removes records from the list afterwards. This way, you're pushing all the buffering time to generate the list in the waiting period rather than having all leads sit through a wait step and then re-qualify.

[Rich_Miller2]

Thanks for the info everyone. Great stuff to know for scalability (the stuff marketo does not tell you about anywhere).

Courtney, the static lists could sit in the email programs I am putting in the engagement streams, right? So the Programs in the stream would look like this:

Program 1 in stream:

Smartlist: If member of stream

Flow: If NA or Blank, send mail. If ASIA or EU, send to respective lists. No wait. Only one condition.

Program 2:

Smartlist: If member of list ASIA and member of stream

Flow: Wait right amount of time to target local time (from stream launch cadence). Send mail as appropriate. No condition in flow.

Program 3:

Smartlist: If member of list EU and member of stream

Flow: Wait right amout of time to target local time. Send mail as appropriate. No condition in flow.

Since I am using the same email for all three programs, there is no risk of double sending.

There is a BANG BANG instance on the database here. While program 1 is assigning to the lists, programs 2 and 3 are looking for list members. How do I deal with that?

Seems to lead me back to different streams per geo. But that gets complicated just based on the number of streams that would be required.

Thanks,

[Josh_Hill13]

If you are using a Choice step, then yes, it will take some time to process.

No, Marketo takes in everyone who hits the Wait Step and dumps them out as their time comes up. There was a thread the other day suggesting Wait Steps might be a low priority processing time, but I haven't heard of any other problems with this.

As I suggested, your Cadences on separate Geo streams would be a safer way to handle the time zone thing. I suspect you will encounter problems where the Cadence triggers the Program's campaign and then it waits 7 hours. I/m sure your Cadence isn't every day, so it won't be a problem there. I'm just not convinced this is an efficient way to handle this situation.

As Courtney suggests, it isn't the Wait Step, it's the Choice step that will slow you down. You may also want to say Remove Americas Leads from Flow after it is sent out. Essentially you are asking Marketo to process 500k leads at EVERY STEP instead of kicking out the ones you don't want.

[Rich_Miller2]

Remove from flow will not kick them out of the engagement stream, right? It just removes them from flow processing and they already got the mail, so it should be OK. 95% of our records are in one geo, so if we could do that, it would be great for those that follow. Sorry, may be obvious, but I want to make sure.

That could be the easiest solution of all!

Thanks,
RM