Request Campaign vs. other triggers

[Michael_Florin]

Hello!

We have built a system of lead entry and scoring campaigns, which heavily relies on "Request Campaign" flows. Under heavy traffic our campaign queue gets pretty crowded and processing tends to slow down which causes various headaches.

Now I have heard that "Request Campaign" is actually not an ideal way to connect two Smart Campaigns. Records would flow faster, if for instance "Add to list" and "Was added to list" was used as the connector.

Can anybody confirm that is true?

Thank you!

Michael

[Keenan_Murphy1]

Hi Michael Florin, did changing to an "Add to list"/"Was added to list" setup improve your speed issues?

I've been looking at building out our operational structure following the modular Etumos Architecting Best Practices (Edward Unthank, Marketo Summit) - YouTube  method, but with such heavy reliance on requested campaigns I'm unclear if this would be setting us up for instance-wide slowness at a scale of 100k leads/month or week or quicker as we grow.

I've received ad hoc advice to avoid anything beyond 1 link in the "Request Campaign" daisy chain, but the Etumos method uses a minimum of 4 just off the initial "Person is Created" flow down to the level of lead scoring, enrichment, etc., all of which are triggered simultaneously from the initial "controller campaign".

Using "Add to list"/"Was added to list" seems like a simple enough modification, but is that the real solution, or is there some inherent flaw with architecting Marketo to daisy chain potentially dozens of campaigns simultaneously at scale? Sanford Whiteman‌ would be curious to hear your thoughts on this topic.

Thanks,

Keenan

[Michael_Florin]

Hi Keenan - I didn't execute that change. I didn't like the idea of "Add to list/Was added to list" all that much, as it looks clumsy to me. And I didn't read from this thread or from talking to Marketo support, that "Request Campaign" is per se bad. So I kept it.

What I did do though, was to create more exceptions - let's call them "jumps" - in the chain. Guided by the permanent reasoning: Does a record need to go through this step or can they skip it? New records usually need to take all steps but existing records might be able to skip some. So in Step #3 I added constraints to "Request #4" that filter on the jump criteria and if they meet, let records jump to #5 or even #6 or #7. That seems to have taken some pressure off, so the process runs pretty smoothly since.

[Keenan_Murphy1]

Yes it's definitely messier with an additional list for every campaign. 

But glad to hear that design approach has helped minimize pressure, thanks for the update!

[Michael_Florin]

Thank you, Grace & Grant!

Yes, I know that trigger campaigns run with different priorities. But I'm wondering if that is actually relevant for my case. If we talk about trigger priorities, we're talking about what runs earlier and what runs later, right? But I need to process stuff faster - not relative to other campaigns, but absolute.

So if I have 10 campaigns that request each other in a chain, and 50.000 new records hit that chain at the same moment - which unfortunately is our reality - and the processing of the whole chain takes 4 hours, can I somehow cut that time down by not using "Request Campaign" and "Campaign is Requested" anymore?

Well, Grant, if Add to List is four times as fast as Request Campaign, this might actually do something. I'll test it.

Thanks again!

Michael

[Mark_Price]

When the system is backed up, what do you see when you go to the Campaign Queue? 

You should be able to see the campaign priorities as High, Medium, Low.  This might help determine where the blockage is occurring: 

To weight something as 'high priority' that isn't normally, you can add a blank 'Send Alert' step as the first item in the campaign OR try the campaign priority override within Marketo Sky (I'm not using Sky at the moment so can't comment on how well this works)   

Campaign Queue Priorities
https://nation.marketo.com/docs/DOC-1137-how-campaign-processing-works

Overriding Priority in Marketo Sky
https://help.marketo.com/hc/en-us/articles/360010183814-Priority-Override-for-Trigger-Campaigns

Do your 10 campaigns have any steps that are calling Webhooks? 

In my experience, these are rather slow and bulk data updates are best done via the API

It's hard to troubleshoot without knowing your database size and what the campaigns are doing, but hopefully this gets you started. 

[Mark_Price]

You're welcome!  This problem can be a tricky one to solve so am happy to share some insight that may help.  

Jep lays out a good breakdown of what could be happening with response times.  Adding to what Jep said, the webhooks are also prioritized as 'low' priority unless you implement a workaround.  This can add to the processing time because as other activities run in the system, the webhooks can keep getting bumped down the queue. 

As Sanford mentioned, you can simplify/optimize the webhooks into one but it would take a little development and restructuring of your campaigns.  Sometimes these campaigns to request others is an overly complicated way to do what simple development can do.  

In addition to the webhooks you might review campaigns that run on webpage views.  If you have a lot of traffic or are Enterprise with millions of records in the db, these can be somewhat taxing on the system.   You'll have to monitor the campaign queue when a blockage occurs which will tell you where to start.

[Michael_Florin]

Thanks Mark!

Yes, out of our 18 - 10 was just a number - lead entry Smart Campaign, 5 contain webhooks. De-duping, creating a hash value, and calling DemandBase and two more lead augmentation services. That's probably a little too much, and I'm thinking about how to peel complexity off of it. Something like: If the DemandBase or DiscoverOrg were able to fill all blanks in the data, don't call anything else anymore. But honestly I don't have a good idea as to how to formulate this scenario in a flow step choice.

[SanfordWhiteman]

...and I'm thinking about how to peel complexity off of it. Something like: If the DemandBase or DiscoverOrg were able to fill all blanks in the data, don't call anything else anymore. But honestly I don't have a good idea as to how to formulate this scenario in a flow step choice.

Move the complexity to the webhook tier. Run those checks in parallel from the intermediate service, so they finish in one webhook call. Independent services that each take < 1s can take < 1s in total.

[Jep_Castelein2]

I agree with Mark on webhook performance: if processing 50k records takes 4 hours, that is 3.5 seconds per record. You mention there are 5 webhooks. Generally, a webhook that performs reasonably well would respond in half a second. For 5 webhooks, that will already result in 2.5 seconds out of thee 3.5 seconds. And that assumes all webhooks perform well: it could be that one of your webhooks is the slow poke. To determine the response time, consider calling the Webhook with a tool like Postman. That shows the response time in milliseconds. 

There have been a lot of optimizations in trigger campaign performance, so while - generally speaking - batch campaigns will be faster, the difference isn't as big as it used to be. 

Then for "request campaign": there is overhead in putting a new trigger in the queue and waiting until it gets picked up. I would only use "request campaign" if it is not possible to put it in 1 Smart Campaign. But I don't think it's the primary cause of the slow performance you are seeing. 

[Josh_Hill13]

So this is a challenge. I have been dealing with this for years and you really have to look at your architecture and ask yourself some questions:

1. Does this really need to be a trigger?
   1. For Scoring flows, I'd say 100% of campaigns should be batches. Seriously. 
   2. Contact Me and Freemium Trial flows should be Triggers/Fast Track that are part of your Lead Lifecycle system.
2. When you say Lead Entry, I take that as some sort of Lead Lifecycle management initial processor.
   1. Ed Unthank has thoughts about daisy chaining these with Request Campaigns. Check his site at etumos.com or my articles on marketingrockstarguides.com.
   2. Think hard about what really needs to happen RIGHT NOW vs. LATER. I'd say you can have a fairly efficient LLC with Request flows that do hand offs for various reason. In fact, I run a fair amount of Request flows but they branch off from 3 centralized Flows that are the only CORE entry points for most of the records.
   3. Do not use Person Is Created except in your FIRST entry trigger. Otherwise you will overwhelm your trigger eval queue. 50000 records x Number of Person Is Created Triggers = X (50000 X 10 = 500,000) which will take forever and BLOCK everything. Same is true for ANY other trigger like List is Imported or Data Value Changes. Be real careful how and when you use these.
   4. A LOT of things can simply be batched, especially Engagement Transitions (read my slides on this). Really zero reason to ever use a trigger for Engagements.
3. Put this all on paper/whiteboard/lucidchart and consider when things actually need to occur to drive business,

[Michael_Florin]

Thanks Josh!

I read about daisy chains on etumos.com and in fact our own lead entry flow looks very much like the Lifecycle Processing programs here. Yes, "Person is created" is only used as a trigger in the first campaign. And yes, all our Engagement Program transitions are batches.

You're saying all scoring flows should be batches. Some of our scoring flows are batches, but most of them are triggers. With the idea that the MQL - or lead that passed the scoring threshold - should be transported to sales as quick as possible. That needs to be trigger-based concept, doesn't it?

[Josh_Hill13]

Speed to MQL is overrated. The only MQL Sales wants is the Contact Me/Call Me one. Or a direct dial  Your flow should auto MQL that one.

Yes, I get that if someone is live on the site and triggers an MQL (Visits Page or something), it'd be good to let a rep know. There are a couple of thoughts I have about that:

1. if they didn't ask for a call, why call right this moment?
2. if you do call, have a script better than "I saw you were on our site..." And even then it's creepy to call in the middle of browsing. People have done that to me and it's weird. Don't be that guy.
3. If the MQL is triggered overnight, the salesperson has a list to call the next day and may be better received by the lead.
4. If your sales funnel is more like ecommerce, high speed, then a rep is probably not involved until after the CC swipe...but that is a good reason to use triggers for scoring.

[Josh_Hill13]

Also, you can trick Marketo by placing an Empty Send Email flow step at the TOP of any Trigger campaign to instantly push it to the TOP of the queue.

Any Wait Step more than 4:59 mins will instantly DROP your trigger or batch down the queue. 

[J_Grant_Gray]

From Scott Nash‌ Under the Hood II session:

https://nation.marketo.com/thread/45832-under-the-hood-ii-batch-campaigns-recording

for reference and guidance

Per this comparison, add to list is quicker.

[Grace_Brebner3]

Hey Michael,

In high traffic cases, yes this is typically true: Marketo basically has an internal prioritisation system that decides which types of processes are run at a higher vs lower priority - and request campaign flow steps are known for falling in the lower priority bucket. In low traffic cases it's not noticeable, but in high traffic cases you'll definitely see that "add to list" alternative processes will run faster.