Request Campaign vs. other triggers

[Jep_Castelein2]

I agree with Mark on webhook performance: if processing 50k records takes 4 hours, that is 3.5 seconds per record. You mention there are 5 webhooks. Generally, a webhook that performs reasonably well would respond in half a second. For 5 webhooks, that will already result in 2.5 seconds out of thee 3.5 seconds. And that assumes all webhooks perform well: it could be that one of your webhooks is the slow poke. To determine the response time, consider calling the Webhook with a tool like Postman. That shows the response time in milliseconds. 

There have been a lot of optimizations in trigger campaign performance, so while - generally speaking - batch campaigns will be faster, the difference isn't as big as it used to be. 

Then for "request campaign": there is overhead in putting a new trigger in the queue and waiting until it gets picked up. I would only use "request campaign" if it is not possible to put it in 1 Smart Campaign. But I don't think it's the primary cause of the slow performance you are seeing. 

[SanfordWhiteman]

...and I'm thinking about how to peel complexity off of it. Something like: If the DemandBase or DiscoverOrg were able to fill all blanks in the data, don't call anything else anymore. But honestly I don't have a good idea as to how to formulate this scenario in a flow step choice.

Move the complexity to the webhook tier. Run those checks in parallel from the intermediate service, so they finish in one webhook call. Independent services that each take < 1s can take < 1s in total.

[Mark_Price]

You're welcome!  This problem can be a tricky one to solve so am happy to share some insight that may help.  

Jep lays out a good breakdown of what could be happening with response times.  Adding to what Jep said, the webhooks are also prioritized as 'low' priority unless you implement a workaround.  This can add to the processing time because as other activities run in the system, the webhooks can keep getting bumped down the queue. 

As Sanford mentioned, you can simplify/optimize the webhooks into one but it would take a little development and restructuring of your campaigns.  Sometimes these campaigns to request others is an overly complicated way to do what simple development can do.  

In addition to the webhooks you might review campaigns that run on webpage views.  If you have a lot of traffic or are Enterprise with millions of records in the db, these can be somewhat taxing on the system.   You'll have to monitor the campaign queue when a blockage occurs which will tell you where to start.

[Keenan_Murphy1]

Hi Michael Florin, did changing to an "Add to list"/"Was added to list" setup improve your speed issues?

I've been looking at building out our operational structure following the modular Etumos Architecting Best Practices (Edward Unthank, Marketo Summit) - YouTube  method, but with such heavy reliance on requested campaigns I'm unclear if this would be setting us up for instance-wide slowness at a scale of 100k leads/month or week or quicker as we grow.

I've received ad hoc advice to avoid anything beyond 1 link in the "Request Campaign" daisy chain, but the Etumos method uses a minimum of 4 just off the initial "Person is Created" flow down to the level of lead scoring, enrichment, etc., all of which are triggered simultaneously from the initial "controller campaign".

Using "Add to list"/"Was added to list" seems like a simple enough modification, but is that the real solution, or is there some inherent flaw with architecting Marketo to daisy chain potentially dozens of campaigns simultaneously at scale? Sanford Whiteman‌ would be curious to hear your thoughts on this topic.

Thanks,

Keenan

[Michael_Florin]

Hi Keenan - I didn't execute that change. I didn't like the idea of "Add to list/Was added to list" all that much, as it looks clumsy to me. And I didn't read from this thread or from talking to Marketo support, that "Request Campaign" is per se bad. So I kept it.

What I did do though, was to create more exceptions - let's call them "jumps" - in the chain. Guided by the permanent reasoning: Does a record need to go through this step or can they skip it? New records usually need to take all steps but existing records might be able to skip some. So in Step #3 I added constraints to "Request #4" that filter on the jump criteria and if they meet, let records jump to #5 or even #6 or #7. That seems to have taken some pressure off, so the process runs pretty smoothly since.

[Keenan_Murphy1]

Yes it's definitely messier with an additional list for every campaign. 

But glad to hear that design approach has helped minimize pressure, thanks for the update!