Campaign Queue Backlog

[Anonymous]

We are experiencing a huge backlog of triggered campaigns in the Campaign Queue, which is causing delays to our data updates and programs that sync to Salesforce.

I initially engaged Marketo Support when I noticed the delays, and they told me that it was the sheer number of trigger campaigns we had active that was causing the backup. That combined with inefficient smart list operators (using "contains" or "is not", instead of "is") was severely slowing down Marketo's ability to process new trigger campaigns.

This article on optimizing smart lists was helpful in explaining the impact of those filters: https://community.marketo.com/MarketoArticle?id=kA050000000KzcsCAC

I was wondering if anyone has faced these challenges, and what were some steps that you took to alleviate the problem?

[Dan_Stevens_]

Lots of great insight here.  One of things that needs to be mentioned is when Marketo changed the way it processes - or makes available - anonymous lead activity, it required us to modify many of our smart campaigns with additional triggers/filters that weren't required prior to this.

Since lead records are no longer created for anonymous leads - and created only when the lead becomes known - we had no choice but to add several "Lead is Created" triggers to our campaigns (quite a few) - mainly those campaigns that are used to "Change Program Status" (this approach isn't needed for Change Score, Interesting Moment, Change Data Value, Add to List and Remove from List - see this post for more info on what's processed once an anonymous leads becomes known: Anonymous Lead Promotion to Known Lead – Munchkin V2 Behavior).  Even though we include specific filters alongside these triggers within the smart list of a smart campaign, evaluation of these leads must still occur each and every time a lead is created in Marketo.  While the objective of eliminating the creation of anonymous leads was to increase performance, the workarounds required have seemed to counteract those objectives for some.

Here's one example of a common smart campaign that's part of our website content programs (tracks content engagement on our website and progresses leads into the appropriate program status within this channel).  Prior to the anonymous leads changes, we only needed the "Visits Web Page" trigger.  Now, we also need to include a "Lead is Created" trigger and a "Visited Web Page" filter.  So if we have 100 of these programs/smart campaigns in our instance (and that's for just these types of programs), you can imagine the additional overhead that's now required each time a new lead is created in Marketo (even if they don't qualify for these campaigns).

Here's more information on this: Anonymous Lead Promotion to Known Lead – Munchkin V2 Behavior

[Dan_Stevens_]

Mike Reynolds​ is Munchkin v2 now fully rolled out to all customers?

[Dan_Stevens_]

Mike Reynolds​, can you confirm if this is now deployed to all customers?  We're finding that those smart campaigns that include the dual triggers (so that both existing and new leads can qualify) are not triggering properly.  And we're trying to identify the culprit - which could also be related to this existing issue: For those of you on Corona (part of Project Orion), are you experiencing issues explained here?

[Mike_Reynolds2]

Hey Dan. Sorry, been at a conference. Munchkin v2 is part of orion, which is still in progress being rolled out. First there's the orion server infrastructure, then it's trigger processing, then munchkin v2. It's a rolling staggered roll out, so there isn't a way to say clearly that you have it, because it's all rolling out in phases

[Dan_Stevens_]

Thanks Mike.

[Anonymous]

Hey Dan Stevens​, Do the filters apply for both triggers and filters? I thought that if you have two triggers in a campaign, then it automatically does either/or (either lead created or visited web page)

[Mike_Reynolds2]

Allison Sparrow​

Triggers do work independently from one another (either/or). Whichever one is triggered, the filter would still have to apply as well before the campaign can advance into the flow.

[Christina_Phill]

Hi all-

This exchange is super helpful! We have been seeing an increased slowdown as well and we run a ton of triggered campaigns. Does anyone know if those on the Enterprise level subscription experience the same problems? We are relatively good about cleaning up our triggered campaigns and removing old or obsolete ones. We are growing rapidly and will only see an increase in triggered campaigns going forward, so we are trying to determine if we need to jump to the Enterprise licensing level or would we experience the same thing? Would appreciate any insight you have.

Thanks,

Christina

[Anonymous]

Thanks Josh,

We have almost 1,800 active trigger campaigns in the Campaign Inspector! I have not paid much attention to this in the past, reasoning that it would not cost anything to keep campaigns active if nothing was actively triggering them. Do you know if it affects the sytem performance to have these campaigns active even if they are not running on a regular basis? 

I did convert many smart campaigns to batch campaigns and deactivated a bunch that were redundant - the campaign queue cleared up relatively quickly. Still, I would like to know if there is a limit to how many smart campaigns Marketo can process at once? I suppose it depends ont he conditions/filters involved, but is there a ballpark number I should be aware of?

[Justin_Cooperm2 · Accepted solution]

Yes, it absolutely will negatively impact performance of other campaigns in your subscription if you have tons of unused active trigger campaigns. You should do routine cleanups and de-activate anything that is not necessary. I also encourage people who use trigger campaigns to see if they can execute the same operations with a recurring batch campaign. Trigger campaigns really should be reserved for operations that must occur in real time. In general, trigger campaigns are slower if you are attempting to execute an operation for 5,000+ leads.

We have two queues that are part of trigger campaign processing. The first evaluates whether an activity has caused an active campaign to be triggered. It then pushes that over to a second queue that is a queue of campaigns that have already been triggered that need to be processed. If you were to import 1M leads in a list import and also had an active campaign with “New lead was created” trigger, it would take some time (it will vary, hard to estimate and also involves whether the trigger campaign has additional filters as part of the smart list qualification) to evaluate and push these over into the second queue to await processing. The next queue contains the campaigns that have been triggered and need to be executed. This queue is a bit faster than the first as we have parallel processing in place (6 processors) to execute these campaigns. There is also prioritization in place for the second queue. There are three levels of priority and the highest level will be processed faster. The type of trigger doesn’t matter (assuming there aren’t additional filters specified in addition to the trigger, which can also slow things down). What matters is the flow step. Trigger campaign processing is prioritized as follows: Campaigns with “Send Email” or “Send Alert” flow step are highest priority, campaigns with “Request Campaign” flow step are second-highest, and campaigns with any other flow steps (including “change data value") are lowest priority. The amount of time a trigger campaign takes to execute depends on what is presently in the trigger campaign processing queue. If there are many active trigger campaigns and several campaigns are triggered at once, there can be some wait time, especially for low priority trigger campaigns. It is always best to avoid trigger campaigns if the campaign will trigger extremely often. Otherwise, if there are a bunch of active trigger campaigns then this second queue could grow to be quite large. Finally, the more active trigger campaigns you have, the longer it will take to determine if the campaign was trigger by an event that took place. Best practice is to clean up (de-activate) any old active trigger campaigns that aren't needed any longer.

[Anonymous]

Thanks Justin Cooperman​!

That's by far the most information I've received on this topic. Very helpful.

[Anonymous]

Justin Cooperman​

Follow up question: are triggers in engagement program streams and segmentation filters processed the same way?

[Kenny_Elkington]

In addition to these, Revenue Cycle Model transitions, and Auto-Assignment rules follow this model.

[Justin_Cooperm2]

Yes. Say I had a segmentation on "Job Title" and I just changed mine in the Lead DB. It does not mean that instantaneously, that would be reflected in the segmentation. The trigger would go through same evaluation and execution to update my segment. So, there would still be a period where I was in my old segment.

[Caryl_Mostacho]

Hi Robberto McDowell​ this is interesting.

I've also been told that having many "Campaign is Requested" triggers can slow down Marketo as well.

[Josh_Hill13]

What is the count of triggers? Many SMB accounts seem to slow down around 100-150 triggered campaigns.

I would go through the list via Campaign Inspector and see if you can convert some to Batch campaigns, especially data management/appending/normalization campaigns. scoring demographics are sometimes better with batch, depending on your needs.