Apologies in advance for the rather lengthy use case example that follows, which many of you may have experienced. However, it provides context for my suggestions at the end.
We recently experienced a massive backlog with hundreds of thousands of leads backed up in dozens of smart campaigns. It was taking almost 15 hours for some to execute and for leads to sync to SFDC. We could see in which smart campaigns this was occurring with the Campaign Queue, which helped to identify the smart campaigns that were causing this. One anomaly we observed was that the flow of entries in the Results tab of our key Lead is Created triggered smart campaign that syncs new leads to SFDC had numerous long (>30 minutes) periods of no results being logged when normally it has results logged every minute or so, and strangely, the Campaign Queue showed no backlog in this smart campaign.
We finally identified a couple of new lead scoring smart campaigns that were effectively causing a chain reaction of smart campaigns to be triggered and when we revised those, the backlog decreased.
Then we sent an email blast to over 300K leads (our largest) and observed that hundreds of leads were stuck in Wait flow steps for hours beyond the wait period, webinar registration confirmations and nurture emails were being delayed as long as a day and again we were seeing the flow of entries in the Results tab of our key Lead is Created triggered smart campaign have long periods of inactivity. However, the Campaign Queue showed virtually no backlog in our smart campaigns. Tier 3 support used a 'Trigger Queue' report to identify two smart campaigns that were "expensive in regards to the trigger resource and causing delays."
We were able to focus on these smart campaigns from this insight and identified the cause of the problem as an Email is Delivered trigger, that had been inadvertently changed from 'contains' to 'not starts with', which caused it to evaluate more than the intended few thousand emails once a week. As soon as we made this one change, all of the problems stopped.
So we learened a few things about smart campaign processing from this.
- Although a smart campaign's smart list has filters that qualify only a few leads to go through the flow, that does not limit the number of leads that have to be processed by the trigger (300K in our case)!
- The Campaign Queue only shows leads that have qualified for the flow of smart campaigns, but have not entered the flow. Only the Trigger Queue report shows leads that are pending qualification by triggers and that report is not available to users.
- Each Smart Campaigns is evaluated by their flow actions and are assigned a priority as explained in the How Campaign Processing Works help article, but this doesn't explain how Trigger processing works. Each Marketo instance has a total of 6 'workers' that are a shared resource across triggered and batch smart campaigns. 2 workers are dedicated for smart list qualification, and 4 workers are dedicated for campaign execution. So if you have an enormous number of leads qualifying for triggers, with only 2 workers, you can get a significant backlog, but the UI has no tool that can show the backlog and its source.
And here are my suggestions.
- Expand the How Campaign Processing Works help article,to include an explanation of how Trigger processing works.
- Expose the Trigger Queue in the UI, which would help identify 'expensive' smart campaigns that might be negatively impacting smart campaign performance. Even speeding up a SFDC sync or email send by 1 minute is a big positive!
- Build-in some intelligence to protect users from themselves similar to the prompt that makes you type in the number of leads you want processed in a single flow action. So, in the case where sending an email may cause a massive number of triggers to fire (e.g. Email is Delivered), display a warning at some point before the email is sent.
- Allow for both smart campaigns and trigger queus to be prioritized (vote for ).
.png)