Darn that's what I figured in terms of Marketo being unhappy with high peaks. We'll have to find a way that doesn't overwhelm it and slow everything down. Or if thousands to 10s of thousands per hour is fine usually, maybe we just suffer on peak days, but that's not ideal. For this particular scoring use case, I don't technically need the scores to be always up to date, but then the problem is that if they aren't, it becomes difficult to know if it's safe to use the latest score or not. For example, one strategy I have now is to use a webhook to update the category of a lead's highest score, but then do it at most once every 2 hours, then wait 3 hours and run the webhook again. This is an approximation of throttling and is much better than firing webhooks on every page view for performance. The hard part is that it makes it more difficult for automation to work since you can't really trust whether the score is up to date, or whether it's a few hours old. ... View more

Up to hundreds of thousands or millions per hour during high peaks ... View more

Hi Sanford. It seems like what I have down gives some balance between adding a little safety and not increasing complexity too much, adding external dependencies, or slowing down performance by more than a couple hundred ms, but I'd love to hear more about your thoughts on compare-and-swap via webhook. First, I think what I have now in the screenshot reduces the race condition window to a reasonable size. There's probably a 50-100 ms window between where we check to remove them from the flow and when we actually get them removed from the list. Not ideal, but I don't expect too much concurrency except when a cron / scheduled Smart Campaign overlaps with triggers. As I'm typing this, another way I could reduce the likeliness of a race condition is to add a wait step before putting people into a cron list, so that the cron is truly more of a fallback and won't compete with triggered campaigns that execute immediately, but that would add yet another step. When new leads are created OR when leads are associated, I try to kick off actions to send emails that were triggered by activity that was logged while the lead was anonymous. The ideal scenario is that I can use a trigger on a server-side form post that I use to associate leads, but if that fails or if I'm associating via REST API, I depend on 24 scheduled batches (running once per hour) to try and flush the queues out as a fallback. If any of those scheduled campaigns run at the same time as the lead association, that's where the race is most likely to happen. The super ugly part with all of this is that I'd have to have our marketing team implement all of these things every time the want to set a triggered email, which seems tedious. If I was using a webhook to do a lock externally, I'd need to set up another service, have a key/value store with tracking for every campaign that's running, and then I think I'd be writing back to a Marketo field with the result by campaign, so I'd need a new lead field for every campaign that I run. Or this kind of what you're thinking with a compare-and-swap via webhook? If so, I think that's probably too much. Worst case if the race condition happens, is somebody gets 2 emails. If that happens 1 in 100,000 times that's probably not a huge deal. While I agree the smart thing to do would be to never use an inequality query, but I was imagining if that were the case, the Smart List Best Practices doc wouldn't need to recommend use positive over negative operators, but this is just speculation of course. Either way, I think these lists will probably be small so Denise's advice to go with more readable solution may be the most practical if I was keeping this strategy. ... View more

Thanks for the advice! While normally, I think readability is king, I was opting for the positive lookup to go along with the Smart List Best Practices #3: Use positive over negative operators - While "not" filters are available, they have to search the entire data set in your instance, which can be extremely time-consuming. Positive "is" filters are able to leverage more effective search algorithms. As long as we had good docs on how to set things up correctly, I'm hoping the confusion won't be too bad and that the potential performance will be worth it. Thinking about the backend, they probably make this recommendation because it's common that databases can't use an index for a negation query, but they can for instantaneous lookup for an equals query. https://stackoverflow.com/questions/1759476/database-index-not-used-if-the-where-criteria-is​ discusses this for SQL. I'm not sure what database Marketo Static Lists are stored in and if the same issue applies, but it seems like better safe than sorry when working with potentially millions of things. ... View more

Thanks for taking a look Denise. It reads in a confusing way, but I think it's doing what I want, while avoiding "not it" / negation for the best list querying performance although I haven't had a chance to test yet. The intent is that we don't remove from flow if you're still in the list, then continue on to remove you and send emails or fire webhooks. However, if you aren't in the list, I'd expect the default choice to kick in and remove you from the campaign to avoid firing the steps in case you were removed from the list by another concurrent campaign between list creation and flow execution. Does this seem right? ... View more

Seems like without the ability to lock, you basically can't ever implement at-most-once. Outside of Marketo we use Pub/Sub, which has at-least-once delivery as is common with distributed systems, so we have to track each message ID centrally in MySQL with locking to avoid duplicate processing. It seems like something similar to the list / flow I have in my screenshots in my other post are as close as I'm going to get in Marketo. Thanks for verifying, just wanted to make sure I wasn't missing some expert strategy. It would be kind of nice if Marketo couldn't run a specific SC for the same user more than once in parallel, but that's probably way too complicated in a distributed system. ... View more

Hi Denise, Yes in addition to lack of "locking read" support for exclusive access to the list within a few second time period (much like Sanford is talking about and I'm thinking about as an engineer), I'm also concerned about the gap of time between member list creation and campaign execution (from Under the Hood II: Batch Campaigns recording and How Campaign Processing Works ). Since it seems like there could be a big delay between member list creation / initial eligibility via the Smart List and the actual Flow steps, it seems like the secondary check would help. Hopefully that extra check doesn't slow down flows too much, but it seems like that's a fast (< 50 ms) operation so it's probably a good idea. Upon thinking about this race condition more, it seems like I want to remove the user from the list ASAP to reduce the time window between checking eligibility and list removal. Since flow steps like email and webhook sending might take a while, I'm now thinking I'd use a choice step to Remove From Flow as the first step if the user isn't in the list. Then proceed to remove them from the list and then do any flow like webhooks and emails. Here's the template I think we'll use for each campaign where we use a queue. ... View more

Yeah who knows, but glad it's working now. Thanks for sharing about the broken pattern matcher bug. It's good to know what kind of things can and have been going wrong, so we know it's a possibility before we go crazy ourselves. ... View more

I'm using the Munchkin JavaScript API to record page views, then using a trigger campaign to run automation when certain page views are seen. For example, we'd expect this trigger to fire on this page view, but it doesn't. The top yellow part is the web page URL, which is an exact page for Web Page: Contains. The bottom part is the querystring, and I've highlighted the matching part that should match contains at the very end (in hard to see light blue). What's more baffling is that when we fire similar page views, the trigger DOES work. This verifies that the campaign is set up in a sane way, but that there's something buggy going on with Marketo. For example, if we change certain characters in the tracked page URL or query string, the trigger works. However, we haven't found a pattern yet and we've had a support case open for multiple weeks, so I'm looking for more help. ... View more

Exactly what I was thinking in general for cases where we didn't have an event In our specific case, we write info to Salesforce first via our own backend, then that API only returns a hash to the frontend if the lead was already synced from Salesforce to Marketo. This way, new leads have a deferred association through Salesforce to prevent duplicate creation, but existing leads can be associated before events we want to take action on after email collection, are fired on the frontend. ... View more