Jon_Wu

In case anybody else is hitting this, I tried to sign up for the beta twice in Chrome and again in Safari, but I got an error when trying to sign the contract. The contract form just seems to be broken, but the signup itself is OK. I emailed  Steven about it. ... View more

I've been a user of said forbidden methods for a while (set up once years ago, and had just been chugging along). I recognize this was not recommended, but we needed it to get around some limitations (e.g. concurrency) with the REST API, so I'm glad this beta is an option and will surely be signing up.   Performing anonymous person association for thousands of leads in a short amount of time without this wasn't possible. We also defer all associations to wait for Salesforce to Marketo sync to occur, which is why they need to be done server side, otherwise we get duplicate leads.   @Steven_Vanderb3 in the meantime, I was wondering if the latest October release had finally shut the door on the forbidden server-side POST. On the night of October 16, we started getting a 404 response when trying to server-side POST and wanted to know if this was an intentional permanent change, and if this beta would be the best way to replace our previous integration. ... View more

Good to hear! In practice, do I need a significant delay before checking the 2nd box? My concern is that if I update 1 million contacts in Salesforce, both boxes will sync at the exact same time due to the sync backlog (which might take days at that volume). Can't really think of a way around this though besides having a huge delay. ... View more

Hi Veronica, Thanks for that insight. I agree that people could be confused into thinking you could hide something that was already synced so I'm trying to think how we could work around that. We're trying to keep a superset of people in our Salesforce instance and trim people from Marketo to save money on people who aren't valuable to email to. We're about to build this out, but I'm wondering if we could also detect when somebody who was previously synced to Marketo was set to not be synced anymore after a checkbox update in Salesforce. Ideally, if we could detect that sync was turned off for a given lead from the Marketo side, we could just delete them automatically, but it seems like this wouldn't happen since sync would just be blocked and we wouldn't know about it from the Marketo lead's perspective. Maybe we can have a trigger that calls the Marketo API to delete the lead when the checkbox is checked. Downside here is that this requires light programming. Alternatively, we could have 2 checkboxes in Salesforce. One of them is user-facing and isn't the one that filters the Marketo sync. When you change it, we can use a Salesforce workflow rule to wait 1 day (to be pretty confident that this value has been synced back to Marketo), then check 2nd box that tells Marketo to skip the sync. Then we can have a Smart Campaign on the Marketo side that waits to see a change of the first checkbox value to known when to delete the lead from Marketo since that will be synced before we use the 2nd checkbox to stop syncing. ... View more

Veronica Holmes​ do you mainly recommend against "Hide from Marketo" because it's a negative? Are there any technical reasons not to go that route? For an existing database of millions of leads that should be synced by default (and are already synced), a technical reason to go with the negative from the Salesforce side is that you can't add a new field on leads and automatically populate all existing leads with it. You'd have to do a bulk data load. By checking the box, the UX is something that's opted in to, and isn't the default. I think of it kind of like unsubscribe, so if the usual case is to be synced, but you just want to deprecate some people, it seems like having a "Hide from Marketo" field could have some benefits. ... View more

Darn that's what I figured in terms of Marketo being unhappy with high peaks. We'll have to find a way that doesn't overwhelm it and slow everything down. Or if thousands to 10s of thousands per hour is fine usually, maybe we just suffer on peak days, but that's not ideal. For this particular scoring use case, I don't technically need the scores to be always up to date, but then the problem is that if they aren't, it becomes difficult to know if it's safe to use the latest score or not. For example, one strategy I have now is to use a webhook to update the category of a lead's highest score, but then do it at most once every 2 hours, then wait 3 hours and run the webhook again. This is an approximation of throttling and is much better than firing webhooks on every page view for performance. The hard part is that it makes it more difficult for automation to work since you can't really trust whether the score is up to date, or whether it's a few hours old. ... View more

Up to hundreds of thousands or millions per hour during high peaks ... View more

Hi Josh. Not sure I follow. Are you saying that maybe I could prevent the decrement from happening if I could tell that events were being replayed during association? Even if I could prevent it from happening at that moment, I'm not sure how I would actually decrement the score by the appropriate amount in 14 days. Thanks for your help ... View more

Hi Sanford. It seems like what I have down gives some balance between adding a little safety and not increasing complexity too much, adding external dependencies, or slowing down performance by more than a couple hundred ms, but I'd love to hear more about your thoughts on compare-and-swap via webhook. First, I think what I have now in the screenshot reduces the race condition window to a reasonable size. There's probably a 50-100 ms window between where we check to remove them from the flow and when we actually get them removed from the list. Not ideal, but I don't expect too much concurrency except when a cron / scheduled Smart Campaign overlaps with triggers. As I'm typing this, another way I could reduce the likeliness of a race condition is to add a wait step before putting people into a cron list, so that the cron is truly more of a fallback and won't compete with triggered campaigns that execute immediately, but that would add yet another step. When new leads are created OR when leads are associated, I try to kick off actions to send emails that were triggered by activity that was logged while the lead was anonymous. The ideal scenario is that I can use a trigger on a server-side form post that I use to associate leads, but if that fails or if I'm associating via REST API, I depend on 24 scheduled batches (running once per hour) to try and flush the queues out as a fallback. If any of those scheduled campaigns run at the same time as the lead association, that's where the race is most likely to happen. The super ugly part with all of this is that I'd have to have our marketing team implement all of these things every time the want to set a triggered email, which seems tedious. If I was using a webhook to do a lock externally, I'd need to set up another service, have a key/value store with tracking for every campaign that's running, and then I think I'd be writing back to a Marketo field with the result by campaign, so I'd need a new lead field for every campaign that I run. Or this kind of what you're thinking with a compare-and-swap via webhook? If so, I think that's probably too much. Worst case if the race condition happens, is somebody gets 2 emails. If that happens 1 in 100,000 times that's probably not a huge deal. While I agree the smart thing to do would be to never use an inequality query, but I was imagining if that were the case, the Smart List Best Practices doc wouldn't need to recommend use positive over negative operators, but this is just speculation of course. Either way, I think these lists will probably be small so Denise's advice to go with more readable solution may be the most practical if I was keeping this strategy. ... View more

Thanks for the advice! While normally, I think readability is king, I was opting for the positive lookup to go along with the Smart List Best Practices #3: Use positive over negative operators - While "not" filters are available, they have to search the entire data set in your instance, which can be extremely time-consuming. Positive "is" filters are able to leverage more effective search algorithms. As long as we had good docs on how to set things up correctly, I'm hoping the confusion won't be too bad and that the potential performance will be worth it. Thinking about the backend, they probably make this recommendation because it's common that databases can't use an index for a negation query, but they can for instantaneous lookup for an equals query. https://stackoverflow.com/questions/1759476/database-index-not-used-if-the-where-criteria-is​ discusses this for SQL. I'm not sure what database Marketo Static Lists are stored in and if the same issue applies, but it seems like better safe than sorry when working with potentially millions of things. ... View more

Thanks for taking a look Denise. It reads in a confusing way, but I think it's doing what I want, while avoiding "not it" / negation for the best list querying performance although I haven't had a chance to test yet. The intent is that we don't remove from flow if you're still in the list, then continue on to remove you and send emails or fire webhooks. However, if you aren't in the list, I'd expect the default choice to kick in and remove you from the campaign to avoid firing the steps in case you were removed from the list by another concurrent campaign between list creation and flow execution. Does this seem right? ... View more

Seems like without the ability to lock, you basically can't ever implement at-most-once. Outside of Marketo we use Pub/Sub, which has at-least-once delivery as is common with distributed systems, so we have to track each message ID centrally in MySQL with locking to avoid duplicate processing. It seems like something similar to the list / flow I have in my screenshots in my other post are as close as I'm going to get in Marketo. Thanks for verifying, just wanted to make sure I wasn't missing some expert strategy. It would be kind of nice if Marketo couldn't run a specific SC for the same user more than once in parallel, but that's probably way too complicated in a distributed system. ... View more

Hi Denise, Yes in addition to lack of "locking read" support for exclusive access to the list within a few second time period (much like Sanford is talking about and I'm thinking about as an engineer), I'm also concerned about the gap of time between member list creation and campaign execution (from Under the Hood II: Batch Campaigns recording and How Campaign Processing Works ). Since it seems like there could be a big delay between member list creation / initial eligibility via the Smart List and the actual Flow steps, it seems like the secondary check would help. Hopefully that extra check doesn't slow down flows too much, but it seems like that's a fast (< 50 ms) operation so it's probably a good idea. Upon thinking about this race condition more, it seems like I want to remove the user from the list ASAP to reduce the time window between checking eligibility and list removal. Since flow steps like email and webhook sending might take a while, I'm now thinking I'd use a choice step to Remove From Flow as the first step if the user isn't in the list. Then proceed to remove them from the list and then do any flow like webhooks and emails. Here's the template I think we'll use for each campaign where we use a queue. ... View more

Yeah who knows, but glad it's working now. Thanks for sharing about the broken pattern matcher bug. It's good to know what kind of things can and have been going wrong, so we know it's a possibility before we go crazy ourselves. ... View more

I escalated to our account manager who said somebody would call me yesterday morning, but then instead I got an email saying that they won't support us customizing the URL (or having a fake URL as they put it), despite the docs saying: You can customize the URL and querystring which are sent with the data object in the second argument. They also gave a bogus reason that they filter out stuff for abuse and spam prevention and said stuff has to be on our domain (which it is). This also isn't true since all our random dev domains record actions anyways. Since they seem to be so caught up that me logging page views at /_event/... isn't cool, I just made the URL an actual page, and submitted another screencast to them showing that this still doesn't work, even if it's a real page. Waiting to hear back from support escalations now. Separately, another support rep just told me that the case sounded like something they'd be able to handle, but that I just need to keep hammering escalations until I'm heard. ... View more

Thanks for the offer, I hope I don't get there, but you never know. The worst part is, in my screencast, it's the fake URLs that do work, and the real URL that doesn't work. Oh well, hoping we'll find somebody to bump the case back into play. Please let me know if you hear any good updates in yours. ... View more

I don't know, but it's super ridiculous. Just check out their response: Hello Jon, per Tier 3 they can see that munchkin tracking is working as expected. Since you are saying, that not all the web page visit activities are not showing for an invalid/fake website (per your testing), then this will be considered a request for us to trouble shoot your custom code on your website. Unfortunately, this is not within the parameters of our support. Per Tier 3, you will have to work with your account manager to put you in touch with our Professional services team to debug your code. I suppose they're trying to say that it's expected for Munchkin to only work sometimes? In addition, I'm pretty offended that they call our production website "invalid/fake" per the tests I did on production website, in which I included a screencast and showed the dev tools and activity log side by side. This makes me want to start blogging about our Marketo experiences. ... View more

So frustrating, they closed my case, said everything works fine, and we can pay professional services for help. Infuriating! ... View more

Are you on Munchkin v2? I was speculating that maybe a v2 upgrade could fix our issues, but if you're on v2, perhaps there's no immediate hope. If Munchkin is going to have a case of the Munchkin-Maybes, we'll probably just re-implement basic event tracking through Salesforce and create  new Salesforce object for each would-have-been Munchkin activity to trigger our campaigns instead. Not ideal, but I don't see any good ways forward since we can't depend in web page view activities being logged, AND once they're logged, they don't always trigger correctly (Are there known gotchas where Smart Campaign triggers always fail on certain page views? ). ... View more

Very interesting Sanford! For your weird URLs, does a given URL + querystring combo break consistently, or does it seem random? The breakage feels random but appears to deterministic from what I've observed in our case so far. Since I can't figure out what's going wrong, I've been unable to find a workaround because it seems like when I change some random things in the URL, one page starts to get logged, then another stops getting logged. It's like playing whack a mole. Random hypothesis: perhaps the activities are getting sharded, but one node is failing? I'm stumped on what to do for now. ... View more

I'm using the Munchkin JavaScript API to record page views, then using a trigger campaign to run automation when certain page views are seen. For example, we'd expect this trigger to fire on this page view, but it doesn't. The top yellow part is the web page URL, which is an exact page for Web Page: Contains. The bottom part is the querystring, and I've highlighted the matching part that should match contains at the very end (in hard to see light blue). What's more baffling is that when we fire similar page views, the trigger DOES work. This verifies that the campaign is set up in a sane way, but that there's something buggy going on with Marketo. For example, if we change certain characters in the tracked page URL or query string, the trigger works. However, we haven't found a pattern yet and we've had a support case open for multiple weeks, so I'm looking for more help. ... View more