Long delay between Marketo and SFDC?

[Anonymous]

I have our Marketo support person on this as well but I'm posting this to see if anyone else has experienced this with their SFDC/Marketo sync.

Around 3 days ago our Marketo/SFDC sync either completely stopped working or is on a mutliple day delay and as far as I know nothing has changed on either end. It's set to sync via the program/campaign sync function on the program setup level in Marketo which has worked for as long as I can remember until now. Marketo says our sync is set up properly and the campaign/program statuses match, so they're still searching. Spot checking some other leads it looks like some of our website leads for example are downloading an asset but then not being added to the campaign until 2-3 days later.

If I manually run a flow step to "add to sfdc campaign" it works fine, but I've never had to do this before. 

Anyone ever experience this? 

[Anonymous]

EDIT: Posted from the wrong account. Please delete this, admins!

[Edward_Unthank_ · Accepted solution]

Hey Amanda,

You're getting into the dredges of how API calls and how the sync works between Marketo and Salesforce, and a new (for most people) consideration of how to make syncing more efficient. 

First off, the behind-the-scenes work between Marketo and Salesforce. Generally it looks like a magic "it all works" natively, but really Marketo is just a really fancy, built-out API integration with Salesforce just in the same way as any other API integration into Salesforce. What this means really is that Marketo is subject to a limited amount of API calls to SFDC just like every other app, and there are some efficient ways to use API calls and some inefficient ways to use API calls. 

When your database is at small sizes (generally under 100k leads), it's pretty rare to run up against sync issues like this (unless something wonky happened like your SFDC-Marketo sync user credentials changed/expired). Once you start running into the API calls-per-day limit, you'll notice things like delays. In reality, every time a lead's value changes or a Marketo/SFDC campaign membership changes or a lead is created, that requires a call to SFDC to update. 

API Call Efficiency between Marketo and Salesforce

• "Sync lead to SFDC" is a 1:1 for API calls, aka very inefficient use of API calls.
• "Add lead to SFDC Campaign" is a 1:1 for API calls, aka very inefficient use of API calls.
• A Marketo program/SFDC campaign is (if I remember right) around a 200:1 for API calls, aka you can sync 200 leads with one API call.

  
Timeliness of sync between Marketo and Salesforce

• "Sync lead to SFDC" and "Add lead to SFDC Campaign" is immediate, without a queue. When a lead runs through that flow step, the sync to SFDC happens.
• Marketo program/SFDC campaign sync happens through a queue every 7 (or so) minutes. That's the one that you see in the Admin section of Marketo, where "Last time Salesforce was synced" is. While this happens every 7 (or so) minutes, there's a set limit of campaign syncs happening with each call, and extra campaign syncs will be queued, waiting for a later program/campaign sync opportunity if there's room next time. SO, Marketo program/SFDC campaign syncs are more API-call efficient, but you run into queueing bottlenecks faster. I don't remember exact throughput numbers, but when I was doing a huge sync, the leads-synced-per-hour was around 8,000. 
• Lead updates also happen every 7 (or so) minutes. That is, when you update a field value on a lead in Marketo, it's checked against SFDC every time to see if the value is different, then if it's different, Marketo updates the new value in SFDC (or vice versa). This faces the same lead-value-change queueing problem as Marketo programs/SFDC campaigns.

 

What does this actually mean, then?

• If you're running out of API calls on a daily basis, check and remove "sync to SFDC" flow steps and instead sync via Marketo programs/SFDC campaign syncs. 
• SFDC workflows that update lead fields with a new value on a daily (or more frequent) basis mean that every updated lead will add to the queue and API limit of updates. "Days since created" that actually changes the lead field means that you have to resync your entire database every day—at small database sizes, might not be a problem, but at 100k+, you're looking at a backlog that is going to slow all API-sync related processes.

 
Munchkin API—the API you never thought about.

• Munchkin communicates with your database through API calls, but it's seamless enough that you don't know about it. 
• "Visits web page" is the one piece that constantly ends up being a "my Marketo activity is being really slow for some reason" gotcha—if you have lots of web activity on your website (I'd consider around 80k sessions per day to be "lots"), your "web activity" ends up queueing. If you're running triggers off of "visits web page," web page visits is going to be too noisy to be reliable for timely actions. With large-volume websites, I pretty consistently see this delayed by multiple hours before "visits web page" shows up in a lead's activity log. This is considered a more "noisy" activity. Imagine if each session consists of 1.5 page visits, that's 160k activities to be logged. (Then spread that out over business hours, so around 16k page visits per hour.) That's a ton of activity to log.
• "Fills out Form" is an activity that is more reliable and timely for triggering and showing up in Marketo. If you have 100k visitors per day, and 7 out of 100 fills out a form (a desirable form conversion rate, and probably an exaggeration for actual traffic numbers), you only have 7k form fills per day. With the same business-hour spread, that's around 700 form fills per hour.

 
You can see this intuitively makes sense if you imagine that each Marketo Munchkin action is just a big team of interns in a room typing in separate actions into each of your leads' Activity Logs—when you only have a few hundred visitors per day, your intern team can type fast enough that you don't notice. But when you have tens of thousands of web visits per day, and each visitor visits something like 10 pages per visit, you're going to overwhelm that intern team. In the example above, you'd be picking between working at 700 activities per hour (form fills) versus 16,000 activities per hour (web page visits).      

The problem you're talking about could be any of these. When you're on the upper end of database size (400k+ of inbound leads—if you're doing list purchases a lot, you can deal with a larger database with fewer problems, because you have a significantly lower percentage of leads actually active... also shame on you for doing that much list purchasing), you're reaching the right side of the bell curve for Marketo database sizes. Marketo isn't built for this much volume, and customers at that size represent a small percentage of revenue—so don't expect to have the Product Team prioritize your "this is really slow and inefficiently built" screams when there's other work they can do to bring in a greater amount of money sooner. You'll have to tweak what you're doing to play into existing efficiencies to get the performance you need. 


Best,
Edward Unthank | Founder, Etumos

[Anonymous]

We've had the same issues where syncs were waiting 3-4 hours or even longer; when the standard sync behavior is supposed to be roughly every five minutes after the last sync (with SFDC). We are currently using SFDC Performance edition that does not have any API issues. I opened a case and it was escalated to tier 2 and was investigated.

Eventually the case was closed and the issue addressed as such:

After a review of the sync process our engineers have implemented some improvements on throughput that will help to clear up bandwidth during the sync. You should notice the improvements over the next week.

I get the sense that there are some bandwidth issues in Marketo's infrastructure.

If you need more help I can provide you our case number if you want to chase it.

[Anonymous]

Thanks for the speedy and thorough responses!

@Edward I very rarely run "Add to Campaign" as a flow step for the very reasons outlined above -- I do my syncing through the program sync as you described. I do have several behavior scoring campaigns that have been running since December and haven't posed any issue, but maybe that's part of it. 

One of the scenarios you described is intriguing though -- we just moved to a global form on our website, so one form for all assets to make form updates easier. The trigger on our content campaigns is now Fills out From is Global Download Form with a constraint for Visits Webpage is (xyz page). I wonder if that might be doing it? I've set up my content campaigns like this at multiple companies so this is still kind of puzzling to me but I suppose anything is possible. I don't think we have anywhere near 80K sessions per day... 

@Mark How long ago did you log your case? Did you notice better performance after they closed the case? I see that the sync is running every 7 minutes or so as it's supposed to, just nothing seems to be happening. 

My case has now been escalated as well so hopefully they figure something out soon. I'll ask them to see if those global download campaigns have anything to do with it.

Thanks again for your help!


 

[Anonymous]

We opend our case on 2/3. What we were seeing in Marketo's SFDC page was no update on 'Last Sync with SFDC' which made tech support initially think that something changed with Marketo's login to SFDC. But in SFDC we could see that every 5 minutes Marketo was successfully connecting to SFDC.

Also, there's a syncing viewing tool that Marketo can enable--it ain't pretty. It will allow you to check--by SFDC object--where it is in terms of syncing (how many records, direction of sync, that kind of stuff). If you know there's a large sync in progress and you want to make sure everything is up-to-date, it can be handy.

[Anonymous]

Got it -- didn't know about the sync viewing tool. Thanks for the info. I added some of these suggestions to our support case, so appreciate your responses.  

[Anonymous]

Just an update on this in case anyone finds this thread useful:

Turns out we had a HUGE backlog of activity in Marketo trying to sync to SFDC, so it was kind of a combo of a few things going on. A handful of programs that get very high traffic were triggered off of a "visits web page" contains "xyzlink parameters" -- which as Edward suggested can slow things down. Additionally because those programs had so much activity passing through them and leads were allowed to enter the flows more than once, Marketo ended up trying to add people to several different SFDC campaigns at once, which obviously doesn't work so well. 

Basically we are just letting it do its thing -- Marketo support was able to open things up so we could get through the backlogged activity a bit faster. I'm working to edit the campaigns so that leads aren't added to mutliple campaigns at once. 

Thanks for your feedback, I think it helped solve this problem a little faster than it would have otherwise. 

[Anonymous]

Awesome explainer Edward Unthank​.