Marketo / Salesforce Sync Limits & Priority Info

[Anonymous]

Hi Everyone,

    Since the beginning of the year, I’ve been working with Marketo support on questions around the limitations and priorities between Marketo and Salesforce.com syncing. I have received some answers from Marketo Support that I have not been able to find anywhere else so I figured I’d share them here to spread the knowledge.

At the beginning of the year, we went through a rather large data migration. We knew from past experience that making updates to large amounts of data would cause a rather large sync backlog (one was as large as a several-day backlog). These backlogs would also result a major impact to our daily operations.

Even though we knew these backlogs would occur, we were never really sure how much data would actually cause the problem to happen. This time, we were hoping to get ahead of it and try to calculate the impact on our sync cycle based on the amount of data we needed to migrate.

Working with Marketo support, below is what we found to help us calculate how much data we could migrate at a time, without causing an impact to our operations:

• The sync cycle between Marketo and Salesforce.com will run approximately 10,000 record updates per hour, per object. Please note that some object types have different batch sizes. Only one object can sync at a time and Marketo handle them in the order of the schema / object list shown below.
  • For example: if you have 20,000 lead updates to pull and 20,000 contact updates to pull, Marketo will handle 10,000 lead updates, then 10,000 contact updates, then return to another 10,000 lead updates, finishing with the final 10,000 contact updates.
    • NOTE: This example does not consider any other objects being updated.
    • One caveat with Contacts is that updates will only occur as long as there is no backlog on the Account object.
  • The native sync between Marketo and Salesforce identifies records that need to be evaluated for an update based on the Last Modified Timestamp (also often called the system mod stamp). This timestamp is updated any time there is a change written to any field on the record (Note that formula and lookup fields do not update this timestamp). Whether one field is updated, five fields or a field that is not visible to the sync user, this update will queue for one evaluation for this record.
  • The creation of a new lead record (or other object) is an insert action. The initial sync from Marketo to Salesforce, must be through a Sync to SFDC flow step. The insertion process is a direct API call per record to Salesforce and can take 2-5 seconds to complete per record.
    • NOTE: the 2-5 seconds is just the timing for the sync process to create the lead and update the fields from the initial list. Our system also has several operational programs that append data and assign leads to sales territories. This information also needs to be considered as there can be additional updates queued after the initial insertion.
    • So, if you upload the list in Marketo, you will need to consider: 1) the insert action, 2) the number of fields in the list, 3) any Marketo operational smart campaigns that append data, and 4) any Salesforce logic that appends data and assigns leads to Sales. This Salesforce data still needs to sync back to Marketo after it is appended to the record. So this data should also be considered.
    • If the size of your data is large enough to trigger a sync backlog, the sync will continue to operate in a First in First out order.
      
      
• Forcing a sync using Marketo’s “Sync Lead to SFDC” flow step will always take priority over the standard sync cycle, and will push the update through even if there is a current sync backlog. However, make sure to use caution as the Salesforce flow steps can use anywhere between one and five API calls. Large pushes through these flow steps will use your API quickly.
  
  
• Marketo syncs schemas and objects in a very specific order. So, if you are making updates to multiple types of objects at the same time, the sync will update one object type at a time.  Below is the order that Marketo prioritizes these objects
  • Lead: synch schema
  • Account: synch schema
  • Contact: synch schema
  • User: synch schema
  • Task: synch schema
  • Opportunity: synch schema
  • Opportunity Contact Role: synch schema
  • Campaign: synch schema
  • Campaign Member: synch schema
  • Campaign Member Status: synch schema
  • Lead: pull conversions
  • Account: pull deletes
  • Account: pull updates
  • User: pull updates
  • Lead Queue: pull updates
  • Lead Status: pull updates
  • Account Type: pull updates
  • Lead: pull deletes
  • Contact: pull deletes
  • Lead: pull updates
  • Contact: pull updates
  • Lead: push updates
  • Contact: push updates
  • Campaign: pull deletes
  • Campaign Member: pull deletes
  • Campaign Member Status: pull deletes
  • Campaign: pull updates
  • Campaign Member: pull updates
  • Campaign Member Status: pull updates
  • Opportunity: pull deletes
  • Opportunity Contact Role: pull deletes
  • Opportunity: pull updates
  • Opportunity Contact Role: pull updates
  • Event: pull updates
  • Task: pull updates
  • Email Template: push new
  • Email Template: push updates
  • Task: push new
  • Task: push updates
    
    
• The above are all standard objects. Any custom objects you may have created will be prioritized after all of the standard objects have synced. Reminder: Contact updates will only occur as long as there is no backlog on the Account object.
  
  

Getting around the sync backlog

• Using the above info, you should be able to identify roughly how long the sync will take based on the amount of records and fields you will be updating.
  • There isn’t really a way around this that I have found except for spreading out any large data updates over time. I asked Marketo if there was a way to get a temporary increase in bandwidth for the sync cycle while we performed our annual maintenance, the answer was no. The speed in which the sync operates is dependent on Salesforce’s API. Increasing the batch sizes will only increase the duration of the sync cycle for the object that has been increased, it will not increase the throughput for the overall sync process.
  • As is a recommended best practice, if your update is going to be very large, it is recommended to break down the update into smaller chunks and spread them out over time to avoid this backlog. Also, schedule the updates to occur outside of business hours, to avoid any impact to your daily operations.
    
    
• Take note of any sales alerts that this update could impact. For instance, if you have a sales alert sent out of Marketo, and you are currently experiencing a backlog, Marketo might alert Sales of information that doesn’t exist in Salesforce yet. This can definitely create frustration among Sales!
  • If volume here is relatively low, you can always consider adding a “Sync Lead to SFDC” flow step to trigger before your alert is sent. Just be careful with how many of these are used as each time this flow step occurs, it counts as an API call. Depending on the system you are syncing to, some have limits on how many API calls they can receive in a day.
    
    
• Document any operational rules that automatically append data in different scenarios (e.g. when a new lead is created) and how many fields the operational rules update. This should be done for both Marketo and your CRM so you know how many total fields will be updated when performing large list uploads or data migrations.
  
  

That’s about all I have for this right now. Hope this information can help anyone else here in the community.

Thanks!

Jason

NOTE: Post updated on 04/03/17 to clarify a few technical details provided by Marketo support.

[Chris_Saporito]

This is great information. We run into a lot of these issues. When we run territory adjustments, we do so over the weekend and for a long time it seemed to do the trick. Recently we have seen that even doing so we see a backlog for a couple of days. Definitely frustrating! Curious what other companies are doing, I feel that we cannot be the only ones to "update" 500K accounts at a time.

[Dan_Stevens_]

Is this something Orion will help once it's fully deployed?

[Anonymous]

Hi Dan,

     I just asked Marketo support about this and they said no, Orion will not help with the syncing process. They mentioned that the "sync is a separate entity and the Orion project only has an effect on internal processes."

Jason

[Anonymous]

We were on a call with our Marketo rep today, and he had a slide showing Salesforce Sync Acceleration as a roadmap item under "Post-Orion PSR Enhancements".

[Grégoire_Miche2]

Hi Jason,

Thanks for this detailed and very informative post.

A few comments :

• I have obtained from support a temporary increase in sync bandwidth for a large insertion of records (>3 000 000 records)
• The limit of 10000 field updates per hours seem very low to me compared to my experience. for instance, I create 4600 leads in Marketo from SFDC in 10 minutes with 6 fields in each record.

-Greg

[Anonymous]

Hi Greg,

     Thanks for the input! That's unfortunate to hear about the temporary increase in sync bandwidth, I put in a support case asking specifically for this and they told me that it was not possible and that I need to break up my data and perform my annual updates over a longer period of time.

As for the 10,000 field updates, I agree, it sounds very low from what I've experienced with past systems. I received this number from two different support contacts at Marketo, so I am wondering if it is a "soft" limit that they try to encourage customers to stick to.

Jason

[Anonymous]

Hi Greg,

    I found out more information on this:

• The 10,000 updates are per object, not per field, as I originally thought. Marketo does say that this number is a rough estimate and will vary based on the object type.
• When I asked for clarity around the bandwidth increases, they mentioned that they cannot raise the total bandwidth as this is a limitation of the Salesforce API. What can be changed (on a circumstantial basis) is the amount of records that are synced per object type before moving onto to the next object type.
  • From what they explained, the sync process handles objects in a "waterfall-style" where it will sync ~10,000 of one object type, then continue to the next object type. If the amount of updates on an object is greater than the ~10,000 cap, the sync will not just sit on that single object until it is done with all updates. It will sync the ~10,000 updates on the object type, then move to the next object type and will go through the list I have above in that order.
  • So, what the increase can do, is lift the cap from ~10,000 to, say, 20,000 so it can handle more of one object at a time. But this would only be done in a situation where there needed to be a one-time effort to do a large update on a single object type, like the example you provided.

Jason

[Grégoire_Miche2]

Hi Jason,

Thx, this is much more in line with my own experience

-Greg

[Mike_Reynolds2]

Hi all. Just wanted to drop a note in to mention that we had some incorrect information given out in one of our Support cases, which is why Jason edited the post above. Felt it was was worth mentioning that this is an excellent discussion with some really good detailed info. The clarification edits made were due to errors on our side in the case. We got one of our integration experts to confirm that the info in the original post above is 100% accurate. Again, excellent post with lots of great information!