Since the beginning of the year, I’ve been working with Marketo support on questions around the limitations and priorities between Marketo and Salesforce.com syncing. I have received some answers from Marketo Support that I have not been able to find anywhere else so I figured I’d share them here to spread the knowledge.
At the beginning of the year, we went through a rather large data migration. We knew from past experience that making updates to large amounts of data would cause a rather large sync backlog (one was as large as a several-day backlog). These backlogs would also result a major impact to our daily operations.
Even though we knew these backlogs would occur, we were never really sure how much data would actually cause the problem to happen. This time, we were hoping to get ahead of it and try to calculate the impact on our sync cycle based on the amount of data we needed to migrate.
Working with Marketo support, below is what we found to help us calculate how much data we could migrate at a time, without causing an impact to our operations:
- The sync cycle between Marketo and Salesforce.com will run approximately 10,000 record updates per hour, per object. Please note that some object types have different batch sizes. Only one object can sync at a time and Marketo handle them in the order of the schema / object list shown below.
- For example: if you have 20,000 lead updates to pull and 20,000 contact updates to pull, Marketo will handle 10,000 lead updates, then 10,000 contact updates, then return to another 10,000 lead updates, finishing with the final 10,000 contact updates.
- NOTE: This example does not consider any other objects being updated.
- One caveat with Contacts is that updates will only occur as long as there is no backlog on the Account object.
- The native sync between Marketo and Salesforce identifies records that need to be evaluated for an update based on the Last Modified Timestamp (also often called the system mod stamp). This timestamp is updated any time there is a change written to any field on the record (Note that formula and lookup fields do not update this timestamp). Whether one field is updated, five fields or a field that is not visible to the sync user, this update will queue for one evaluation for this record.
- The creation of a new lead record (or other object) is an insert action. The initial sync from Marketo to Salesforce, must be through a Sync to SFDC flow step. The insertion process is a direct API call per record to Salesforce and can take 2-5 seconds to complete per record.
- NOTE: the 2-5 seconds is just the timing for the sync process to create the lead and update the fields from the initial list. Our system also has several operational programs that append data and assign leads to sales territories. This information also needs to be considered as there can be additional updates queued after the initial insertion.
- So, if you upload the list in Marketo, you will need to consider: 1) the insert action, 2) the number of fields in the list, 3) any Marketo operational smart campaigns that append data, and 4) any Salesforce logic that appends data and assigns leads to Sales. This Salesforce data still needs to sync back to Marketo after it is appended to the record. So this data should also be considered.
- If the size of your data is large enough to trigger a sync backlog, the sync will continue to operate in a First in First out order.
- Forcing a sync using Marketo’s “Sync Lead to SFDC” flow step will always take priority over the standard sync cycle, and will push the update through even if there is a current sync backlog. However, make sure to use caution as the Salesforce flow steps can use anywhere between one and five API calls. Large pushes through these flow steps will use your API quickly.
- Marketo syncs schemas and objects in a very specific order. So, if you are making updates to multiple types of objects at the same time, the sync will update one object type at a time. Below is the order that Marketo prioritizes these objects
- Lead: synch schema
- Account: synch schema
- Contact: synch schema
- User: synch schema
- Task: synch schema
- Opportunity: synch schema
- Opportunity Contact Role: synch schema
- Campaign: synch schema
- Campaign Member: synch schema
- Campaign Member Status: synch schema
- Lead: pull conversions
- Account: pull deletes
- Account: pull updates
- User: pull updates
- Lead Queue: pull updates
- Lead Status: pull updates
- Account Type: pull updates
- Lead: pull deletes
- Contact: pull deletes
- Lead: pull updates
- Contact: pull updates
- Lead: push updates
- Contact: push updates
- Campaign: pull deletes
- Campaign Member: pull deletes
- Campaign Member Status: pull deletes
- Campaign: pull updates
- Campaign Member: pull updates
- Campaign Member Status: pull updates
- Opportunity: pull deletes
- Opportunity Contact Role: pull deletes
- Opportunity: pull updates
- Opportunity Contact Role: pull updates
- Event: pull updates
- Task: pull updates
- Email Template: push new
- Email Template: push updates
- Task: push new
- Task: push updates
- The above are all standard objects. Any custom objects you may have created will be prioritized after all of the standard objects have synced. Reminder: Contact updates will only occur as long as there is no backlog on the Account object.
Getting around the sync backlog
- Using the above info, you should be able to identify roughly how long the sync will take based on the amount of records and fields you will be updating.
- There isn’t really a way around this that I have found except for spreading out any large data updates over time. I asked Marketo if there was a way to get a temporary increase in bandwidth for the sync cycle while we performed our annual maintenance, the answer was no. The speed in which the sync operates is dependent on Salesforce’s API. Increasing the batch sizes will only increase the duration of the sync cycle for the object that has been increased, it will not increase the throughput for the overall sync process.
- As is a recommended best practice, if your update is going to be very large, it is recommended to break down the update into smaller chunks and spread them out over time to avoid this backlog. Also, schedule the updates to occur outside of business hours, to avoid any impact to your daily operations.
- Take note of any sales alerts that this update could impact. For instance, if you have a sales alert sent out of Marketo, and you are currently experiencing a backlog, Marketo might alert Sales of information that doesn’t exist in Salesforce yet. This can definitely create frustration among Sales!
- If volume here is relatively low, you can always consider adding a “Sync Lead to SFDC” flow step to trigger before your alert is sent. Just be careful with how many of these are used as each time this flow step occurs, it counts as an API call. Depending on the system you are syncing to, some have limits on how many API calls they can receive in a day.
- Document any operational rules that automatically append data in different scenarios (e.g. when a new lead is created) and how many fields the operational rules update. This should be done for both Marketo and your CRM so you know how many total fields will be updated when performing large list uploads or data migrations.
That’s about all I have for this right now. Hope this information can help anyone else here in the community.
NOTE: Post updated on 04/03/17 to clarify a few technical details provided by Marketo support.