Batch campaign run to enrich & update 200K+ records

[Nina_Valtcheva5]

We have a new field (Leadspace Architect score) that I need to enrich all marketable leads in our database with which is over 200K. I'm concerned about breaking the SFDC sync or causing issues if I set up a campaign to enrich all leads at once. So instead, I'm thinking of updating 20K leads in one campaign run, so then I need a way to ensure the first campaign run has finished before the next run starts to update the next 20K. What is the best way to set this up?

[Robb_Barrett]

I believe SFDC only allows 100 updates in a single job. We see this happen a lot when we have a program that updates large batches of leads. Here's a link to the error you'll receive: System.LimitException: Too many SOQL queries: 101

https://help.salesforce.com/apex/HTViewSolution?id=000181404&language=en_US​

You might want to look at doing the update in SFDC instead of Marketo.

[Grégoire_Miche2]

Hi Robb,

Sorry, but the limit you are referring to has nothing to do with the volume of leads to be updated. It's about the number of distinct queries fired in a single transaction. You are getting this error because there are too many triggers that are fired as a consequence of an update. It can also be due to the fact that some of your triggers are not correctly developed to accommodate multiple lead updates in a single transactions.

More doc about the Governor limits in SFDC here: Salesforce Developers

-Greg

[Robb_Barrett]

Hmmm.  We had a campaign that wanted to update a field on a couple thousand leads and we hit that error.

[Grégoire_Miche2]

Hi again Robb

It can happen with 10 leads it there is a series of cascading triggers or if a trigger runs 1 query for each lead (which would be a design error, but it happens).

I am 100% sure of this, I have gone through this already

-Greg

[Grégoire_Miche2]

Hi Nina,

There is no specific reason why you should not launch the field update all at once. Anyhow, Marketo will not perform that update in 5 minutes, not even in 10 not 15. So their will be many Marketo-SFDC syncs during the update, in parallel. Furthermore, Marketo-SFDC sync does not sync thousands of leads at once, it does it with smaller batches.

Of course, pushing all this data to salesforce WILL take time, but if there are no other large transaction at the same time, Marketo will handle it. The main drawback is that, if you have other leads that need to be pushed to salesforce at the same time, they will pile up in the backlog and will have to wait. This might be a problem is this other lead needs to be pushed to SFDC rapidly.

Bottom line is: you would rather do it over a week end to make it smoother. You can also do it on a first batch of 20k and measure the time it takes. Then decide if you can process the rest in 1 batch or divide it in smaller batches.

-Greg

[Ed_Dunn]

is there a way for Marketo support to change MKTO to SFDC sync to use 100 per API call rather than 200 records per API call? This would get around the Apex CPU time limit exceeded and System.LimitException: errors