AnsweredAssumed Answered

Lead API bottleneck & multiple threads

Question asked by Michael Tyson on Jul 18, 2017
Latest reply on Jul 18, 2017 by Michael Tyson

We’re working on an automatic creation/update script for Leads in Marketo. Currently, we pull our data and decide to send an update or create request. I batch these in groups of 300 or payload size of ~1MB to comply with the API restrictions. We wound up going with option A in https://nation.marketo.com/thread/40577-syncing-data-with-marketo.

 

We’ve come across a case of needing to periodically update every record in Marketo from our data in our CRM, such as adding or deleting entire columns. The CSV upload functionality in Marketo won’t work for this situation as we need to maintain data so we need to use the API.

 

Currently, my script takes about 1 minute to process 1,000 records (give or take a few seconds). Roughly 95% of that time is spent sending that data to Marketo and waiting for the response so we can process.

 

So, we’ve been talking about running multiple threads of the script at the same time to get more data inserted and updated (it’s written in PHP to conform with the rest of our system).

If we do a modulo operation on our audit ID, we can break the records in to sets of 1-10. We then have 10 cron entries each passing which modulo set to use for that thread.

 

Assuming we maintain the ability to process 1,000 records per minute, we’ll then need to handle the 100 calls per 20 seconds. Each call has a max of 300 records so we can process a max of 3,000 per 20 seconds, or 9,000 per minute. This is all assuming peak performance which I’m not 100% sure of yet.

 

 

My first question is, is there any way to improve Marketo’s response time? What’s the bottleneck on that end?

 

Secondly, this solution adds a lot of complexity to an already complex process so I’m wondering if anyone else has experience with something along these lines and what their solution to it was. It’s extremely important that when doing these large data updates we continue to process updates from in the system as well.

Outcomes