Is there anything I can do to speed the getMultipleLeads API call?

[Anonymous]

I'd like to stay current on lead changes, so I've set up an app to call getMultipleLeads on an hourly basis, with a last modified threshold of an hour.  Or half an hour.  Or even ten minutes - it still usually takes over an hour to get any results.  If I pull as little as the last three minutes' updates, this may complete in just five minutes.  Obviously at that rate I won't be able to keep current.

We do have a fairly high volume of touches, as our data is synched with Salesforce on a near-real time basis.  Still, a report in the web GUI for all leads updated in the last hour returned "about 377" records (though it took several minutes to return the first page of results).  Surely 377 records shouldn't take minutes to process!

What can I do to keep current on lead changes?  Is there a way to speed getMultipleLeads, or is there a better approach I should be using?

To be clear, the span of time I'm measuring is literally the length of time for the getMultipleLeads method to return.  I'm using very verbose logging to pin this down.

[Anonymous]

One piece that can really help is reducing the number of fields you need. That can make things quite a bit quicker.

[Anonymous]

That (setting parameters.includeAttributes and parameters.activityFilter.includeTypes) has helped, but it's still extremely slow.  Examining leads updated in the last hour still often takes longer than 58 minutes, considering both leads and their activities.  Just 10,000 leads takes around ten minutes, and a typical hour sees 10 to 100K updated leads.

Is there a more scalable approach I should be using?  Or are daily updates the only realistic option?

[Anonymous]

Well, 100k an hour is pretty high volume, so getting some kind of backlog in those cases seems possible. Is the goal to sync all leads, or do you only really need a subset?

Also, you could possibly try a multi-threaded approach where you use the timeframe constraints on getMultipleLeads to have different threads taking different time slices.

[Anonymous]

For now, I'd like to capture changes to all leads.  If we were to reduce this to a subset (maybe only high-score leads?), could we use a smart list generated in the Marketo web GUI?  I see the LeadSelector parameter, but it seems to be limited to filtering by date or "static list", which based on the name I assume is distinct from a smart list.

Another option might be to update leads only daily, but activities hourly.  Unfortunately activities are taking more time than leads, even with an activityFilter in place.  There doesn't seem to be an equivalent to includeAttributes for the getLeadActivity method.  I am calling getLeadActivity once each for the tens-of-thousands of leads identified as having changed, which entails a lot of overhead; is there a method I've missed which would let me query activities posted in the last hour, for all leads?

I like your multi-threaded suggestion.  I've generally avoided doing so with APIs, as most use some kind of authentication token which cannot be used for multiple simultaneous queries, but if the Marketo API is cool with that, that opens up some posibilities!  I'll check that out next week.

Thank you for your suggestions, I'm hopeful that this will get me where I need to be.

[Anonymous · Accepted solution]

You can't do UI Smart Lists, but you can use recurring batch campaigns to populate the static lists.

Are you pulling all the activities for each lead? That is probably the core area where we can speed this up. There is a getLeadChanges API call which gives you new activities. Many folks use that for capturing what is needed. Even better, filering out some activity types will help. This is probably the most important thing to look at.

You can have multiple connections to the Marketo API at once, although keep it in the lower numbers (less than 5).

-Erik

[Anonymous]

Also, if you're making a call to fetch activity every time a lead changes (and 100k changes per hour) you're probably making a ton of API calls. You know the limit is at 10K/day right? It is a soft limit, but with what you described you may be going waaay over.

Hopefully getLeadChanges will help with all of that.

[Anonymous]

An auto-updated static list sounds like a good option, though if I can get adequate performance using other calls or multiple threads that would be even better.  I'd prefer not to have to choose a population of leads and assume the rest are not interesting.

Ah, so getLeadChanges actually gets recent activities across all leads?  That sounds like exactly what I need!  Probably paired with separate calls to get new leads.  I've set up some draft code to test this.

Yes, I'm filtering activity types.  I'll work with the marketing group to see if we can pare down what they want to monitor.

I'd not noticed that API limit, but I think you're right, getLeadChanges will address that nicely.

[Anonymous]

The getLeadChanges method has been working well for me in keeping current, thank you.

However, I'd also like to backfill old data, which entails calling getMultipleLeads with an early lastUpdatedAt param.  Unfortunately, this method takes forever to return the first rows for a large query, so there's no opportunity to break up the query into separate threads before it times out.

Have you successfully used the multi-threaded approach you mentioned?  If I could specify both start and end dates, it wouldn't be a problem, I'd pick up data one week at a time (or whatever date range was manageable).

[Anonymous]

Well, can't you specify both start and end dates for getMultipleLeads? We added that to the 2.0 version of the WSDL. The various activity calls don't have that option yet, but you can at least try getting old leads that way.