getting all leads from REST API - lead IDs are not sequential

[Anonymous]

We are trying to get all the leads using the method mentioned here:
http://developers.marketo.com/blog/get-all-leads-from-the-marketo-rest-api/

and in particular the programatic way referenced from there to here:

Get all leads programmatically in Marketo v1 - Stack Overflow

The problem is, that in the account we test with, the leadIds are combined of 2 different sequence of numbers, one that starts at 1 (and currently for the instance we check ends at around 16,000), and one that starts at around 1,000,000 (1 million).  They interlace each other, and we don't know what's the reason and logic behind this.

This means that if we try to get the last lead ID using "get activities" for activity type 12 (new leads), we might get say something like 16,000 but we also might get something like 1,000,000 and it's a matter of luck. It also means that if we do get the millions, there would be a huge gap where we would query IDs that are simply not there (say between around 16,000 in our case, and million).

Any idea about the following:

1. What's the logic in the lead ID numbering?

2. Considering this, how can we get all the leads? The algorithm described above assumes the leads IDs are sequential, but this does not seem to be the case.

Thanks! 

[SanfordWhiteman]

You definitely cannot assume the lead IDs are contiguous nor sequential!  Deleted leads always create gaps, and other events (I believe turning SFDC sync on and off is one of them) create big jumps in the sequence.

Imagine that it's like fetching rows from a SQL database without an explicit ORDER BY.  There may be long runs where there appears to be a sequence (ORDER BY <primary key> ASC), but that's just coincidental and in fact they can be returned in any order.

The fact is that there's no easy way to do this, and that blog post and accompanying SO answer are out of whack. You have to page through all the Type 12 activities.

[Anonymous]

I expected gaps (for deleted leads and so on), but like you said, the fact I see 2 separate "sequences" interlaced for some reason - that does indeed break that blog post strategy and Stack Overflow post it links to.

If lead IDs were always growing (based on creation time, newer ids are higher), even with reasonable gaps (not about a million size gap) then it could have worked. 

The getting lead Ids from all Type 12 activities approach would work, but would require doubling the amount of requests (since after getting the lead IDs you'd have to group those IDs in 300s and call get leads.).

Eran

[SanfordWhiteman]

Oh yes, this is not at all fun. It's a severe shortcoming (not to mention that the REST API has had some regressions lately that make it difficult to trust field information).

[Anonymous]

Hi Nirav

I thought about an another approach for

2. Considering this, how can we get all the leads? The algorithm described above assumes the leads IDs are sequential, but this does not seem to be the case.

My idea

- Make a static list in your Makreto

- Make and run batch SC adding exist leads into the list

- Make and activate trigger SC adding new leads into the list

- If you need all leads, Make a call http://developers.marketo.com/documentation/rest/get-multiple-leads-by-list-id/ 

[SanfordWhiteman]

Hi Takehiro!

That's an alternate approach, but the problem is that it requires API consumers (who are often outside companies) to coordinate with admins.  The API should have the "All Leads" list (List ID 0) built in. It seems clear that the REST API is deliberately designed to make offline mirroring difficult.

[Anonymous]

I agree.

Eran