AnsweredAssumed Answered

getMultipleLeads - streamPosition ID not respected?

Question asked by 37838 on Oct 8, 2013
Latest reply on Apr 25, 2014 by 56617
I'm using the getMultipleLeads API call to retrieve mutliple Leads.  There is a local limit to the number of leads I can process in each run, so I need to be able to page by lead record in some way.  I tried paging using the IDs by inputting the id into the streamPosition as such:
"id:foo:ts:0:os:0:rc:0"
where the foo is a number corresponding to the Lead's ID such as 500.  
The result is that the os and ts are respected (so pulling records from offset 0 / time 0 onward as the functionality would seem to indicate) and the ID position is not respected.  Is there some way to make the request such that the ID passed in the streamPosition is respected as a "pull records with ID after X".

I looked into using the timestamp (ts) and the offset/index (os) fields individually and in combination.  Unfortunately this results in exrtremely complex coding to walk backward through records to account for situations where records are deleted (os shifts behind the scenes, so when I make the call using the streamPosition with the ts and os from the previous request the results are missing the leading edge records).  This is the solution I'm currently working through due to the ID issue noted above, and it requires I keep a handle on all of the previously seen records to check as I walk backwards until I find the correct record to restart from.  The gap in fetches could be up to a day long, so I cannot rely on the interval being so short as to make the deletion of a record inbetween syncs unlikely.

I also see that there is the option to use the leadSelector and pass in a leadSelectorRef key enum of IDNUM, however this takes in a list of IDs, which I would not already have.  It does not appear to provide the ability to do any sort of greater-than operation, so this does not appear to be an option for my case.

Since I know the question will come up ;)
request trying to use ID for streamPosition:
<env:Body>
  <mkt:paramsGetMultipleLeads>
    <streamPosition>id:500:ts:0:os:0:rc:0</streamPosition>
    <batchSize>100</batchSize>
  </mkt:paramsGetMultipleLeads>
</env:Body>


Situation mentioned in the second part:
request 1:
<env:Body>
  <mkt:paramsGetMultipleLeads>
    <streamPosition>id:0:ts:1381251748:os:0:rc:0</streamPosition>
    <batchSize>100</batchSize>
  </mkt:paramsGetMultipleLeads>
</env:Body>
newStreamPosition returned is "id:1214:ts:1381251748:os:100:rc:50"

Record with ID 1211 is deleted for example then the second request is made:

<env:Body>
  <mkt:paramsGetMultipleLeads>
    <streamPosition>id:1214:ts:1381251748:os:100:rc:50</streamPosition>
    <batchSize>100</batchSize>
  </mkt:paramsGetMultipleLeads>
</env:Body>

record 1215 will have moved one position earlier in the total list of lead records.  As a result it will be at os:99 and will not be returned when this second call is made.  In our situation the user is able to configure runs with periods of up to 1 day.  In the event during that day a record is deleted, for example it is a duplicate, then the shift would happen and we would miss pulling out the leading record in the next batch as it would have shifted to the previous batch.

Any suggestions?

Outcomes