REST Folder API Duplicates while Paging

[xceled]

Hello,

While paging through /rest/asset/v1/folders.json using the offset parameter, we are seeing duplicate folder ids in later pages that were returned in earlier pages. 

E.g. With a maxReturn=200 and offset=0, we get folder id 130 in the results. Then with the same maxReturn=200 and offset=200, we again get folder 130 in the results. This is breaking our existing integration we built and not sure why this API is now providing records shown in earlier pages in later pages. Any idea what might be behind this? Has Marketo broken paging in their API to now not page properly?

[SanfordWhiteman]

Seeing the same folder ID on a different page is always something you should build for, for the simple reason that the entire pageset is not fetched in advance. So if folder 123 and folder 456 appear on page 1 and then someone deletes both of those folders, by definition everything is going to be shifted back 2 positions, so 2 other folders will appear on 2 different pages.

The fact that you fetch page 1, page 2, and page 3 over just a couple of seconds apart makes this less likely but by no means impossible, and your integration must take this case into account.

Now as for a deeper, recent regression in the API, that’s possible. I’d have to run a bunch more tests to try to reproduce it. But since your integration already needs to be ready for this possibility, that would be the place to fix it.

[xceled]

That is fair to a degree (mainly for delete/adds during a paging operation) however paging requires consistent sorting or else there is no guarantee that I will retrieve all items from all pages if the order changes every time I call the API. While I agree that handling duplicates is something I can handle on my side (for the deletes/adds scenario affecting page content) that is not what is happening here and makes me doubt the consistency of the sorting of the results being returned by the API and if so, then do I really have any guarantee that I'm getting all records if records can just change pages randomly and I don't know if a record will appear ever in the paging process due to changing position?

[SanfordWhiteman]

Did you check the Audit Trail to see if indeed there were adds/deletes during processing? Very unlikely that any random sorting has been introduced.

[xceled]

I don't need to as its repeatable 100% of the time. Adds/deletes wouldn't explain that.

[SanfordWhiteman]

Humor me. Examine the Audit Trail for the entire period of a job run (a run where you observe the behavior obviously).

[xceled]

Its not a job run, I'm literally just calling the API with return = 200 and offset 0 & then a second call with same return and offset 200 and in both is an instance of the exact same folder JSON object with a specific id/type. Audit trail shows 0 change during my API call tests demonstrating this.

[Jo_Pitts1 · Accepted solution]

@xceled ,

in addition to what @SanfordWhiteman has already said, don't forget to check the folderType value.

It is possible for a folder of type 'Program' to have the same id as a folder of type 'Folder'.

Cheers

Jo

[xceled]

Thanks Jo, yes I confirm we have the identity properly defined to include id and folder type, it is not that causing the new issue we haven't seen before.