Why do no records appear when I change pagingtoken date to a later date?

[Anonymous]

Not sure if it's something I'm doing wrong (usually it is).

The call I'm making in PHP is this:

$ch = curl_init("https://273-EQL-130.mktorest.com/rest/v1/activities.json?activityTypeIds=1&leadIds=1166102&nextPageT..." . $pagingToken . "&access_token=" . $accessToken);

Let's take it as read that the access token is working.

I'm getting the paging token like this:

$ch = curl_init("https://273-EQL-130.mktorest.com/rest/v1/activities/pagingtoken.json?sinceDatetime=2016-11-01&access..." . $accessToken);

When I use November 1st as the sinceDateTime value, it works great: I get back 300 records (the max).

When I use October 1st as the sinceDateTime value, I get a similar result (300 records).

$ch = curl_init("https://273-EQL-130.mktorest.com/rest/v1/activities/pagingtoken.json?sinceDatetime=2016-10-01&access..." . $accessToken);

But when I use September 1st as the sinceDateTime value, I get no records returned, but "moreResult" is true:

This page says

The first is a paging token which represents a date.  These are used to retrieve activities, data value changes, and deleted leads which occurred after the date represented by the paging token.

Should it not be the case that if I go to an earlier date, everything after that date would be subsumed under the description "which occurred after the date represented by the paging token"? And if that's true, should I not be getting results in my call with the September paging token, just like I do for October and November? Our Marketo installation was active as of July, so it's not that I'm going back before we had records...

Any help is much appreciated - thanks in advance!

[SanfordWhiteman]

Sometimes, moreResult returns true even with no data in the current payload.  It's the way the cursor is implemented (maybe some kind of global checkpointing).

[Anonymous]

Thanks for the quick response! My question would then be: Why would there be no payload if the calls for later dates DO have a payload and all else is equal?

Edit: I should add the call for the September date just in case there is something wrong with the syntax:

$ch = curl_init("https://273-EQL-130.mktorest.com/rest/v1/activities/pagingtoken.json?sinceDatetime=2016-09-01&useBlo..." . $accessToken);

[SanfordWhiteman · Accepted solution]

Imagine -- I'm not saying this is the exact implementation, as it's a black box fo us, but food for thought  -- the pages are based on all activity types in the log, regardless of the applied filter. 

[Anonymous]

Wow - so there is a possibility that we'd get pages and pages with no results! That adds complexity if you're only looking for one result because instead of just making one API call you have to loop through the nextPageTokens until you find something. Arg - I hope that's not the implementation!!! Makes filters a lot less useful! But thank you for your response - I will make this assumption and proceed. I'll mark your answer as correct if this is indeed what I find.

[SanfordWhiteman]

OK, keep me updated. I think the max number of consecutive empty payloads is mitigated by other factors so it's not just waiting to find a match. But I sure wish empty ones didn't count against our API call limit, since there's nothing we're doing wrong in this case.

[Steven_Vanderb3]

Hi Matt,

This is how the call works, yes.  The call only uses a finite amount of resources to look for qualifying activities (I think a certain number of seconds the call is allowed to run but not sure).  If you get a call with no results but moreResults = true, it means that it did not find any results during the time in searched but there are still more results to search through.

There could be hundreds of millions of activities in an instance (I've seen a case where number of total activities in an instance lifespan was over a trillion) so if there weren't some kind of constraint to how many activities a call could search through in one go, it could run a very long time.  There is a note about receiving less than 300 activities in one go in the docs here http://developers.marketo.com/rest-api/lead-database/activities/

[Anonymous]

Thank you for your response!

I wrote a loop into my script to continue searching with the updated nextPageToken if "moreResult" was true. The loop is working, and eventually "moreResult" equals false, suggesting it has reached the end of the data search. I am still stumped though, and maybe the devil is in the details of the particular call I'm trying to make. Here is a screenshot of my PHP log. The single integers are the loop iteration number and the Objects are the decoded JSON responses. As you can see, the nextPageToken is different each time, and the last response does not have moreResult so the loop terminates.

Here is the call I'm attempting to make:

https://273-EQL-130.mktorest.com/rest/v1/activities.json?activityTypeIds=1&leadIds=1166102&assetIds=...

I'm attempting to see if a lead (leadIds=1166102) has filled out a form (activityTypeIds=1) with a form ID of 1238. The lead happens to be myself and I know I filled out that form, and can verify this by looking at my lead activity record, yet I get no found results. Can you see if I'm doing anything wrong? Thanks so much in advance.