Lead API Activities Syncing Approach

[Anonymous]

I've come up against a problem when writing a cron script to pull down all activities. I need to keep a copy of all the activities for leads in our CRM as we plan to do quite a bit with like processing unsubscribes or showing an activity log on the lead page of our CRM. There are more use cases and some that haven't been fully defined, so keeping a copy of these activities makes the most sense to avoid being bottlenecked by (and wasting) REST calls.

Currently my script pulls down the full list of activity types to make sure nothing changed there. If a new one appears, it adds it to our cache. So that's one call.

After that I loop over the types and pull new activities for each type individually. In our sandbox, we're not doing too much just yet aside from testing webhooks and contact syncing, so the new activities are generally limited to about 3 different activities. There's currently a total of 47 activities so that's 47 calls to the API, 3 of which matter so it's terribly inefficient. I don't know the importance the activity types.

I may be misunderstanding this but I think it's possible that there would be either a gap in data or repeated data if I batch the type calls together using Paging Tokens because they're pulling records after the datetime used to generate the token. For example, if I passed 10 IDs per call to grab with one of those types being updated constantly and another being updated infrequently, I would need to specify the oldest datetime for the set of IDs to make sure I don't miss anything. In that instance, the most frequently occurring activity types would have repeating acitivities that we already cached.

Now I can deal with these problems by removing some of the acitivities from the call that we don't think we'll need and also trying to identify fast moving types and batch those together, for instance.

My questions:

• Am I fundamentally misunderstanding the way the paging tokens work? Results come back with a nextPageToken but I'm unsure if that would handle the above situation.
• Can the activity types increase dynamically? I'm a little confused by the documentation - I know we can add our own, but I'm wondering if we can remove the initial call to retrieve the current types entirely.
• Are there any good examples of how to approach this type of situation already out there?

Thanks for any help or suggestions!

[Kurt_Koller · Accepted solution]

You might want to look at the Bulk Extract API for activities. It has its own issues but it's more straightforward with the Bulk Extract to just slurp all activities in a date range, plus you get activities like "delete lead" which you don't get in the normal activity call.