Delete Lead Activity via Bulk or REST API - how to get all of them?

[Kurt_Koller]

We are trying to keep Leads in our external database in sync with leads in the Marketo database. Nonwithstanding the inablility just get all leads in once call, instead having to go through in monthly gets using a filter like created or updated, once we have all the leads in our own database we figure we could get updates and keep them in sync.

Leads are removed in a variety of ways.

Activities have two main activities that we see (we are testing on instance with a single partition) when they are removed:

37, Delete Lead

32, Merge Leads

We have been using Bulk Export API to get all activities for Leads. In this data we get activity data for both of these activities. The question is, are all leads which are Deleted included in these activities? I know you can only get them for a limited about of time, but will every deleted lead also return a Delete Lead activity or a Merge Leads activity?

I ask because we are trying to keep leads in sync, and using these types and we are off by a large percentage in some instances, where the leads just have a smattering of activities and then are gone, with no 37 or 32 or anything else indicating that they've been deleted, but indeed they are deleted, so we never see that the lead is removed and our database is completely wrong very quickly..

I'd love to know all of the possible types of activities that could signal a Lead has been removed, if there are more than these two. If not, are there cases where the 37/32 wouldn't be pulled down via Bulk API? Or should every lead that has been deleted/merged show in this data?

Alternatively we would have to check the lead API endpoint in batches etc on a constantly rotating basis to figure out if leads are still existing, which is both wasteful and uses many API calls and takes days.

Thanks,

Kurt

[SanfordWhiteman]

Have you worked with the Get Deleted Leads (/activities/deletedleads.json) endpoint, Kurt?

[Kurt_Koller]

Yes we have. That endpoint doesn't give merges, just delete lead (37) activities which we also get through bulk already. There are many leads which are no longer in the system which never had a delete lead (37) or merge lead (32) activity, so I'm looking for other ways a lead could be deleted from marketo and not throw a delete lead or merge lead activity. We need to solve this, after time goes by we aren't mirroring data any more.

[SanfordWhiteman]

Yeah, I recall -- must be a couple of years ago at this point -- we'd started tombstoning leads instead of deleting them, and sending webhooks to our archive to set the tombstone timestamp. Idea being that anything that wasn't a "managed deletion" (excluding merges) was known to sometimes drop out of sight without a log. Let me see if I have any of the cases still around where we were seeing this. (Was also around this time that we realized datatypes from Describe weren't suitable for sizing SQL columns without losing data.)

[SanfordWhiteman]

tombstoning leads instead of deleting them

(meaning instead of deleting them immediately, instead aging them out based on tombstone timestamp)

[Kurt_Koller]

Sanford,

Thanks for this. We are definitely seeing things vanish, I assumed there would always be a delete lead or merge lead, I think that was a bad assumption. I'm going to verify today or Monday via the deleted lead endpoint, and then I guess I need to build a rotating/drip sync-check to determine if leads are still in the system. Let me know if you have any other info