API questions: Programs and Lead Activity

[Dan_Stevens_]

We're building out a data warehouse of lead activity that will be used for our dashboards/reports in Power BI.

QUESTION 1 (PROGRAMS): During our call today, we learned that the developers are having to make daily calls to pull all PROGRAMS into our DB.  I asked them why then aren't just grabbing new programs and appending them to a master Program table.  Then state that it's not possible.  While the GET programs call does return a "CreatedAt" date:

There's no way to make subsequent calls to only return those programs created since the last call (since there's no filter type to do so):

That makes no sense.  Are our developers looking at this the wrong way - and instead should be using another approach to get just new programs (rather than the entire list each day)?

QUESTION 2 (LEAD ACTIVITY): Ideally, we need to capture specific activity based on PROGRAM STATUS (and also attribute it to the appropriate PROGRAM).  For example, one of our metrics is to measure the number of RESPONSES/ENGAGEMENT (based on PROGRAM STATUS, eg., "fills out form", "engaged - downloaded", etc.) for each reporting period (daily, monthly, quarterly).  So we need to filter just that activity that contains any of the defined PROGRAM STATUSES.  What is the appropriate call(s) in order to grab something that ultimately looks like this (especially when ChannelID and/or ProgramID aren't included attributes)?:

ActivityID, LeadID, ProgramID, ProgramStatusID

The "GET Lead Activities" doesn't seem to contain all of these attributes - specifically the key attribute for us, PROGRAM STATUS.

Hopefully Sanford Whiteman​ (in addition to other API experts in the community) will provide some insight here.

[SanfordWhiteman]

Hey Dan,

IIRC, the Get Lead Activities stream contains the event Change Status in Progression, which is the old-school term for a change in Program Status. This includes the Program ID, so while it doesn't let you backfill programs that don't have any members, it will expose the ID of a program as soon as someone is added to it, at which the Asset API can be used to get the Program metadata.

[Dan_Stevens_]

Thanks for chiming in, Sandy.  The GET LEAD ACTIVITIES stream only returns the following:

Unless I'm looking at this wrong, I don't see anywhere information around Program/Progression Status.

Also, with regards to my first question on building an active PROGRAMS table in our data warehouse - and considering we have over 2,500 programs today - is it true that it's required to download this massive number each day (wasting precious API calls) vs. simply updating the DB with just those programs that have changed (or that have been added/deleted)?

[SanfordWhiteman]

Unless I'm looking at this wrong, I don't see anywhere information around Program/Progression Status.

One of the activity types (activityTypeId 104 in my primary instance, though this is not guaranteed to be the same in all instances) is Change Status in Progression, with primary attribute Program ID and auxiliary attributes:

       [
        {
          "name": "Acquired By",
          "dataType": "boolean"
        },
        {
          "name": "New Status ID",
          "dataType": "integer"
        },
        {
          "name": "Old Status ID",
          "dataType": "integer"
        },
        {
          "name": "Reason",
          "dataType": "string"
        },
        {
          "name": "Success",
          "dataType": "boolean"
        },
        {
          "name": "New Status",
          "dataType": "string"
        },
        {
          "name": "Old Status",
          "dataType": "string"
        }
      ]

These activities are a substitute for notification that a new Program ID exists. So you don't have to download all the Program IDs every day to find new Programs, as long as you limit yourself to Programs that have had at least one member.

Detecting changes to the Program-level config, though, or if a Program has been deleted, I think requires download-and-compare via the Asset API like you're doing now.  You could decide to redownload the program when triggered by a Change Status activity (that is, the first time you see a Change Status for a Program on a given day, update the corresponding Program metadata, regardless of whether you have existing metadata).  Then maybe check for deletes once a week.

[Dan_Stevens_]

Thanks again, Sandy, but I'm afraid I'm still not following.  I'm going to loop in our developer, Grant Stephens​, to see if he can make better sense of this.

Grégoire Michel, have you any experience with what we're trying to do here?

[Grégoire_Miche2]

Hi Dan,

Sorry for answering late, I was off with the flu.

May be would it be worth starting with the activity type endpoint http://developers.marketo.com/rest-api/endpoint-reference/lead-database-endpoint-reference/#!/Activi...  that will provide you with the complete list of codes and labels for activities in your instance.

-Greg

[Anonymous]

So the elements map as below for activity 104

primaryAttributeValueId: = programID

primaryAttributeValue = programName

[SanfordWhiteman]

Right.

ETA: ... though this isn't really apparent from the endpoint documentation.  In any case you should only use the ID as the name may change.

[Dan_Stevens_]

It seems that we're still not able to pull in the appropriate data properly.  Here's a an example of what we're aiming for (this is from RCE):

Can this or can this not be accomplished use a combination of API calls?

[Dan_Stevens_]

Just to further clarify, when pulling lead activities, the desire is to include the "programName", "programID", and "Program Status".  I think it's that last attribute that's still unclear.  Where is "Program Status" found within the Lead Activity?  I'd have to think this would be key data as part of the data returned with Lead Activity, but it's not apparent by looking at the documentation.