Marketo Database Commit Lags

[Pam_Hudadoff2]

I have recently been corresponding a lot with Marketo support/engineering about significant lags in database commits that are wrecking havoc with our smart campaign operations.   The final response from product management and engineering is that this lag is acceptable, add a wait step.  First, the recommendation for the wait step was 2 minutes.  But I was finding lags of 10 minutes.   Then, the recommendation for the wait step became 10 minutes.  

I have been designing Marketo instances for clients for the past 9 years and have found this lag behavior to be surprising.   I suspect that you will too once you understand the implications.  

First, the issue.  Database commits for fields and segmentations from activities such as new lead creation and form completion are lagging behind their activity triggers.   What this means is that the trigger happens before the data associated with it is committed to the Marketo database.  We have found lags as long as 10 minutes so far.   This all happens under-the-covers which means that lags greater than that could easily go unnoticed.  You would only see it in finding issues in the results of this behavior.

Let me illustrate with a couple of examples:

1. A new lead is created in Marketo from Salesforce.   
   • We have smart campaign logic that processes the lead differently if their GDPR segmentation is one value or another value.  We have a conditional flow step with conditions that have Segmentation = segment 1, do A.  Segmentation = Segment 2, do B.   
   • We use a simple Person is Created trigger. 
   • When the smart campaign flow processing reaches the flow step with the segmentation choice/condition, the segmentation value hasn't been written to the database.   There is NO value for the segmentation when it hits the flow step.   And there may not be a value for 10 minutes (or maybe more)
2. A lead fills out a form. 
   • We have smart campaign logic that processes the lead differently based on values returned in the form's fields.  We have a conditional flow step with conditions Form Field value 1, do A.   Form Field value 2, do B.
   • We use a simple Fills out Form trigger.   
   • When the smart campaign flow processing reaches the flow step with the field choice/condition, the form field value hasn't been written to the database.   If this is a new lead, the field will have NO value.   If this is an existing lead, the field could have a different value (prior) to that in the form completion.

From these smart campaigns, the flow steps are all completed before the field commits to the database from the activity are done.  

Why this is such a big problem:

1. This isn't an easy problem to work around because it means you have to pay attention to the sequencing of related activities.  This isn't simply adding a wait step to the activity's smart campaign flow and you're done.  It is potentially managing the sequencing of all downstream activities as well.  And some of those downstream activities might not be in Marketo but in your CRM.
2. You may have to craft different solutions for new leads and existing leads.   
   • For existing leads you can leverage data value change/segmentation change triggers to "wait" for the field data to be committed to the database.   (I think you can even leverage the Reason/Source constraint to look for the field updates from specific sources.)
   • For new leads, there is no data value change activity to look for.   You have to know what data to expect in each situation and wait for the field to go from empty to not empty and then process accordingly.
3. You would have to identify situations where you use field and segmentation values that change based on the activity and craft appropriate solutions for each of those. 

How I identified the problem:

I was investigating why there was an inconsistency in our GDPR field data as related to the segmentations that we use to manage it.  This is where I found the segmentation database commit lag issue. I found that this was occurring for new leads from non-Marketo sources.  It was occurring to 40% of the records coming in from those sources.  I was able to reproduce it with a simple list import of 20 people into SFDC and watch as those individuals synced down to Marketo.

While investigating those data inconsistencies, I ran into the field database commit lag issue.  This was with an Unsubscribe from All form where the Unsubscribe field is hidden and set to "Yes".  The occurrence of the commit lag of the new Unsubscribed field value was much smaller in percentage, but still an important issue.

The On-Going Search for Solutions (feel free to add your ideas to this discussion)

First, let me explain that I hate solving timing issues with wait steps.  If you have timing issues, the first thing you learn about them is that they can be incredibly inconsistent in length.  The wait step time you set today might not be long enough tomorrow.  You need a solution that is more resilient.  

1. To deal with the segmentation commit lag issue (as it seemed to be occurring for new leads), I am using a wait step "process" with a daily back up.   
   
   • I have a smart campaign that has a series of wait steps and between them evaluates whether the segmentation is no longer empty.  As soon as the segmentation is no longer empty, I request campaign the new lead process I want it to flow through and remove it from the wait step process smart campaign.  My combined set of waits go up to 19 minutes.  This helps me keep the processing as close to the segmentation commitment as possible (as this affects downstream activities).
   • I have a batch smart campaign that runs once a day and if it sees the field values inconsistent with the segmentation value, it fixes them.   
   • I no longer let my flow choies logic fall through to a default choice for segmentations.   I have a flow step choice for each segment and have the default do nothing.   In this case, rather than populating a field with the wrong value, I have an empty field which is a clearer indication that something is wrong.
2. To deal with the field commit lag issue, for this specific situation, I pulled the Unsubscribe field out of the form and processed in a triggered smart campaign.  I generally never place important data change values in hidden fields on forms. I would rather be able to search for it in Campaign Inspector, so I put it in flow steps.   
   • However, this doesn't really address the underlying issue that form completion triggers in Marketo might trigger before the form data is committed to the database.  I expect to be playing around with data value changes using the Reason/Source field to see if I can at least separate the processing from the form trigger itself.
3. I haven't yet figured out the impact on downstream activities or syncs to SFDC.  As yet another example, we are having issues where I have a form completion which captures UTM parameters which I write to a SFDC campaign member record when the campaign member status is updated in SFDC.   (It is simple workflow logic in SFDC that I have been using for years.)  Because of the field update lags, the UTM parameter values (which reside in fields on the Lead/Contact record) are getting updated after the SFDC campaign member record has been updated with the status change.
   • My solution is likely to be something that minimizes the timing issue (such as writing the UTM parameters when available with campaign name to a separate field (think name:value pairs list) and then parsing that field in SFDC to update the campaign member record with those values.  This seems like a lot of work to get me around the timing issue of the database commit versus the campaign member update.  However, I have found no way to guarantee (other than some other type of handshake) that the UTM field data is populated in SFDC before the SFDC campaign member update is triggered from the Marketo program status update.

This experience is changing the whole way that I design Marketo implementations.  Marketo's reluctance to see these timing issues as a problem they need to address means that we all are going to be seeing more of this type of thing in our instances.   I have only noticed this in one of my client instances so far.   (I don't consider that instance to be particularly large at 1M records.)  But I haven't yet started to audit the other client instances for this problem.  However, just this on-going set of timing issue has eroded my confidence in the foundational assumptions that I (and perhaps many others of us) have been working under.

Pam Hudadoff

[SanfordWhiteman]

This isn't my full response, don't worry! But I wanted to quickly point out that this modification...

... I pulled the Unsubscribe field out of the form and processed in a triggered smart campaign...

... changes the characteristics of the data value change. When the system Unsubscribed field is sent in the form payload, that allows attribution to the source asset (i.e. Unsubscribed from Email). If it's merely set in a Smart Campaign, that important part doesn't happen.

[Pam_Hudadoff2]

Sanford,

    I didn't know that.   have never set up the "unsubscribe from all" this way, to have discovered that.  This was implemented by someone else on the team.  I have consistently seen Unsubscribe from Email activities when the Unsubscribe form was completed for other clients.   But I think that was because we made the Unsubscribe field visible  (un-hidden) on the form.   That would give us the correspondence that you mentioned.  

    Well, there are a number of options to work around this if I put it back as hidden.  Maybe I'll find out how well the Data Value Change with Reason/Source conditions works instead of the form completion trigger.

     Will be waiting for the rest of your response.  I feel like I am a solo voyager in space with these issues.

-Pam

[lorie_witmer4]

Pam, thank you so much for posting this, I have been dealing with a database lag issue for about 6 months - seems to be twofold; 1) slow to right form submit field values to database up to 10 minutes lags impacting my smart campaign triggers/flows like you mention above and 2) slow to update Marketo UI.  Both issues are intermittent where I see a lag of 10 to 20 minutes.

When I create a support ticket or #1 they tell me to add a wait step and this is expected behavior.  Regarding #2, here's an explanation from support, which I totally appreciate the time spent to explain however I don't feel it is acceptable to have intermittent lags that take up to 20 minutes or to have to add 10 minute wait steps for form data to write to database.   I've been using Marketo for over 10 years and never had this problem until recently.

I think there must be a reason for this to happen recently and requires a fix.  We can't be the only folks experiencing this.  

Explanation:

There are two main processors we use in Marketo. One of them is faster to write to but is slower to search from, this is used for things like the Lead detail pages, and the other is slower to write to but is much faster to search from, this is used for things like our Smart Lists.

Whenever activities are processed in Marketo, they are written in both places. First, they are written to the one that writes faster and then to the one that writes slower. In this way, they can only be written to one at a time. When the activity makes it to the person's activity log, the Flow Step shows as processed, i.e. when the box shows as Completed. Yet, the activity may still be writing to the second database which is why the activity is not always showing when we load the Campaign Results. However, once this data is indexed and written here, it is much faster to search for which is one way we optimize things like Smart Lists.

For context, this is in no way meant to be a comprehensive breakdown on how the different processors work, it is only meant to serve as a high level overview, to help shed light on the perceived lag. Beyond this, we would not be able to discuss the infrastructure further as it is proprietary. But it is in this way, that we would see this behavior as expected. There is not a benchmark or expectation for how long it takes for the activity to write to this database.

[Pam_Hudadoff2]

Lorie,

    Thank you for providing Marketo's discussion about this issue.   I didn't get that level of detail from all of my discussions (of which there were many) with the Marketo technical support engineers.

    About the time that you posted this, another of my clients started having issues with the form field updates not committing to the database before the Fills Out Form triggered the smart campaign to process the form completion.  So, this problem is definitely becoming more prevalent and more people are going to have issues around this.

   There were two problems that the lag in field commit triggered in this situation. 

     1) I was using the field token on the confirmation page. 

     2) It was generating emails with calendar files that were picking up the field token default values. 

  Neither of these was in any way ideal.  And in certain cases, when the field was updated multiple times, it actually generated the wrong values.

    The good news is that I had time this morning to try a partial solution that I think is better than a wait step.  I created new logic that triggers off of Data Value Changes.  In the Data Value Changes, I added the Reason constraint and pulled the value from the Activity Log.  This seemed to work nicely as a substitute.  For existing persons in Marketo, it will wait until the field update is committed, then trigger the smart campaign.   This is much better.   

    However, I this solution will not for new persons from form completions.   The Data Value Changes activities are not triggered for brand new persons from form fills.  And if you put the Person is Created trigger with a field value filter, I have seen the smart campaign not trigger because the field in the filter condition is not set.  I've seen that issue before even though it wasn't occurring in my testing today.   No lags were occurring during my testing. 

     I hate having two solutions around to address this issue.   So, I will probably continue to solve with my wait processing smart campaign as a work-around solution.   

     As this issue comes more to light across Marketo's clients, and they see the ramifications and how difficult it is to design around this behavior, we can hope that Marketo will try to address the issue for us.

Pam

[SanfordWhiteman]

     1) I was using the field token on the confirmation page. 

But this is never guaranteed to work, and it doesn't have to directly with field commit latency. The Munchkin session is never guaranteed to be associated by the next pageview, it is a background process.