I have recently been corresponding a lot with Marketo support/engineering about significant lags in database commits that are wrecking havoc with our smart campaign operations. The final response from product management and engineering is that this lag is acceptable, add a wait step. First, the recommendation for the wait step was 2 minutes. But I was finding lags of 10 minutes. Then, the recommendation for the wait step became 10 minutes.
I have been designing Marketo instances for clients for the past 9 years and have found this lag behavior to be surprising. I suspect that you will too once you understand the implications.
First, the issue. Database commits for fields and segmentations from activities such as new lead creation and form completion are lagging behind their activity triggers. What this means is that the trigger happens before the data associated with it is committed to the Marketo database. We have found lags as long as 10 minutes so far. This all happens under-the-covers which means that lags greater than that could easily go unnoticed. You would only see it in finding issues in the results of this behavior.
Let me illustrate with a couple of examples:
From these smart campaigns, the flow steps are all completed before the field commits to the database from the activity are done.
Why this is such a big problem:
How I identified the problem:
I was investigating why there was an inconsistency in our GDPR field data as related to the segmentations that we use to manage it. This is where I found the segmentation database commit lag issue. I found that this was occurring for new leads from non-Marketo sources. It was occurring to 40% of the records coming in from those sources. I was able to reproduce it with a simple list import of 20 people into SFDC and watch as those individuals synced down to Marketo.
While investigating those data inconsistencies, I ran into the field database commit lag issue. This was with an Unsubscribe from All form where the Unsubscribe field is hidden and set to "Yes". The occurrence of the commit lag of the new Unsubscribed field value was much smaller in percentage, but still an important issue.
The On-Going Search for Solutions (feel free to add your ideas to this discussion)
First, let me explain that I hate solving timing issues with wait steps. If you have timing issues, the first thing you learn about them is that they can be incredibly inconsistent in length. The wait step time you set today might not be long enough tomorrow. You need a solution that is more resilient.
This experience is changing the whole way that I design Marketo implementations. Marketo's reluctance to see these timing issues as a problem they need to address means that we all are going to be seeing more of this type of thing in our instances. I have only noticed this in one of my client instances so far. (I don't consider that instance to be particularly large at 1M records.) But I haven't yet started to audit the other client instances for this problem. However, just this on-going set of timing issue has eroded my confidence in the foundational assumptions that I (and perhaps many others of us) have been working under.
This isn't my full response, don't worry! But I wanted to quickly point out that this modification...
... I pulled the Unsubscribe field out of the form and processed in a triggered smart campaign...
... changes the characteristics of the data value change. When the system Unsubscribed field is sent in the form payload, that allows attribution to the source asset (i.e. Unsubscribed from Email). If it's merely set in a Smart Campaign, that important part doesn't happen.
I didn't know that. have never set up the "unsubscribe from all" this way, to have discovered that. This was implemented by someone else on the team. I have consistently seen Unsubscribe from Email activities when the Unsubscribe form was completed for other clients. But I think that was because we made the Unsubscribe field visible (un-hidden) on the form. That would give us the correspondence that you mentioned.
Well, there are a number of options to work around this if I put it back as hidden. Maybe I'll find out how well the Data Value Change with Reason/Source conditions works instead of the form completion trigger.
Will be waiting for the rest of your response. I feel like I am a solo voyager in space with these issues.
Pam, thank you so much for posting this, I have been dealing with a database lag issue for about 6 months - seems to be twofold; 1) slow to right form submit field values to database up to 10 minutes lags impacting my smart campaign triggers/flows like you mention above and 2) slow to update Marketo UI. Both issues are intermittent where I see a lag of 10 to 20 minutes.
When I create a support ticket or #1 they tell me to add a wait step and this is expected behavior. Regarding #2, here's an explanation from support, which I totally appreciate the time spent to explain however I don't feel it is acceptable to have intermittent lags that take up to 20 minutes or to have to add 10 minute wait steps for form data to write to database. I've been using Marketo for over 10 years and never had this problem until recently.
I think there must be a reason for this to happen recently and requires a fix. We can't be the only folks experiencing this.
There are two main processors we use in Marketo. One of them is faster to write to but is slower to search from, this is used for things like the Lead detail pages, and the other is slower to write to but is much faster to search from, this is used for things like our Smart Lists.
Whenever activities are processed in Marketo, they are written in both places. First, they are written to the one that writes faster and then to the one that writes slower. In this way, they can only be written to one at a time. When the activity makes it to the person's activity log, the Flow Step shows as processed, i.e. when the box shows as Completed. Yet, the activity may still be writing to the second database which is why the activity is not always showing when we load the Campaign Results. However, once this data is indexed and written here, it is much faster to search for which is one way we optimize things like Smart Lists.
For context, this is in no way meant to be a comprehensive breakdown on how the different processors work, it is only meant to serve as a high level overview, to help shed light on the perceived lag. Beyond this, we would not be able to discuss the infrastructure further as it is proprietary. But it is in this way, that we would see this behavior as expected. There is not a benchmark or expectation for how long it takes for the activity to write to this database.
Thank you for providing Marketo's discussion about this issue. I didn't get that level of detail from all of my discussions (of which there were many) with the Marketo technical support engineers.
About the time that you posted this, another of my clients started having issues with the form field updates not committing to the database before the Fills Out Form triggered the smart campaign to process the form completion. So, this problem is definitely becoming more prevalent and more people are going to have issues around this.
There were two problems that the lag in field commit triggered in this situation.
1) I was using the field token on the confirmation page.
2) It was generating emails with calendar files that were picking up the field token default values.
Neither of these was in any way ideal. And in certain cases, when the field was updated multiple times, it actually generated the wrong values.
The good news is that I had time this morning to try a partial solution that I think is better than a wait step. I created new logic that triggers off of Data Value Changes. In the Data Value Changes, I added the Reason constraint and pulled the value from the Activity Log. This seemed to work nicely as a substitute. For existing persons in Marketo, it will wait until the field update is committed, then trigger the smart campaign. This is much better.
However, I this solution will not for new persons from form completions. The Data Value Changes activities are not triggered for brand new persons from form fills. And if you put the Person is Created trigger with a field value filter, I have seen the smart campaign not trigger because the field in the filter condition is not set. I've seen that issue before even though it wasn't occurring in my testing today. No lags were occurring during my testing.
I hate having two solutions around to address this issue. So, I will probably continue to solve with my wait processing smart campaign as a work-around solution.
As this issue comes more to light across Marketo's clients, and they see the ramifications and how difficult it is to design around this behavior, we can hope that Marketo will try to address the issue for us.
1) I was using the field token on the confirmation page.
But this is never guaranteed to work, and it doesn't have to directly with field commit latency. The Munchkin session is never guaranteed to be associated by the next pageview, it is a background process.