I have recently been corresponding a lot with Marketo support/engineering about significant lags in database commits that are wrecking havoc with our smart campaign operations. The final response from product management and engineering is that this lag is acceptable, add a wait step. First, the recommendation for the wait step was 2 minutes. But I was finding lags of 10 minutes. Then, the recommendation for the wait step became 10 minutes.
I have been designing Marketo instances for clients for the past 9 years and have found this lag behavior to be surprising. I suspect that you will too once you understand the implications.
First, the issue. Database commits for fields and segmentations from activities such as new lead creation and form completion are lagging behind their activity triggers. What this means is that the trigger happens before the data associated with it is committed to the Marketo database. We have found lags as long as 10 minutes so far. This all happens under-the-covers which means that lags greater than that could easily go unnoticed. You would only see it in finding issues in the results of this behavior.
Let me illustrate with a couple of examples:
- A new lead is created in Marketo from Salesforce.
- We have smart campaign logic that processes the lead differently if their GDPR segmentation is one value or another value. We have a conditional flow step with conditions that have Segmentation = segment 1, do A. Segmentation = Segment 2, do B.
- We use a simple Person is Created trigger.
- When the smart campaign flow processing reaches the flow step with the segmentation choice/condition, the segmentation value hasn't been written to the database. There is NO value for the segmentation when it hits the flow step. And there may not be a value for 10 minutes (or maybe more)
- A lead fills out a form.
- We have smart campaign logic that processes the lead differently based on values returned in the form's fields. We have a conditional flow step with conditions Form Field value 1, do A. Form Field value 2, do B.
- We use a simple Fills out Form trigger.
- When the smart campaign flow processing reaches the flow step with the field choice/condition, the form field value hasn't been written to the database. If this is a new lead, the field will have NO value. If this is an existing lead, the field could have a different value (prior) to that in the form completion.
From these smart campaigns, the flow steps are all completed before the field commits to the database from the activity are done.
Why this is such a big problem:
- This isn't an easy problem to work around because it means you have to pay attention to the sequencing of related activities. This isn't simply adding a wait step to the activity's smart campaign flow and you're done. It is potentially managing the sequencing of all downstream activities as well. And some of those downstream activities might not be in Marketo but in your CRM.
- You may have to craft different solutions for new leads and existing leads.
- For existing leads you can leverage data value change/segmentation change triggers to "wait" for the field data to be committed to the database. (I think you can even leverage the Reason/Source constraint to look for the field updates from specific sources.)
- For new leads, there is no data value change activity to look for. You have to know what data to expect in each situation and wait for the field to go from empty to not empty and then process accordingly.
- You would have to identify situations where you use field and segmentation values that change based on the activity and craft appropriate solutions for each of those.
How I identified the problem:
I was investigating why there was an inconsistency in our GDPR field data as related to the segmentations that we use to manage it. This is where I found the segmentation database commit lag issue. I found that this was occurring for new leads from non-Marketo sources. It was occurring to 40% of the records coming in from those sources. I was able to reproduce it with a simple list import of 20 people into SFDC and watch as those individuals synced down to Marketo.
While investigating those data inconsistencies, I ran into the field database commit lag issue. This was with an Unsubscribe from All form where the Unsubscribe field is hidden and set to "Yes". The occurrence of the commit lag of the new Unsubscribed field value was much smaller in percentage, but still an important issue.
The On-Going Search for Solutions (feel free to add your ideas to this discussion)
First, let me explain that I hate solving timing issues with wait steps. If you have timing issues, the first thing you learn about them is that they can be incredibly inconsistent in length. The wait step time you set today might not be long enough tomorrow. You need a solution that is more resilient.
- To deal with the segmentation commit lag issue (as it seemed to be occurring for new leads), I am using a wait step "process" with a daily back up.
- I have a smart campaign that has a series of wait steps and between them evaluates whether the segmentation is no longer empty. As soon as the segmentation is no longer empty, I request campaign the new lead process I want it to flow through and remove it from the wait step process smart campaign. My combined set of waits go up to 19 minutes. This helps me keep the processing as close to the segmentation commitment as possible (as this affects downstream activities).
- I have a batch smart campaign that runs once a day and if it sees the field values inconsistent with the segmentation value, it fixes them.
- I no longer let my flow choies logic fall through to a default choice for segmentations. I have a flow step choice for each segment and have the default do nothing. In this case, rather than populating a field with the wrong value, I have an empty field which is a clearer indication that something is wrong.
- To deal with the field commit lag issue, for this specific situation, I pulled the Unsubscribe field out of the form and processed in a triggered smart campaign. I generally never place important data change values in hidden fields on forms. I would rather be able to search for it in Campaign Inspector, so I put it in flow steps.
- However, this doesn't really address the underlying issue that form completion triggers in Marketo might trigger before the form data is committed to the database. I expect to be playing around with data value changes using the Reason/Source field to see if I can at least separate the processing from the form trigger itself.
- I haven't yet figured out the impact on downstream activities or syncs to SFDC. As yet another example, we are having issues where I have a form completion which captures UTM parameters which I write to a SFDC campaign member record when the campaign member status is updated in SFDC. (It is simple workflow logic in SFDC that I have been using for years.) Because of the field update lags, the UTM parameter values (which reside in fields on the Lead/Contact record) are getting updated after the SFDC campaign member record has been updated with the status change.
- My solution is likely to be something that minimizes the timing issue (such as writing the UTM parameters when available with campaign name to a separate field (think name:value pairs list) and then parsing that field in SFDC to update the campaign member record with those values. This seems like a lot of work to get me around the timing issue of the database commit versus the campaign member update. However, I have found no way to guarantee (other than some other type of handshake) that the UTM field data is populated in SFDC before the SFDC campaign member update is triggered from the Marketo program status update.
This experience is changing the whole way that I design Marketo implementations. Marketo's reluctance to see these timing issues as a problem they need to address means that we all are going to be seeing more of this type of thing in our instances. I have only noticed this in one of my client instances so far. (I don't consider that instance to be particularly large at 1M records.) But I haven't yet started to audit the other client instances for this problem. However, just this on-going set of timing issue has eroded my confidence in the foundational assumptions that I (and perhaps many others of us) have been working under.