Leads failing to qualify for triggers

[Liliana_Cheng1]

Hey community, our instance has been experiencing major delays (two to four hours) in our triggers. Some leads who fill out a form do not qualify for the triggers they're supposed to, sometimes for hours, sometimes for over a day. This issue is not isolated to one particular form or trigger, it's across our instance and it's inconsistently happening because some of my test leads have qualified for the trigger without an issue.

I've spoken to two Marketo support reps who confirmed that our triggers are set up correctly to capture, and the affected leads SHOULD be qualifying for the triggers. They also confirmed that we do not have a heavy volume of leads in the trigger queue. I am still waiting for a manager to update me but this is troubling.

What confuses me is when leads just sit in Marketo for two+ hours without any further activity will suddenly qualify for their trigger. No further actions were taken from the lead and I didn't manually request the campaign but they qualified - why is this happening? So most of the leads who failed to trigger immediately yesterday are okay after four hours... Is anyone else experiencing this massive delay in triggered behaviors?

If it makes a difference, I noticed this yesterday 3/6/2017 around 1:50pm Eastern and leads kept failing to qualify until 3pm. Another round of trigger failures happened between 7pm and 9pm. Most of them were caught up this morning but the most recent round of failures started around 11:30am today.

Thanks.

[Liliana_Cheng1 · Accepted solution]

5/1/17 UPDATE - this issue looks like it's been resolved (for now). We had a call with our account exec and a support escalation manager this past Friday, and they said the engineers planned to push a patch over the weekend. Since then, I've been monitoring and testing and our triggers seem to be working as they're built to be.

They haven't given us an in-depth analysis of what happened, but they said it was related to "the high speed database used to capture activity and how it was tied to trigger processing". The patch that was pushed this weekend was to remove the dependency. I'm not sure what their plan is after this (whether or not they'll tie these two problematic things together again).

If you are still seeing issues related to delayed triggers, you should contact support with some examples so they're aware this is still an issue for you.

[Dan_Stevens_]

Here's an official statement by Marketo Support: Marketo Service Interruption Update

[Josh_Hill13]

Support says it was fixed at 1028am. Sounds like there's still work to do though.

Open a ticket if you aren't sure the leads were processed.

[Dan_Stevens_]

We never got any official communication around this. In fact, the link I shared here didn't even mention the specific pods that were affected (or if it was system-wide).

[Bruce_Jeffers]

Maybe they were secretly hoping we wouldn't have noticed last week, being at Summit and all....

[Liliana_Cheng1]

Thanks for the link Dan Stevens​, I didn't see that post and I don't understand why Marketo support didn't specifically reach out to those of us who have active tickets open for this issue. Especially since I've been contacting support once a week since March 6th and twice per day since last week about this issue.

Yesterday, we did not have leads get stuck in the "Fills out form" trigger which is an improvement, however, we did have a significant amount of leads who were stuck in various "Request campaign" triggers. This is still true today. We currently have 50+ leads stuck in a request campaign trigger.

[Molly_Cousins]

I'm experiencing this issue as well - and on the Filled Out Form trigger, to boot! Nothing amiss in Campaign Queue for us...

Has this issue resolved for you Liliana Cheng

[Liliana_Cheng1]

Molly Cousins​, Marketo acknowledged that this is "an issue with their infrastructure causing performance delays" and said it could take months to roll out upgrades to their infrastructure. Incredibly frustrating for us and we are considering other options...

As of last week, the frequency and length of delays had slowed down for us - we noticed it happening once a week or so, and the leads would start moving through the triggers within two hours or so. For better or for worse, they had told us that these trigger delays seemed to be limited to just a handful of instances so it was not a widespread issue.

But today is a totally different story. Today has been significantly worse - my test leads from 10am still have ZERO activity on them apart from the original filled out form activity and the list of leads failing to qualify has grown. The senior support rep on our case advised that there was a server outage today that is affecting performance across a lot more instances. They had a P1 alert banner in the support portal that talked about performance issues all morning and most of this afternoon, but it's gone now.

I suggest that you open a ticket right away so they know your account is affected as well. They are planning to "replay the triggers" today but they have not confirmed what time this will happen.

Edit: never mind, I still see the P1 alert banner in the support portal. And my list of leads who are failing to qualify for their triggers keeps growing. They haven't fixed anything yet.

[Dan_Stevens_]

Right now, I'm noticing a pretty significant delay in trigger campaigns being run.  And nothing in my campaign queue.

[Liliana_Cheng1]

Dan Stevens​, our Senior Support Rep told us as of 8:30am Eastern today, they still have not replayed the triggers so this issue is not resolved yet. If you haven't already done so, I suggest opening a ticket with support and making sure they know how big of a deal this is for your instance.

Since we depend heavily on the Fills Out Form and Request Campaign triggers for syncing and routing, I have been manually syncing leads and routing since last night. I'm still seeing new leads from this morning who fail to qualify for any triggers, which makes sense because Marketo hasn't pushed a patch or resolved the root cause, whatever it is. So frustrating.

[Sam_Chou1]

Are you guys still having issues this week? I thought it would resolve over the weekend, but my campaigns are still inconsistently firing or taking 3+ hours to trigger. Had to re-open my case.

[Liliana_Cheng1]

Sam Chou​, yes this is still ongoing for us. I asked for an update today but haven't gotten anything useful. We have a call scheduled with one of the support managers later this week. I suggest consistent follow ups on your ticket with examples of the leads and the triggers they should be qualifying for.

I've been updating my ticket every few hours with new examples. Admittedly overkill, but I think it helps to drives the point home how much of a negative impact this has on our company. We can't have SQL leads sitting in Marketo without qualifying for their triggers for hours at a time, especially not during the end of the month, and especially not since we've raised this issue since March 6th without resolution. I have been manually reconciling and routing leads all day and it's an inefficient use of time.

[Bruce_Jeffers]

We also have been experiencing this quite a bit, off and on, over the past couple weeks. Most of the failures/delays are in campaigns with 'Filled Out Form' or 'Lead is Created' triggers. Support's response was just to convert as much as possible to Batch -- but that of course can't work for certain business-critical flows and processes. Looking forward to having this resolved.

[Molly_Cousins]

Our ticket has been escalated to engineering as well. Still seeing 8+ hours of delays. Bruce Jeffers​ definitely recommend opening a ticket if you haven't done so already -- my "Filled Out Form" and "Lead is Created" are the only triggers I've seen affected as well. Total pain!

[Josh_Hill13]

I can confirm our instance is also affected.

I spoke to a key person at Marketo yesterday. They are working on it.

[Kelly_Schmid3]

We have been affected too. Unfortunately, this has had a huge impact on our business, as our trial forms are processed through Marketo. No response from our Account Manager and still no known cause from Support...

[Josh_Hill13]

I agree that the Communication on this was less than optimal. No one posted that Update until Dan saw it and the update was from April 24.

Did anyone receive an email or support call about this?

[Josh_Hill13]

This appears isolated to some pods. Several pods went down and up over the past few weeks.

I've had a lot of similar issues with Marketo and did some of the same things you have all tried. Their answer has been "Move to Orion" if you aren't already on it.

Triggers typically are not "replayed" if the leads were not already in the Trigger Eval Queue. So you'd need a separate batch to fix things; adding a Campaign is Requested to triggers downstream of the batch might help you call the items you wanted to kick off. Use carefully.

Could be wrong about this, however, that's been my experience.

I'm not seeing the P1 notice right now. Did they resolve Liliana Cheng​ ?

[Liliana_Cheng1]

Hey Josh Hill​, thanks for your insight. A Senior Support Manager confirmed last week that "most" of our instance has already been moved to Orion, most notably the Spark Indexer component and some other components that would most affect trigger actions.

Additionally, our support rep has been working with engineering to get our triggers replayed regularly because of the large amounts of leads who just sit in our instance without action for 8+ hours at a time.

The latest from yesterday was "We cleared out our lag and replayed events this morning and we will do the same again later this evening as the backlog builds up. We are still working on a solution. We will keep you updated on a fix." so nope, they haven't resolved anything.

The volume of leads who fail to qualify for triggers, combined with the extended delays has gotten significantly worse in the past week. Back in March and early April, it used to affect a handful of leads per day that I could manage to manually reconcile without major interruption to my day. Since last week, the volume has grown to 70+ leads per day who don't qualify for one trigger or another and it has become obnoxiously disruptive.

I also want to note that the triggers that are failing are limited to the "Fills Out Form" trigger and the "Request Campaign" trigger, so those two are totally unreliable and cannot be used when I reconcile.

[Anonymous]

We noticed the delay as well, or rather that we have leads not going through the proper trigger campaigns. Any delay on 'Lead is Created' has caused certain triggers not to fire or become delayed. We've been running ad hoc programs (carefully designed and implemented) so we continue to run relatively smoothly. It sounds like you've had to do something similar,

I understand the frustration though! I spent a good chunk of time on Monday trying to figure out why these leads were not getting assigned a country and thus not syncing to sfdc.

Thanks for updating this thread as you hear from support and engineering.