29 Replies Latest reply on May 1, 2017 2:43 PM by Liliana Cheng

    Leads failing to qualify for triggers

    Liliana Cheng

      Hey community, our instance has been experiencing major delays (two to four hours) in our triggers. Some leads who fill out a form do not qualify for the triggers they're supposed to, sometimes for hours, sometimes for over a day. This issue is not isolated to one particular form or trigger, it's across our instance and it's inconsistently happening because some of my test leads have qualified for the trigger without an issue.

       

      I've spoken to two Marketo support reps who confirmed that our triggers are set up correctly to capture, and the affected leads SHOULD be qualifying for the triggers. They also confirmed that we do not have a heavy volume of leads in the trigger queue. I am still waiting for a manager to update me but this is troubling.

       

      What confuses me is when leads just sit in Marketo for two+ hours without any further activity will suddenly qualify for their trigger. No further actions were taken from the lead and I didn't manually request the campaign but they qualified - why is this happening? So most of the leads who failed to trigger immediately yesterday are okay after four hours... Is anyone else experiencing this massive delay in triggered behaviors?

       

      If it makes a difference, I noticed this yesterday 3/6/2017 around 1:50pm Eastern and leads kept failing to qualify until 3pm. Another round of trigger failures happened between 7pm and 9pm. Most of them were caught up this morning but the most recent round of failures started around 11:30am today.

       

      Thanks.

        • Re: Leads failing to qualify for triggers
          Dan Stevens

          We encountered something similar a few months ago (For those of you on Corona (part of Project Orion), are you experiencing issues explained here? ), and occasionally experience this behavior.  I wonder if this has anything to do with the Orion migration.

            • Re: Leads failing to qualify for triggers
              Liliana Cheng

              Thanks for the link, Dan Stevens. I added it to the case that I opened and asked if it was related to the Orion migration. I see that Ajay Awatramani replied to your discussion saying this bug was patched in November, but you mentioned that you still experience this behavior occasionally. How does your team handle these occasional delays? Does Marketo know this is still happening to your instance even after their patch?

               

              Right now I'm just refreshing a smart list that catches leads who filled out any form, but do not have a SFDC created date, and then manually reconciling them using the "request campaign" action for the triggers they were supposed to qualify for. I'm also a little frustrated by the lack of communication from the support team so far - the ticket was pushed up to a manager over 24hrs ago and I haven't heard anything from them even though the reps I spoke to this morning said there are new internal notes on my case...

            • Re: Leads failing to qualify for triggers
              Sanford Whiteman

              Liliana, in addition to the great tips from experience from Dan & Josh, bear in mind that trigger backlogs -- though always unwanted -- can theoretically happen regardless of the load on your instance or pod, by virtue of the very fact that there is a queue.

               

              That is, when a system has multiple services handing off data to each other -- as any scaleable system must -- instead of a direct throughline from the client to all the back-end servers, that means more moving parts. And it also means temporary failure of any of the parts that enqueue and dequeue data can result in a later "replay" of delayed actions. That replay may seem mysterious, even impossible, but actually is due to the (relative) resilience of the architecture. Of course we don't want any failures, but when they happen it's reassuring that the queue pipeline continues where it left off instead of losing data.

              • Re: Leads failing to qualify for triggers
                Molly Cousins

                I'm experiencing this issue as well - and on the Filled Out Form trigger, to boot! Nothing amiss in Campaign Queue for us...

                 

                Has this issue resolved for you Liliana?

                  • Re: Leads failing to qualify for triggers
                    Liliana Cheng

                    Molly Cousins, Marketo acknowledged that this is "an issue with their infrastructure causing performance delays" and said it could take months to roll out upgrades to their infrastructure. Incredibly frustrating for us and we are considering other options...

                     

                    As of last week, the frequency and length of delays had slowed down for us - we noticed it happening once a week or so, and the leads would start moving through the triggers within two hours or so. For better or for worse, they had told us that these trigger delays seemed to be limited to just a handful of instances so it was not a widespread issue.

                     

                    But today is a totally different story. Today has been significantly worse - my test leads from 10am still have ZERO activity on them apart from the original filled out form activity and the list of leads failing to qualify has grown. The senior support rep on our case advised that there was a server outage today that is affecting performance across a lot more instances. They had a P1 alert banner in the support portal that talked about performance issues all morning and most of this afternoon, but it's gone now.

                     

                    I suggest that you open a ticket right away so they know your account is affected as well. They are planning to "replay the triggers" today but they have not confirmed what time this will happen.

                     

                    Edit: never mind, I still see the P1 alert banner in the support portal. And my list of leads who are failing to qualify for their triggers keeps growing. They haven't fixed anything yet.

                    Screen Shot 2017-04-20 at 4.24.46 PM.png

                    2 of 2 people found this helpful
                      • Re: Leads failing to qualify for triggers
                        Molly Cousins

                        Liliana Cheng Wow. I opened a P2 ticket 3+ hrs ago and hadn't heard anything back so I guess that explains their slow response. Thanks for the info! This is nuts.

                        • Re: Leads failing to qualify for triggers
                          Dan Stevens

                          Right now, I'm noticing a pretty significant delay in trigger campaigns being run.  And nothing in my campaign queue.

                          1 of 1 people found this helpful
                            • Re: Leads failing to qualify for triggers
                              Liliana Cheng

                              Dan Stevens, our Senior Support Rep told us as of 8:30am Eastern today, they still have not replayed the triggers so this issue is not resolved yet. If you haven't already done so, I suggest opening a ticket with support and making sure they know how big of a deal this is for your instance.

                               

                              Since we depend heavily on the Fills Out Form and Request Campaign triggers for syncing and routing, I have been manually syncing leads and routing since last night. I'm still seeing new leads from this morning who fail to qualify for any triggers, which makes sense because Marketo hasn't pushed a patch or resolved the root cause, whatever it is. So frustrating.

                              1 of 1 people found this helpful
                                • Re: Leads failing to qualify for triggers
                                  Sam Chou

                                  Are you guys still having issues this week? I thought it would resolve over the weekend, but my campaigns are still inconsistently firing or taking 3+ hours to trigger. Had to re-open my case.

                                    • Re: Leads failing to qualify for triggers
                                      Liliana Cheng

                                      Sam Chou, yes this is still ongoing for us. I asked for an update today but haven't gotten anything useful. We have a call scheduled with one of the support managers later this week. I suggest consistent follow ups on your ticket with examples of the leads and the triggers they should be qualifying for.

                                       

                                      I've been updating my ticket every few hours with new examples. Admittedly overkill, but I think it helps to drives the point home how much of a negative impact this has on our company. We can't have SQL leads sitting in Marketo without qualifying for their triggers for hours at a time, especially not during the end of the month, and especially not since we've raised this issue since March 6th without resolution. I have been manually reconciling and routing leads all day and it's an inefficient use of time.

                                        • Re: Leads failing to qualify for triggers
                                          Josh Hill

                                          This appears isolated to some pods. Several pods went down and up over the past few weeks.

                                           

                                          I've had a lot of similar issues with Marketo and did some of the same things you have all tried. Their answer has been "Move to Orion" if you aren't already on it.

                                           

                                          Triggers typically are not "replayed" if the leads were not already in the Trigger Eval Queue. So you'd need a separate batch to fix things; adding a Campaign is Requested to triggers downstream of the batch might help you call the items you wanted to kick off. Use carefully.

                                           

                                          Could be wrong about this, however, that's been my experience.

                                           

                                          I'm not seeing the P1 notice right now. Did they resolve Liliana Cheng ?

                                          2 of 2 people found this helpful
                                            • Re: Leads failing to qualify for triggers
                                              Liliana Cheng

                                              Hey Josh Hill, thanks for your insight. A Senior Support Manager confirmed last week that "most" of our instance has already been moved to Orion, most notably the Spark Indexer component and some other components that would most affect trigger actions.

                                               

                                              Additionally, our support rep has been working with engineering to get our triggers replayed regularly because of the large amounts of leads who just sit in our instance without action for 8+ hours at a time.

                                               

                                              The latest from yesterday was "We cleared out our lag and replayed events this morning and we will do the same again later this evening as the backlog builds up. We are still working on a solution. We will keep you updated on a fix." so nope, they haven't resolved anything.

                                               

                                              The volume of leads who fail to qualify for triggers, combined with the extended delays has gotten significantly worse in the past week. Back in March and early April, it used to affect a handful of leads per day that I could manage to manually reconcile without major interruption to my day. Since last week, the volume has grown to 70+ leads per day who don't qualify for one trigger or another and it has become obnoxiously disruptive.

                                               

                                              I also want to note that the triggers that are failing are limited to the "Fills Out Form" trigger and the "Request Campaign" trigger, so those two are totally unreliable and cannot be used when I reconcile.

                                              1 of 1 people found this helpful
                                                • Re: Leads failing to qualify for triggers
                                                  Haleigh Stothers

                                                  We noticed the delay as well, or rather that we have leads not going through the proper trigger campaigns. Any delay on 'Lead is Created' has caused certain triggers not to fire or become delayed. We've been running ad hoc programs (carefully designed and implemented) so we continue to run relatively smoothly. It sounds like you've had to do something similar,

                                                  I understand the frustration though! I spent a good chunk of time on Monday trying to figure out why these leads were not getting assigned a country and thus not syncing to sfdc.

                                                  Thanks for updating this thread as you hear from support and engineering.

                                              • Re: Leads failing to qualify for triggers
                                                Bruce Jeffers

                                                We also have been experiencing this quite a bit, off and on, over the past couple weeks. Most of the failures/delays are in campaigns with 'Filled Out Form' or 'Lead is Created' triggers. Support's response was just to convert as much as possible to Batch -- but that of course can't work for certain business-critical flows and processes. Looking forward to having this resolved.

                                                1 of 1 people found this helpful
                                    • Re: Leads failing to qualify for triggers
                                      Dan Stevens

                                      Here's an official statement by Marketo Support: Marketo Service Interruption Update

                                      • Re: Leads failing to qualify for triggers
                                        Liliana Cheng

                                        5/1/17 UPDATE - this issue looks like it's been resolved (for now). We had a call with our account exec and a support escalation manager this past Friday, and they said the engineers planned to push a patch over the weekend. Since then, I've been monitoring and testing and our triggers seem to be working as they're built to be.

                                         

                                        They haven't given us an in-depth analysis of what happened, but they said it was related to "the high speed database used to capture activity and how it was tied to trigger processing". The patch that was pushed this weekend was to remove the dependency. I'm not sure what their plan is after this (whether or not they'll tie these two problematic things together again).

                                         

                                        If you are still seeing issues related to delayed triggers, you should contact support with some examples so they're aware this is still an issue for you.