ISSUE: Trigger campaigns not firing for specific lead activity

[Dan_Stevens_]

On Wednesday, we imported some lists into our event programs (event attendees) - where we specifically defined the program status ("attended"); and many of these leads were new leads.  We have the following trigger campaigns setup to listen for specific activity:

• Lead is created - country is 'x'.  New leads are routed immediately to the appropriate country lead partition based on their country.  This is especially important when the imported leads are done in the global/default partition (which is common for global events)
  
  
• Program status is changed.  When leads reach specific program statuses - especially a success status like "attended" - a lot of processing takes place as a result (proper scoring, defining last touch channel, LT program, LT programID, LT date, assignment/alert to marketing if fast-tracked to MQL, etc.).

None of these triggers fired.  I immediately opened a ticket with Support and they are currently looking into it. 

They mentioned that there was some sort of incident or known issue that took place on March 16 that could be affecting this.  Today, I received the following update:

We are currently investigating our HBase system as a potential root cause.  We are testing this and will update you further as additional progress is made.

Just curious - have any of you had issues this week with your trigger campaigns not firing for specific activities like this - in our case, through a list import?  I know there was another issue related to list imports where the import would fail.  This is different.

It's probably also worth mentioning that yes, we did ensure there was no backlog of campaigns in the "Campaign Queue"

[Jenn_DiMaria2]

Hey, Dan!

Funny I came across this right now. I just posted that we had a trigger campaign not fire recently and I'm trying to diagnose the cause (ie: a First Touch campaign that's supposed to fire when a person is created in the past 48 hours, didn't). Wondering if this has something to do with why.

[Dan_Stevens_]

Hi Jenn!  Might be worth opening a support case as well, so that Marketo can see that this is affecting multiple customers (and hopefully indicate the severity of this). 

[Loren_Posendek]

Hi Dan! We had a similar issue to triggered campaigns not firing for a specific activity. We immediately opened a Marketo Support case and within about 20-30 minutes after opening the case the triggers resumed normal functionality.

What was interesting is that we dug into it more to realize the triggers had misfired on and off for about 2 hours. Our campaign queue wasn't overloaded any more than normal. The activities eventually caught up, but it took about 5-7 hours after the initial activity to catch up. Marketo Support gave us a basic response related to the number of triggers we had firing, but we still thought it was odd.

We've been monitoring and haven't seen any issues since. Just thought I'd share!

[Dan_Stevens_]

Hi Loren - apparently this was an outage that just happened to take place when members were being added to programs last week.  Consistent with what you've been told, Support told us:

There was an issue affecting customers that had their instances residing on servers in our Ashburn Data Center from March 14 - 16. In your case, there was a hiccup for SJ pods from Noon - 2:00PM PDT on March 21st. Unfortunately, it is not feasible to go back in time to evaluate previous state data and to perform a data fix.

We, too, have been issue-free since the 21st.

[David_Desrosie2]

Observing the same on form fill trigger tonight...Anyone else?

[David_Desrosie2]

New thread here: Leads failing to qualify for triggers

[Robb_Barrett]

We're experiencing this issue over the last couple of days as well.  This morning, several leads were imported and some of the DVCs should have triggered campaigns but did not.

[Robb_Barrett]

Make sure you're on Orion by submitting a support ticket asking to verify. That helped out our trigger queue.  I find it a bit too ironic that, at summit, a new trigger queue tool was announced as being in development and recently people on all pods across the country are having trigger queue issues.

Submit a support ticket asking for a snapshot of your trigger queue.  Also, go into your Campaign Inspector and turn off any trigger campaigns that aren't needed, such as event verification SC's for past events - don't wait the 6 months for them to automatically turn off.  Change any triggers you can into recurring batch campaigns.

Take a look in your Campaign Queue to see if you have one particular campaign causing a bottleneck. Sometimes imports can do this if someone saves an XLS to CSV and forgets to delete the empty or formula rows.