2 Replies Latest reply on Jun 23, 2017 11:56 AM by Sanford Whiteman

    Another service interruption affecting trigger campaigns - when is Orion going to prevent this?

    Dan Stevens.

      This morning, we received a troubling notification that our instance experienced a service interruption this week (from Monday-Wednesday):


      The Marketo system hosting your subscription experienced a Service Interruption which affected the logging of activities. We have identified the issue and current activity logging has been fully restored. 


      When: Wednesday, June 21st 5:30 AM PT


      Duration: Monday 6/19 - Wednesday 6/21


      Service affected:  During the affected period, you would have seen a delay in building smart campaign membership, along with a small subset of activities that were not processed.  Login to the Marketo application, Landing Pages, and the rest of the platform 

      were not affected.


      Cause: The cause has been identified as a storage issue within our  Activity Service Servers that are used to log lead activities.


      Resolution: To address this, we have added more capacity to our Activity Service Servers, along with an optimization patch for faster campaign membership processing. 


      Activities that were not logged during this time are being reinserted. This process can take up to two weeks to complete. During this time, you may see some duplicate activities in lead activity logs. We  will be removing those duplicate entries through a data fix, which could take 4 weeks to process. While the duplicate activities are being reinserted and cleaned up, they will not trigger any campaigns. If you have Smart lists listening for any duplicate activity filters such as (Visit Web Page, Click link, etc) with a constraint of number of times, this may qualify your leads temporarily, until we complete the cleanup. We will be sure to provide a progress update next week.


      We want to assure you that we take these incidents very seriously, and we are taking steps to prevent similar situations going forward. If you have any questions about the Service Interruption, please contact Marketo Customer Support team at http://support.marketo.com and they will be happy to answer any questions you might have.  Thank you for understanding and being a Marketo Customer.

      This is very concerning - especially the amount of time it's going to take to resolve this (which sounds like it will also affect our reporting).  Isn't this something that the Orion architecture is supposed to prevent?  And if so, what is the status of getting Orion deployed to Marketo's customers?  Seems odd, that the server's aren't being monitored today to ensure they contain proper storage.


      Just curious, has this impacted any of your instances?