Scheduled maintenance/outage for Ashburn datacenter on Saturday, February 9, 2019 1pm to 9pm ET

[Liliana_Cheng1]

Hey community,

We received Marketo's email alert about scheduled mandatory maintenance for the Ashburn datacenter which will disrupt all Marketo services. The maintenance itself is only expected to take one hour, but they cautioned there could be service disruptions within that eight hour window. Their maintenance window (1pm to 9pm Eastern) includes some of our high traffic time frames, and I'm sure it include some of your high traffic time frames as well.

We emailed our CSM to see if they would reschedule the maintenance window to the overnight hours, which is how most tech companies handle maintenance that causes outages, but we were denied without explanation from Marketo. We also asked for suggestions for alternatives while Marketo's services are unavailable and they said "Please plan accordingly depending on your business needs. If you have any high value campaigns scheduled, they should be rescheduled" which is laughable because we still have goals to meet and projects that have taken weeks of work across different teams at our company.

I understand the maintenance needs to happen, but I don't understand why it's scheduled for a day time outage. If it's such an emergency, wouldn't they handle it sooner than nine days (email alert was sent yesterday)? And if it's not an emergency, why couldn't they handle it during the evening instead of during the day? Those are questions for Marketo.

My question for the community is: what is your plan for Marketo service outages?

We have lead forms all over our site, we use Marketo for automated emails, scoring, sales routing, etc. What should we do when none of this is available? I'm more concerned about lead capture since we expect that Marketo forms will not be available across our site.

Thanks.

[Liliana_Cheng1]

Sanford Whiteman​, you've helped us so many times - do you have input that we could take into consideration here?

[SanfordWhiteman]

do you have input that we could take into consideration here?

If it's critical (and I absolutely understand that is, as does my fellow MUGger Jeff Pollock​ with whom I was discussing this stuff recently) then there are indeed measures you can take to still serve up Marketo forms and archive the posted form data for later import when the service is up. However, it takes significant developer time.

[Jeff_Pollock]

Thank you for your recommendations on this thread from last month, Sandy! I'm currently working with our in-house devs to set up the appropriate monitor for delays between a physical form fill and when the Filled Out Form activity shows up in Marketo (followed by a new lead created in SFDC).

For this instance, Liliana Cheng​ and I are concerned about the Marketo forms not loading to our site at all.

...there are indeed measures you can take to still serve up Marketo forms and archive the posted form data for later import when the service is up.

Do you have any further info on this at all? We're thinking about building out a custom form to capture the same fields Marketo is built to capture if Marketo doesn't load within ~10s of a page launching, but that kinda defeats the purpose of using Marketo for form fills in the first place.

[SanfordWhiteman]

Oh, I know exactly how to deal with this, but would need more backing to make it generally available in this timeframe.

[Jeff_Pollock]

"backing?"

The timeframe is too short for us to build out what we need currently, but I'm excited to learn what we can do to better prepare next time around!

[SanfordWhiteman]

Like if more people seemed to care.

Check the blog... some direction on there at some point!

[Mike_Reynolds2]

Liliana Cheng​

We apologize for any inconvenience this mandatory maintenance may cause and understand this time window may not be ideal for some customers. We never want to interrupt service without warning, so we always provide a minimum 7-day advanced notice before any maintenance that could cause a disruption.

As a global operation, it’s difficult to find a time that has the lowest traffic in all geographic regions. The maintenance was schedules for a Saturday because, as a weekend, it doesn’t fall on a workday in any country. Additionally, we scheduled the maintenance during the daytime in the US because that gives us the best access to the vendors and technical teams we need standing by to address any problems that may occur. We’ve scheduled additional coverage from our Network, Engineering and Support teams and will be actively monitoring progress throughout the maintenance period.

The maintenance being performed is critical in nature, so it cannot be delayed. As network traffic has increased, we must also scale our infrastructure to prevent service disruptions. We are replacing our network’s firewall devices, which will improve security and add capacity for all customers. While maintenance processes are an inconvenience, we are planning this one in advance to prevent unplanned service outages in the future.

We have posted notifications on our Status page https://status.marketo.com/ and will be updating the alert throughout the maintenance process. If you have any questions, please Contact Marketo Support

Best regards,

Mike