Hey Marketo Community! Some of you may have suffered the second Marketo outage within the last 3 months. We are so worried that this will happen again and want to be cautious and create a backup/alternative plan. Has anyone thought about this?
For the first outage, our web forms did not submit so we put in an error message on the web forms to contact sales via phone or email.
Any ideas to share would be great!
I know Lawrence Mien replaced all marketo forms with CMS forms, so that's one back up plan you could have.
Yes! Since this is the second time in 3 months, we've put together an action plan so we can quickly remediate issues when a catastrophic failure occurs with Marketo. When Marketo went down, it severely affected our site, our salesforce, and our Google Adwords ads so our intention is immediate damage control. Mind you, our plan is pretty primitive and not sophisticated at all.
1. Use a service that tracks response time/uptime such as DownNotifier.com to track our "https://app-xxxx.marketo.com/" and "https://www.pages.ourdomain.com" so we know if access to Marketo or our assets ever goes down. I'm assuming that the forms are tied to the same server that pages are on, so if pages go down we'll know our other assets are down as well.
2. Have backup forms for all of our website forms (we use gravity forms in WordPress) and a google spreadsheet index of the locations of forms on our site, so if anything goes down we can switch it out ASAP to prevent loss of leads and site performance.
3. Have a popup bar ready that notifies visitors that our marketo-hosted assets (only our eBooks) are currently unavailable. Luckily, our brochure, client testimonials, blogs and videos are all hosted on our regular web server. Alternatively, you can index where links to Marketo content are such as a page of all your ebooks, and disable/redirect those links.
4. Double check performance on your site, if your Munchkin code is already loaded at the bottom of your page and it's slowing your load time down, go in and remove it. (find out how to remove it beforehand so you can act quick)
5. We couldn't do anything for google analytics other than disable our ads, which Google does automatically anyways if your target URL does not work. We removed Marketo Sales Insight from our Salesforce layouts so it wouldn't affect the rest of the org who are using Salesforce.
6. Pray Marketo is working their ***** off to fix it.
7. Reinstate everything once it's fixed.
Hope this helps!
Thanks Lawrence! This list is super helpful! After the previous big outage 3 months back, we started putting messages on the web forms if they didn't submit and also started capturing additional data in multiple sources other than Marketo.
Has anyone received any communication from their CSM about this outage? Even an apology or reason for it?
Well, by definition, there's no backup plan for catastrophic service failure that will maintain the user-facing experience.
You should have IT monitor your Marketo services (branding domain, landing page domain, instance host) every minute so you can get immediate alerts without waiting for Marketo's acknowledgment. You can then, for example, switch LP domains to a static backup page with a link to a service like statuspage.io. (Just as you would for your corporate website.)
TTLs on your Marketo-related DNS records should be 10m max to allow for faster response.
A fallback service for branding domains is basically impossible, since you can't rewrite tracking links in emails that have already been received, and you will not have access to the original URL. If you believe Marketo to be unstable, but you are not prepared to disable campaigns entirely, you should temporarily disable link tracking in your emails so you can be certain that the person will have someplace to land.
If you have any integration with third-party platforms that has native form building capability, you can use their native form with API integration, instead of embedding MKTO forms directly to these platforms. In the event of a Marketo service interruption, visitors will still be able to submit forms and data will be saved in these platforms, and you will be able to bring it to your MKTO instance after the issue is resolved.
Hmm this is a great point, we have never really thought about a backup option if Marketo goes down.
It was unfortunate what happened yesterday, a bit annoying I imagine for most. I do appreciate their constant updates on:
Much better communication than the last outage. I imagine it was better this time because they actually had a domain that wasn't expired to let us see the status haha.
Aside from forms, what can we do about an outage like yesterday that effected the REST API and Munchkin? Because the Munchkin is on our site, during the outage we had extremely slow page load speeds. Anyway to avoid that?
That doesn't make sense. If you use the Munchkin async embed code, a total Marketo outage has no effect on page load. (A network failure in the background has no effect on foreground events.)
Marketo lists Munchkin as one of the areas effected yesterday on their site here - Marketo Status and my IT team is saying it effected the page load speeds because the site was waiting for a response from Marketo. Is that not possible?
It is not possible as described, because such "waiting" is in the background.
When Munchkin completely fails, your page is unaffected, unless you're using the synchronous (more properly called the "part-synchronous") form of the embed code.
A partial Munchkin failure, for example loading the Munchkin library successfully but having the Munchkin logging endpoint be slow to respond, will affect the end-user experience.
If your Munchkin code script is at the top of your page, it will slow your site if it goes down. This is because the browser is attempting to load the script at no response, and it will continue to do that until it gives up before it fully loads the rest of your script. A good practice is to put your external tracking codes at the bottom of the page in the event that their service goes down.
When Marketo went down, it heavily slowed down our site performance, so we just removed the munchkin code. Yes, we use asynchronous.
The asynchronous code does not block on load. You are incorrect. Watch your Network tab.
I really don't care if "technically" I am. Yes, in asynchronous script is executed as soon as it's downloaded, without blocking the browser. But the fact is, our site performance was slowed down on all pages, so we removed it.
Why put "technically" in quotes when you're talking about a purely technical matter?
You should care about giving completely incorrect information, because again, people will repeat this later as if it's true ("I've heard...").
Loading a script asynchronously -- any script, not just Munchkin -- is non-blocking. That's the definition of asynchronous.
I did that just for you.
To my experience, it wasn't completely incorrect because the fact of the matter is, it slowed down our site. Things don't always work as expected. I don't see any harm in people removing and later reinstating their Munchkin code if it is slowing down their site for ANY reason during a Marketo outage.
I'm not arguing if asynchronous is suppose to affect your load time or not, because like i also repeated, yes it shouldn't.
I looked around a little more, and people say that async can still slow down your site if it doesn't load, especially if your page is script heavy:
Beware of "script async defer" blocking HTML "load" event · GitHub
"What might be surprising is that
<script async defer>also blocks the raising ofloadevent!In a web dev slang, you may hear a script with
asyncflag to be called non-blocking, but they're not blocking only the construction of the DOM tree (i.e.DOMContentLoadedevent). But they're still blocking theloadevent.Unfortunately we were doing just that: using
window.onload = ...instead of listening toDOMContentLoadedto boot our app. In normal circumstances, the difference was negligible, but today it his us hard.Actually, in our case, we were loading an external script via JS, using
script.async = true, and that external script was in turn loading several others, also setting theasyncflag to each of them. However, no amount ofasynccan save you if you're waiting forloadevent!"
I'm not a JS expert of any sorts, so I can't attest to whether these statements are completely true, but based on my experience I'm going to go with it. They do have some useful steps to add on that will further prevent blocking.
The operation of window.onload doesn't change the asynchronous contract. By choosing to use onload you're artificially creating a load order, just as you would be if you waited for a certain script.onload to fire before making the site ready to use.
Also note that the Window's load event is misnamed/misleading when you compare it to the load events fired by DOM elements like <img> and <script>. While those elements have both load and error events (and without both you aren't covering all the possible conditions), load on Window fires when the load or error state is reached for all assets. Ergo, if the remote <script> that loads Munchkin immediately fails due to a network outage, window.onload will fire without interruption.
They do have some useful steps to add on that will prevent blocking for good.
You can't prevent blocking Munchkin events (not library load, but the page event Click Link) via generic methods. You must turn off exit link tracking to do this.
Things don't always work as expected.
Async loading of the Munchkin library always works the same way, per the contract between all modern browsers and JavaScript authors. Complete failure of the lib to load asynchronously (due to complete outage at Marketo) will not, on its own, affect page rendering.
However, a network slowdown -- not complete outage -- at Marketo is a different story. In that case, if the library is ever able to complete loading, inter-page navigation will be heavily affected (if you haven't already disabled exit link tracking for performance reasons, a very good idea in its own right). It's far more likely that this was what you were experiencing.
I agree that there's no harm in disabling all impacted services during a Marketo outage. If you're not sure whether Munchkin is affected then you can disable it to be safe. But you should know why you're doing everything, including that you may be disabling a service that wasn't impacted and or amounted to a no-op (i.e disabling something that was implicitly disabled).
I don't understand the reference or source of the GIF.
on Nov 20, 2017
Yes! Since this is the second time in 3 months, we've put together an action plan so we can quickly remediate issues when a catastrophic failure occurs with Marketo. When Marketo went down, it severely affected our site, our salesforce, and our Google Adwords ads so our intention is immediate damage control. Mind you, our plan is pretty primitive and not sophisticated at all.
1. Use a service that tracks response time/uptime such as DownNotifier.com to track our "https://app-xxxx.marketo.com/" and "https://www.pages.ourdomain.com" so we know if access to Marketo or our assets ever goes down. I'm assuming that the forms are tied to the same server that pages are on, so if pages go down we'll know our other assets are down as well.
2. Have backup forms for all of our website forms (we use gravity forms in WordPress) and a google spreadsheet index of the locations of forms on our site, so if anything goes down we can switch it out ASAP to prevent loss of leads and site performance.
3. Have a popup bar ready that notifies visitors that our marketo-hosted assets (only our eBooks) are currently unavailable. Luckily, our brochure, client testimonials, blogs and videos are all hosted on our regular web server. Alternatively, you can index where links to Marketo content are such as a page of all your ebooks, and disable/redirect those links.
4. Double check performance on your site, if your Munchkin code is already loaded at the bottom of your page and it's slowing your load time down, go in and remove it. (find out how to remove it beforehand so you can act quick)
5. We couldn't do anything for google analytics other than disable our ads, which Google does automatically anyways if your target URL does not work. We removed Marketo Sales Insight from our Salesforce layouts so it wouldn't affect the rest of the org who are using Salesforce.
6. Pray Marketo is working their ***** off to fix it.
7. Reinstate everything once it's fixed.
Hope this helps!