21 Replies Latest reply on Nov 22, 2017 9:03 PM by Michelle Tang

    Safeguards for Marketo Outages

    Michelle Tang
    Michelle Tang Champion Nov 20, 2017 7:49 PM

      Hey Marketo Community! Some of you may have suffered the second Marketo outage within the last 3 months. We are so worried that this will happen again and want to be cautious and create a backup/alternative plan. Has anyone thought about this?

      For the first outage, our web forms did not submit so we put in an error message on the web forms to contact sales via phone or email.

      Any ideas to share would be great!

      • 908 Views
      • Categories: Marketing: Marketing Operations (MOPS) & Data, Marketing: Lessons Learned (Practitioners and CMOs), Marketing: Building Communities
      • Tags:
        • Re: Safeguards for Marketo Outages
          Darrell Alfonso
          Darrell Alfonso Champion Nov 20, 2017 10:54 PM (in response to Michelle Tang)

          I know Lawrence Mien replaced all marketo forms with CMS forms, so that's one back up plan you could have.

          1 of 1 people found this helpful
            • Re: Safeguards for Marketo Outages
              Lawrence Mien
              Lawrence Mien Nov 21, 2017 10:45 AM (in response to Darrell Alfonso)

              Yes! Since this is the second time in 3 months, we've put together an action plan so we can quickly remediate issues when a catastrophic failure occurs with Marketo. When Marketo went down, it severely affected our site, our salesforce, and our Google Adwords ads so our intention is immediate damage control. Mind you, our plan is pretty primitive and not sophisticated at all.

               

              1. Use a service that tracks response time/uptime such as DownNotifier.com to track our "https://app-xxxx.marketo.com/" and "https://www.pages.ourdomain.com" so we know if access to Marketo or our assets ever goes down. I'm assuming that the forms are tied to the same server that pages are on, so if pages go down we'll know our other assets are down as well.

               

              2. Have backup forms for all of our website forms (we use gravity forms in WordPress) and a google spreadsheet index of the locations of forms on our site, so if anything goes down we can switch it out ASAP to prevent loss of leads and site performance.

               

              3. Have a popup bar ready that notifies visitors that our marketo-hosted assets (only our eBooks) are currently unavailable. Luckily, our brochure, client testimonials, blogs and videos are all hosted on our regular web server. Alternatively, you can index where links to Marketo content are such as a page of all your ebooks, and disable/redirect those links.

               

              4. Double check performance on your site, if your Munchkin code is already loaded at the bottom of your page and it's slowing your load time down, go in and remove it. (find out how to remove it beforehand so you can act quick)

               

              5. We couldn't do anything for google analytics other than disable our ads, which Google does automatically anyways if your target URL does not work. We removed Marketo Sales Insight from our Salesforce layouts so it wouldn't affect the rest of the org who are using Salesforce.

               

              6. Pray Marketo is working their ***** off to fix it.

               

              75Mp1.gif

               

              7. Reinstate everything once it's fixed.

               

              Hope this helps!

              2 of 2 people found this helpful
                • Re: Safeguards for Marketo Outages
                  Michelle Tang
                  Michelle Tang Champion Nov 22, 2017 9:03 PM (in response to Lawrence Mien)

                  Thanks Lawrence! This list is super helpful! After the previous big outage 3 months back, we started putting messages on the web forms if they didn't submit and also started capturing additional data in multiple sources other than Marketo.

                   

                  Has anyone received any communication from their CSM about this outage? Even an apology or reason for it?

              • Re: Safeguards for Marketo Outages
                Sanford Whiteman
                Sanford Whiteman Nov 20, 2017 11:14 PM (in response to Michelle Tang)

                Well, by definition, there's no backup plan for catastrophic service failure that will maintain the user-facing experience. 

                 

                You should have IT monitor your Marketo services (branding domain, landing page domain, instance host) every minute so you can get immediate alerts without waiting for Marketo's acknowledgment. You can then, for example, switch LP domains to a static backup page with a link to a service like statuspage.io.  (Just as you would for your corporate website.)

                 

                TTLs on your Marketo-related DNS records should be 10m max to allow for faster response.

                 

                A fallback service for branding domains is basically impossible, since you can't rewrite tracking links in emails that have already been received, and you will not have access to the original URL. If you believe Marketo to be unstable, but you are not prepared to disable campaigns entirely, you should temporarily disable link tracking in your emails so you can be certain that the person will have someplace to land.

                3 of 3 people found this helpful
                • Re: Safeguards for Marketo Outages
                  Ashley Ayan
                  Ashley Ayan Nov 21, 2017 6:21 AM (in response to Michelle Tang)

                  If you have any integration with third-party platforms that has native form building capability, you can use their native form with API integration, instead of embedding MKTO forms directly to these platforms. In the event of a Marketo service interruption, visitors will still be able to submit forms and data will be saved in these platforms, and you will be able to bring it to your MKTO instance after the issue is resolved.

                  1 of 1 people found this helpful
                  • Re: Safeguards for Marketo Outages
                    Ayan Talukder
                    Ayan Talukder Nov 21, 2017 6:52 AM (in response to Michelle Tang)

                    Hmm this is a great point, we have never really thought about a backup option if Marketo goes down.

                     

                    It was unfortunate what happened yesterday, a bit annoying I imagine for most. I do appreciate their constant updates on:

                     

                    Marketo Status

                     

                    Much better communication than the last outage. I imagine it was better this time because they actually had a domain that wasn't expired to let us see the status haha.

                    • Re: Safeguards for Marketo Outages
                      Rebecca Koonse
                      Rebecca Koonse Nov 21, 2017 8:23 AM (in response to Michelle Tang)

                      Aside from forms, what can we do about an outage like yesterday that effected the REST API and Munchkin? Because the Munchkin is on our site, during the outage we had extremely slow page load speeds. Anyway to avoid that?

                        • Re: Safeguards for Marketo Outages
                          Sanford Whiteman
                          Sanford Whiteman Nov 21, 2017 8:34 AM (in response to Rebecca Koonse)

                          That doesn't make sense. If you use the Munchkin async embed code, a total Marketo outage has no effect on page load. (A network failure in the background has no effect on foreground events.)

                          • Re: Safeguards for Marketo Outages
                            Lawrence Mien
                            Lawrence Mien Nov 21, 2017 10:43 AM (in response to Rebecca Koonse)

                            If your Munchkin code script is at the top of your page, it will slow your site if it goes down. This is because the browser is attempting to load the script at no response, and it will continue to do that until it gives up before it fully loads the rest of your script. A good practice is to put your external tracking codes at the bottom of the page in the event that their service goes down.

                             

                            When Marketo went down, it heavily slowed down our site performance, so we just removed the munchkin code. Yes, we use asynchronous.

                              • Re: Safeguards for Marketo Outages
                                Sanford Whiteman
                                Sanford Whiteman Nov 21, 2017 10:43 AM (in response to Lawrence Mien)

                                The asynchronous code does not block on load. You are incorrect. Watch your Network tab.

                                  • Re: Safeguards for Marketo Outages
                                    Lawrence Mien
                                    Lawrence Mien Nov 21, 2017 10:48 AM (in response to Sanford Whiteman)

                                    I really don't care if "technically" I am. Yes, in asynchronous script is executed as soon as it's downloaded, without blocking the browser. But the fact is, our site performance was slowed down on all pages, so we removed it.

                                      • Re: Safeguards for Marketo Outages
                                        Sanford Whiteman
                                        Sanford Whiteman Nov 21, 2017 10:54 AM (in response to Lawrence Mien)

                                        Why put "technically" in quotes when you're talking about a purely technical matter?

                                         

                                        You should care about giving completely incorrect information, because again, people will repeat this later as if it's true ("I've heard...").

                                         

                                        Loading a script asynchronously -- any script, not just Munchkin -- is non-blocking. That's the definition of asynchronous.

                                          • Re: Safeguards for Marketo Outages
                                            Lawrence Mien
                                            Lawrence Mien Nov 21, 2017 11:00 AM (in response to Sanford Whiteman)

                                            I did that just for you.

                                             

                                            To my experience, it wasn't completely incorrect because the fact of the matter is, it slowed down our site. Things don't always work as expected. I don't see any harm in people removing and later reinstating their Munchkin code if it is slowing down their site for ANY reason during a Marketo outage.

                                             

                                            I'm not arguing if asynchronous is suppose to affect your load time or not, because like i also repeated, yes it shouldn't.

                                              • Re: Safeguards for Marketo Outages
                                                Lawrence Mien
                                                Lawrence Mien Nov 21, 2017 11:19 AM (in response to Lawrence Mien)

                                                I looked around a little more, and people say that async can still slow down your site if it doesn't load, especially if your page is script heavy:

                                                 

                                                https://www.nccgroup.trust/uk/about-us/newsroom-and-events/blogs/2016/september/how-loading-javascript-can-slow-websites…

                                                 

                                                Beware of "script async defer" blocking HTML "load" event · GitHub

                                                 

                                                "What might be surprising is that <script async defer> also blocks the raising of load event!
                                                In a web dev slang, you may hear a script with async flag to be called non-blocking, but they're not blocking only the construction of the DOM tree (i.e. DOMContentLoaded event). But they're still blocking the load event.
                                                Unfortunately we were doing just that: using window.onload = ... instead of listening to DOMContentLoaded to boot our app. In normal circumstances, the difference was negligible, but today it his us hard.
                                                Actually, in our case, we were loading an external script via JS, using script.async = true, and that external script was in turn loading several others, also setting the async flag to each of them. However, no amount of asynccan save you if you're waiting for loadevent!"

                                                 

                                                I'm not a JS expert of any sorts, so I can't attest to whether these statements are completely true, but based on my experience I'm going to go with it. They do have some useful steps to add on that will further prevent blocking.

                                                  • Re: Safeguards for Marketo Outages
                                                    Sanford Whiteman
                                                    Sanford Whiteman Nov 21, 2017 1:11 PM (in response to Lawrence Mien)

                                                    The operation of window.onload doesn't change the asynchronous contract. By choosing to use onload you're artificially creating a load order, just as you would be if you waited for a certain script.onload to fire before making the site ready to use.

                                                     

                                                    Also note that the Window's load event is misnamed/misleading when you compare it to the load events fired by DOM elements like <img> and <script>.  While those elements have both load and error events (and without both you aren't covering all the possible conditions), load on Window fires when the load or error state is reached for all assets. Ergo, if the remote <script> that loads Munchkin immediately fails due to a network outage, window.onload will fire without interruption.

                                                    • Re: Safeguards for Marketo Outages
                                                      Sanford Whiteman
                                                      Sanford Whiteman Nov 21, 2017 11:30 AM (in response to Lawrence Mien)

                                                      They do have some useful steps to add on that will prevent blocking for good.

                                                      You can't prevent blocking Munchkin events (not library load, but the page event Click Link) via generic methods. You must turn off exit link tracking to do this.

                                                    • Re: Safeguards for Marketo Outages
                                                      Sanford Whiteman
                                                      Sanford Whiteman Nov 21, 2017 11:34 AM (in response to Lawrence Mien)

                                                      Things don't always work as expected.

                                                      Async loading of the Munchkin library always works the same way, per the contract between all modern browsers and JavaScript authors. Complete failure of the lib to load asynchronously (due to complete outage at Marketo) will not, on its own, affect page rendering.

                                                       

                                                      However, a network slowdown -- not complete outage -- at Marketo is a different story. In that case, if the library is ever able to complete loading, inter-page navigation will be heavily affected (if you haven't already disabled exit link tracking for performance reasons, a very good idea in its own right). It's far more likely that this was what you were experiencing.

                                                       

                                                      I agree that there's no harm in disabling all impacted services during a Marketo outage. If you're not sure whether Munchkin is affected then you can disable it to be safe. But you should know why you're doing everything, including that you may be disabling a service that wasn't impacted and or amounted to a no-op (i.e disabling something that was implicitly disabled).