14 Replies Latest reply on Oct 19, 2017 1:06 PM by Sanford Whiteman

    Branded Domain Broke All Email Links

    Rusty Hall

      Hey All,

       

      We recently set up a branded domain as per the instructions from Marketo's product docs. http://docs.marketo.com/display/public/DOCS/Configure+Protocols+for+Marketo

      IT set-up the new CNAME in the DNS record and pointed our new CNAME marketo.armor.com to our tracking link mkto-sjl0027.com

       

      After we changed our Branding Domain in the admin panel to marketo.armor.com all of our email links broke. We simply get a site can't be reached error on any email links.

       

      I know that a DNS update can take 72 hours in some cases but we've checked and it's propagated fully by now.

       

      Any thoughts or help would be greatly appreciated as we are trying to get this resolved as fast as possible.

        • Re: Branded Domain Broke All Email Links
          Sanford Whiteman
          I know that a DNS update can take 72 hours in some cases but we've checked and it's propagated fully by now.

          Actually adding a new CNAME RR to an existing zone is instantaneous. There's no propagation delay.

           

          I do see that marketo.armor.com is pointed to mkto-sjl0027.com.  When you say you're getting a "site can't be reached" (not "the redirect URL is empty") I wonder if this is because you have an internal DNS server that has its own copy of the armor.com zone, and that internal side hasn't been updated by IT. Have you tested from outside your corporate network (including VPN clients)?

            • Re: Branded Domain Broke All Email Links
              Rusty Hall

              Hi Sanford!

               

              Thanks for the reply. Good to know on the propagation timing.

              I have tested it inside and outside our networkand I get the same result. I've attached a screen shot of what happens when you click a link in one of our Marketo emails.

              It's the same for all email links with tracking enabled.

               

              CNAME redirect not working.png

              I also ran a DNS check and it shows to be working and updated as intended.

              DNS Check.png

               

              "I wonder if this is because you have an internal DNS server that has its own copy of the armor.com zone, and that internal side hasn't been updated by IT."

              I'll raise this question with our IT support. I'm stumped as to what the issue could be.

               

              Any additional feedback or ideas are greatly appreciated!

                • Re: Branded Domain Broke All Email Links
                  Sanford Whiteman

                  It's SSL, not DNS, that's the direct problem.

                   

                  You don't have SSL enabled on your branding domain, but your links are being rewritten to https://.

                    • Re: Branded Domain Broke All Email Links
                      Rusty Hall

                      Ah! Good to know Sanford. A good catch.

                       

                      Forgive my lack of knowledge here but, do I just need to have our IT team install a SSL certificate on the domain marketo.armor.com ?

                        • Re: Branded Domain Broke All Email Links
                          Sanford Whiteman

                          Eh no. You have to open a support case and provide your SSL cert to Marketo, then they install the cert. For a fresh SSL install, this can actually take them some time (like weeks) but I'm assuming you already have SSL on your Marketo LP domain, so it should be fast/faster.

                            • Re: Branded Domain Broke All Email Links
                              Rusty Hall

                              Hey Sanford!

                              I really appreciate your feedback on this. This has really been a challenge and your feedback pointed me in the right direction.

                               

                              After working with Marketo support, I was told the following...

                               

                              "The problem is "https". Our tracking server only accepts http but not https. And their own domain would only use https.

                              They need allow http for their brand domain.

                              Your IT team will need to allow http for the branding domain." - Marketo support

                               

                              After some digging, I learned that HSTS set on the top level domain also can be applied to subdomains through using static_sts_include_subdomains: true in the header.

                               

                              This forces all subdomains to use HTTPS which according to Marketo, is not compatible with their tracking server.

                              We are working on a plan to remove static_sts_include_subdomains: true from the top level domain and set HSTS individually for domains in the hopes that this resolves the issue.

                               

                              Marketo's engineering team is also working with us to resolve the issue.

                               

                              Here's some supporting docs which lead us to the issue.

                               

                              Why is HTTP being Changed to HTTPS?

                              nginx - HSTS exclude specific subdomain with "includeSubdomains" - Server Fault

                              Chrome: how to stop redirect from http:// to https:// - Super User

                              ssl - Ignore HSTS in browser for a subdomain - Server Fault

                               

                              Check your Domains HSTS Settings Here

                              chrome://net-internals/#hsts

                              Screen Shot 2017-08-02 at 3.21.22 PM.png

                              What Google is up to with HTTP

                              Chrome is helping kill HTTP | TechCrunch

                               

                              I'll update this post with a final report once we have a resolution and a fix incase anyone else experiences this issue.

                                • Re: Branded Domain Broke All Email Links
                                  Sanford Whiteman

                                  Support's statement is only accurate because they don't have your SSL cert.

                                   

                                  Branding domains absolutely support SSL, and yes, the way you force SSL is via HSTS. (The branding domain server will not force the redirect itself.)

                                    • Re: Branded Domain Broke All Email Links
                                      Rusty Hall

                                      Interesting!

                                       

                                      That's a much easier fix then the answer I got from support. I have to say that the way support worded it in the reply, it sounded as though HTTPS wasn't supported at all for branded domains. Thank you for clarifying Sanford.

                                       

                                      I'll reach back out to support and get them working on adding our SSL certificate which we do have on our LP domain.

                                       

                                      My IT Security team will be glad to hear that's the fix instead of allowing HTTP.

                                       

                                      Thank you again!

                                        • Re: Branded Domain Broke All Email Links
                                          Sanford Whiteman

                                          Yes, what a frustratingly wrong response to what isn't a particularly obscure question.

                                          • Re: Branded Domain Broke All Email Links
                                            Jamie Hunter

                                            Rusty,

                                            Did installing the SSL cert on your email tracking domain resolve the issue and stop the error from occurring? We're experiencing the exact same issue at the moment.


                                            Thanks,

                                            Jamie

                                              • Re: Branded Domain Broke All Email Links
                                                Rusty Hall

                                                Hi Jamie,

                                                 

                                                Yes it did, and I've been meaning to write the follow up conclusion to this post and now might be helpful for you all.

                                                 

                                                The issue began for us when google chrome started enforcing HTTPS for any domain connected with the top level HSTS set. In our case because our top level domain Armor.com has HTTPS, chrome and several other browsers automatically change any HTTP addresses to HTTPS. so HTTP://marketo.armor.com/XXXXXX became HTTPS://marketo.armor.com/XXXXXX in Chrome, Firefox and others (Safari is one of the only browsers we tested which does not enforce HTTPS based on top level domain HSTS)

                                                 

                                                If you were on the default mkto-XXXXX.com tracking domain, you might never notice the change, (as this domain doesn't utilize SSL or HSTS) however, if you use a branded tracking link with a CNAME redirect, and YourDomain.com uses HTTPS, your branded tracking subdomain Example.YourDomain.com will not work in Chrome, Firefox, and most browsers. (It should be noted that there is NO roll-back to the default tracking link)

                                                 

                                                The solution is: Contact your Marketo rep, tell them you need a secure email tracking server running a SSL certificate and the branded tracking domain you have chosen, example: marketo.armor.com pointed to your Marketo email tracking link (found in the admin panel and under email)

                                                It will take a few days to spin up the new server and get the SSL cert. installed but it will fix the issue.

                                                 

                                                One of our frustrations was first level Marketo support told me they can't do SSL certs. on email tracking servers. As Sanford Whiteman correctly points out above, this IS possible and a common practice for security minded users. We have SSL on both our landing page server and email server.

                                                 

                                                Until they can get you the SSL secured email server up, one option we used was to disable tracking on email links. (this bypasses the tracking domain) You will lose tracking on email links but the links will function. It's not ideal, but it will get you by until the new SSL secured email tracking server can be up and running.

                                                 

                                                This was a tricky issue for us to solve and it took our Dev Ops. team, Marketo Support and some excellent Community feedback to solve it. I hope this will help you get it resolved fast and give you an option in the interim.

                                                 

                                                Please feel free to reach out to me with any questions and I'll be happy to help or provide further details.

                                                 

                                                -Rusty 

                                                  • Re: Branded Domain Broke All Email Links
                                                    Jamie Hunter

                                                    Rusty,

                                                     

                                                    Thank you for your very thorough reply! This was extremely helpful for you to document and it validates the issue we've been having the last 4 months. We are actually already in the process of installing our SSL cert on our email tracking domain and already have it installed on our landing pages. It was quite a frustrating process to isolate the issue and even come to terms on a solution to fix it with Marketo Support. There was conflicting information provided from their team on whether the SSL cert would actually work, so hearing this confirmation from you makes me so happy!

                                                     

                                                    As an interim solution, we've disabled the marketo link tracking in our emails so recipients can access our links with no error messages. The negative impact is we're unable to track click link activity but it's the best case scenario until the SSL cert is installed.

                                                     

                                                    I'm looking forward to no longer having email recipients reach out letting us know the email links aren't working.

                                                     

                                                    Thank you!!

                                                    Jamie

                                                      • Re: Branded Domain Broke All Email Links
                                                        Rusty Hall

                                                        Glad I could be of some help Jamie.

                                                         

                                                        I can't imagine how frustrated you must be after 4 months! I ran into much of the same conflicting information and leaned heavily on our internal security operation team and the input here on the community to find the solution. Even then, it took some time to solve the issue.

                                                         

                                                        "As an interim solution, we've disabled the Marketo link tracking in our emails so recipients can access our links with no error messages. The negative impact is we're unable to track click link activity but it's the best case scenario until the SSL cert is installed."

                                                         

                                                        We did the same and lost analytics on links for a time, but as you say it was better than sending broken links out.

                                                         

                                                        Let me know when your new server is spun up and all is back up and functioning. We're 3 months into the new server and no issues.

                                                         

                                                        Happy to help!

                                                        -Rusty

                                                        • Re: Branded Domain Broke All Email Links
                                                          Sanford Whiteman

                                                          Of course the other thing you can do is use a built-in cert on a CDN (like Amazon CloudFront) and route your DNS there. Problem solved for pennies.

                                                           

                                                          Spent much of my day today setting clients' LP and click domains to go through CF, since we're dealing with the Chrome 62 change.