Googlebots Robots.txt Parser Update and Your Marketo Landing Pages

A document by KCS Integration on behalf of 3bc6ab0eb6749ef9c40eb628818ea4b1b56e7627 on Aug 28, 2019Last modified by Roxann McGlumphy on Sep 3, 2019
Version 2Show Document
  • View in full screen mode

Issue Description
Google is changing production of their robots.txt parser to an open source format and removing support for some widely used but unsupported  code:

 

In the interest of maintaining a healthy ecosystem and preparing for potential future open source releases, we're retiring all code that handles unsupported and unpublished rules (such as noindex) on September 1, 2019. For those of you who relied on the noindex indexing directive in the robots.txt file, which controls crawling, there are a number of alternative options:

 

-Link to Blog Post from Google: https://webmasters.googleblog.com/2019/07/a-note-on-unsupported-rules-in-robotstxt.html?m=1

 

Issue Resolution
This should not affect any Marketo Landing pages if the Noindex, NoFollow code has been put in the Robots meta tags as per the documentation: https://developers.marketo.com/blog/block-crawling-and-search-indexing-of-a-marketo-landing-page/

 

This will only affect your landing pages if NoIndex, NoFollow are in the robots.txt file and not in the header meta tags.

Attachments

    Outcomes