Marketo Forms are "Blocked Resources" (blocked by Google Robots for search crawling)

[Anonymous]

Hi--In my Google Webmaster Tools,  I see that all of the pages with our Marketo forms are blocked by Google Robots. This is creating a long list of "blocked resources" for our website.

How do I resolve this issue? Does this have to do with the CNAMES, SPF or DKIM records set-up by my developer? Anyone else have this issue?

Thank you!

[SanfordWhiteman]

Do you have a custom robots.txt?  What is your LP domain (assuming you're talking about Marketo forms on Marketo-hosted LPs)?

[Anonymous]

We're having the same issue. Our robots.txt allows everything:

User-agent: *

Disallow:

Here is an example URL that appears in our 'Blocked Resources' list in webmaster tools. There is over 2000 of these (all with the same hostname):

https://801-tfv-858.mktoresp.com/webevents/visitWebPage?_mchNc=1457913600172&_mchCn=&_mchId=801-TFV-............

[SanfordWhiteman]

That's not a form.  It's the Munchkin analytics endpoint.  And it's blocked in Marketo's robots.txt.

[Anonymous]

Thanks Sanford.

Do you know if there is a way to exclude them or ignore them? The issue is that webmaster tools is telling us that there are 2194 blocked resources, but when we dive into the list it only shows us the 'Top 1000 results', which all happen to be under this hostname. The same thing happens when we download/export it.

So we can't tell if there are other issues on our own site.

[SanfordWhiteman]

Offhand, I don't know of a way, without having a global "nofollow" on the page (which would exclude other links from being followed as well).  EDIT: Even "nofollow" can't cure this. The link is still considered a local asset.

See, you definitely don't want your tracking pixel crawled by bots, which is why it's good that it's blocked.  On the other hand, I can understand the cosmetic problem it causes in the WT console. And the fact that the session information in the URL is different whenever the page gets recrawled means you get a lot of these log lines.

I wonder if Marketo could reduce this to one log line by adding parameter exclusions in their own WT console --

-- but I wouldn't be too hopeful about a perfect solution given the way Munchkin currently works, and the frankly unreasonable way that Googlebot treats included libraries (i.e. not letting you click "This is a non-display-affecting script, exclude from crawling" or something like that).  Maybe someone else has worked on this and will chime in.