SOLVED

Google robots.txt Tester

Go to solution
Pete_Jones
Level 2

Google robots.txt Tester

We have several pages on our website that use Marketo forms. Google's robot.txt testers are not seeing the Marketo form data and have treated the blocked resource to impact indexing of the pages. They say all we need to do is verify ownership of the blocked resource, our Marketo forms. I'm curious if there are any best practices for uploading the HTML file to our Marketo instance and displaying it for Google to see. The domain where we display our files loaded to Marketo and the Marketo domain we use to link to the forms are two separate domains so Google cannot verify them.

Tags (1)
1 ACCEPTED SOLUTION

Accepted Solutions
SanfordWhiteman
Level 10 - Community Moderator

Re: Google robots.txt Tester

You haven't changed your form embed code. You need to change both references to //app-ab13.marketo.com to your now-SSL-protected Marketo LP domain.

View solution in original post

19 REPLIES 19
SanfordWhiteman
Level 10 - Community Moderator

Re: Google robots.txt Tester

I'm curious if there are any best practices for uploading the HTML file to our Marketo instance and displaying it for Google to see.

That question doesn't make sense.

You can manage your Marketo LP domain's robots.txt like so: https://blog.teknkl.com/customizing-the-robots-txt-for-marketo-lps/

Pete_Jones
Level 2

Re: Google robots.txt Tester

I am not interested in managing the Marketo LP domains. I am interested in providing verification of ownership of the Marketo forms I have embedded on my website. Google is not seeing the forms and as a result, it is blocking the robot.txt file from completing the crawl. One way to verify ownership in Google Search Console is to upload the HTML file to Marketo and display so Google can see it and complete the crawl. I am curious if that has been done and if there are best practices available.

SanfordWhiteman
Level 10 - Community Moderator

Re: Google robots.txt Tester

You create an LP with the verification HTML as the content.

There's no other practice for creating a page at the root of your Marketo LP domain.

Pete_Jones
Level 2

Re: Google robots.txt Tester

Thanks, but Google is seeing the Marketo form app and our domain we use for our LP differently. So, to verify the HTML we need to domain structure of the uploaded HTML file in Marketo to mimic that of the Marketo app domain. I'll just have to pursue another means of verification.

SanfordWhiteman
Level 10 - Community Moderator

Re: Google robots.txt Tester

You're not providing a lot of detail (not sure why).

Are you loading your forms, and the forms2.min.js library, from app-nnnn.marketo.com or from your Marketo LP domain?  You should be using the latter.

Pete_Jones
Level 2

Re: Google robots.txt Tester

Well, that may be the issue. Our Marketo forms use app-xxxx.marketo.com as the domain in the embedded form on our website. Should that be the same domain that we use for our LP's?

SanfordWhiteman
Level 10 - Community Moderator

Re: Google robots.txt Tester

You should load from one of your Marketo LP domains, yes.

Note if your website runs over SSL then your Marketo LP domain needs to support SSL as well (that's why this setup is not the default in the embed code).

Pete_Jones
Level 2

Re: Google robots.txt Tester

Thank you Sanford. We're working with Marketo to secure our ssl cert. I'll focus on this once we've confirmed that has been completed.

Pete_Jones
Level 2

Re: Google robots.txt Tester

Sanford. Okay, we have the SSL cert added to the Marketo domain now but still seeing this issue. For example, the domain below. When Google indexes the page with their robots they do not show the form and actally, the form is Blocked in the Google Search Console. Thoughts?

Parachurch Organization Program | GuideOne Insurance