Google robots.txt Tester

[Pete_Jones]

We have several pages on our website that use Marketo forms. Google's robot.txt testers are not seeing the Marketo form data and have treated the blocked resource to impact indexing of the pages. They say all we need to do is verify ownership of the blocked resource, our Marketo forms. I'm curious if there are any best practices for uploading the HTML file to our Marketo instance and displaying it for Google to see. The domain where we display our files loaded to Marketo and the Marketo domain we use to link to the forms are two separate domains so Google cannot verify them.

[SanfordWhiteman]

I'm curious if there are any best practices for uploading the HTML file to our Marketo instance and displaying it for Google to see.

That question doesn't make sense.

You can manage your Marketo LP domain's robots.txt like so: https://blog.teknkl.com/customizing-the-robots-txt-for-marketo-lps/

[Pete_Jones]

I am not interested in managing the Marketo LP domains. I am interested in providing verification of ownership of the Marketo forms I have embedded on my website. Google is not seeing the forms and as a result, it is blocking the robot.txt file from completing the crawl. One way to verify ownership in Google Search Console is to upload the HTML file to Marketo and display so Google can see it and complete the crawl. I am curious if that has been done and if there are best practices available.

[SanfordWhiteman]

You create an LP with the verification HTML as the content.

There's no other practice for creating a page at the root of your Marketo LP domain.

[Pete_Jones]

Thanks, but Google is seeing the Marketo form app and our domain we use for our LP differently. So, to verify the HTML we need to domain structure of the uploaded HTML file in Marketo to mimic that of the Marketo app domain. I'll just have to pursue another means of verification.

[SanfordWhiteman]

You're not providing a lot of detail (not sure why).

Are you loading your forms, and the forms2.min.js library, from app-nnnn.marketo.com or from your Marketo LP domain?  You should be using the latter.

[Pete_Jones]

Well, that may be the issue. Our Marketo forms use app-xxxx.marketo.com as the domain in the embedded form on our website. Should that be the same domain that we use for our LP's?

[SanfordWhiteman]

You should load from one of your Marketo LP domains, yes.

Note if your website runs over SSL then your Marketo LP domain needs to support SSL as well (that's why this setup is not the default in the embed code).

[Pete_Jones]

Thank you Sanford. We're working with Marketo to secure our ssl cert. I'll focus on this once we've confirmed that has been completed.

[Pete_Jones]

Sanford. Okay, we have the SSL cert added to the Marketo domain now but still seeing this issue. For example, the domain below. When Google indexes the page with their robots they do not show the form and actally, the form is Blocked in the Google Search Console. Thoughts?

Parachurch Organization Program | GuideOne Insurance

[SanfordWhiteman · Accepted solution]

You haven't changed your form embed code. You need to change both references to //app-ab13.marketo.com to your now-SSL-protected Marketo LP domain.

[Pete_Jones]

Well, the embed code in the Marketo system is the exact same. Do I need to recreate the form to get the new code??

[SanfordWhiteman]

The code offered in the UI won't change. You must make those 2 edits to the domain.

[Pete_Jones]

Thank you Sanford. But, forgive me, how do I figure out what our new ssl domain is if the system is populating the above domain with new forms, etc?

[SanfordWhiteman]

It's your primary LP domain! (In Admin.)

[Pete_Jones]

Our primary lp domain is https://go.guideone.com. If I hover over our Marketing Activities icon in the Marketo main screen the https://app-ab13.marketo.com/#MA domain pops up in the lower left for the Marketing Activities section. That must still be our domain.

[SanfordWhiteman]

... the https://app-ab13.marketo.com/#MA domain pops up in the lower left for the Marketing Activities section. That must still be our domain.

Nope, the URL of the Marketo UI will not change in any case. You must change the embed code as recommended.

[Pete_Jones]

Okay. Thanks for hanging with me Sanford. I have changed the domain in the code to the go.guideone.com domain I stated above. The form continues to populate, so that is a good thing. But, the first render I did with the Google Search Console still showed the old form domain and stated it was blocked. Guess I'll try it again tomorrow to see if they're able to see the first domain and ideally actually see the form when the robots crawl.

[SanfordWhiteman]

Yes, wait for the next crawl.

[Pete_Jones]

Wonderful! It worked. Thanks so much Sanford!