Block search engine crawlers to access downloadable assets (pdfs, etc)

[sahil_1512]

Hi Everyone,

Is there any way I can hide/or block search engine crawlers to access the assets (PDFs, white papers, etc) within the Marketo platform?

Since the assets are gated via Marketo forms...so I don't want anyone to get the direct access to them.

[SanfordWhiteman · Accepted solution]

---

Is there any way I can hide/or block search engine crawlers to access the assets (PDFs, white papers, etc) within the Marketo platform?

---

Yes, set up your robots.txt accordingly, upload it to Design Studio, and create a Redirect Rule so it appears as pages.example.com/robots.txt.

[Darshil_Shah1]

Well, people will be able to access the files if they have direct URLs to your assets (e.g., someone who filled out the form sent them); however, since your assets are gated behind Marketo forms, crawlers and search engines won't be able to access the content itself. The crawlers follow links to discover and index web pages. Crawlers won’t submit the form, so they won't be able to access the gated content behind the form.