SOLVED

Re: Block search engine crawlers to access downloadable assets (pdfs, etc)

Go to solution
sahil_1512
Level 2

Block search engine crawlers to access downloadable assets (pdfs, etc)

Hi Everyone,

Is there any way I can hide/or block search engine crawlers to access the assets (PDFs, white papers, etc) within the Marketo platform?

Since the assets are gated via Marketo forms...so I don't want anyone to get the direct access to them.

Cheers,
Sahil
1 ACCEPTED SOLUTION

Accepted Solutions
SanfordWhiteman
Level 10 - Community Moderator

Re: Block search engine crawlers to access downloadable assets (pdfs, etc)


Is there any way I can hide/or block search engine crawlers to access the assets (PDFs, white papers, etc) within the Marketo platform?

Yes, set up your robots.txt accordingly, upload it to Design Studio, and create a Redirect Rule so it appears as https://pages.example.com/robots.txt.

View solution in original post

2 REPLIES 2
Darshil_Shah1
Level 10 - Community Advisor + Adobe Champion

Re: Block search engine crawlers to access downloadable assets (pdfs, etc)

Well, people will be able to access the files if they have direct URLs to your assets (e.g., someone who filled out the form sent them); however, since your assets are gated behind Marketo forms, crawlers and search engines won't be able to access the content itself. The crawlers follow links to discover and index web pages. Crawlers won’t submit the form, so they won't be able to access the gated content behind the form.

SanfordWhiteman
Level 10 - Community Moderator

Re: Block search engine crawlers to access downloadable assets (pdfs, etc)


Is there any way I can hide/or block search engine crawlers to access the assets (PDFs, white papers, etc) within the Marketo platform?

Yes, set up your robots.txt accordingly, upload it to Design Studio, and create a Redirect Rule so it appears as https://pages.example.com/robots.txt.