Whitelisting Crawl Tools without removing robots.txt?

Anonymous
Not applicable

Whitelisting Crawl Tools without removing robots.txt?

Hi there,

We use a tool called SiteImprove for our main site, and want to add our Marketo landing pages for improvements. But since our pages are non-crawlable to search engines, the tool doesn't recognize them as a verified group of pages.

Has anyone been able to whitelist certain crawling tools for Marketo landing pages?

1 REPLY 1
SanfordWhiteman
Level 10 - Community Moderator

Re: Whitelisting Crawl Tools without removing robots.txt?

Sounds like you don't need to remove robots.txt: you need to customize it so SiteImprove's User-Agent isn't disallowed.