Whitelisting Crawl Tools without removing robots.txt?

[Anonymous]

Hi there,

We use a tool called SiteImprove for our main site, and want to add our Marketo landing pages for improvements. But since our pages are non-crawlable to search engines, the tool doesn't recognize them as a verified group of pages.

Has anyone been able to whitelist certain crawling tools for Marketo landing pages?

[SanfordWhiteman]

Sounds like you don't need to remove robots.txt: you need to customize it so SiteImprove's User-Agent isn't disallowed.