Marketo blocks image for Twitter card

[Crystal_Pacheco]

Hi,
I created meta og tags for various social media platforms. The code itself is correct, but I get an error from Twitter saying it can't load the image because the robots.txt will not allow the Twitterbot to crawl. 

The image size for twitter is 300px x157px  and 129kb in weight.

<meta property="og:type" content="${og-type}">
    <meta property="og:title" content="${og-title}">
    <meta property="og:url" content="${og-url}">
    <meta property="og:description" content="${og-description}">
    <meta property="og:image" content="${og-image}"> 
    <meta property="article:published_time"  content="2019-04-10" />
    <meta property="article:author"  content="${og-author}" />
    
    <meta name="twitter:card" content="summary_large_image">
    <meta name="twitter:site" content="@Company">
    <meta name="twitter:creator" content="@Company">
    <meta name="twitter:title" content="${og-title}">
    <meta name="twitter:description" content="${og-description}">
    <meta name="twitter:image" content="${og-image-twitter}">‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Error from Twitter is:
WARN: The image URL https://marketoURL.com/xx/xxx-xxx-xxx/images/twitter-image.jpg specified by the 'twitter:image' metatag may be restricted by the site's robots.txt file, which will prevent Twitter from fetching it.

I think Marketo is blocking the Twitterbot from crawling the image.

I know a solution would be to host the image externally.

Is there any way of having the image be hosted in Marketo and allowing twitter to crawl it? 

The robots.txt would look like this:
User-agent: Twitterbot
Disallow:

I'm not sure if I feel comfortable allowing the twitter bot to crawl all of Marketo. 

Any ideas?

[SanfordWhiteman]

You can customize the robots.txt for your LP domain.

[Casey_Grimes]

Hi Crystal,

By default, Marketo itself does not block images from being accessed by Twitterbot, so this is most likely being done through your overall corporate website's robots.txt vs. the Marketo subdomain. 

[Crystal_Pacheco]

Is there any danger in putting this into the robot.txt file and uploading it to marketo and redirecting the marketo robots.txt file?

The robots.txt would look like this:
User-agent: Twitterbot
Disallow:

[Crystal_Pacheco]

The current robots.txt file is:

User-agent: * Disallow: /rs/

Is that the default setting?

[Casey_Grimes]

Hi Crystal,

That is not the default setting (which is just User-agent * Allow: / )--I'm guessing this was set up like this to prevent scraping on Marketo hosting but this has the side effect of blocking legitimate bot use. I'd be more inclined to just add what specific files you don't want to be indexed rather than a blank directory block.

[SanfordWhiteman]

That looks customized -- or maybe just different from the instances I've checked based in instance age. Feel free to replace with what's right for your situation!