Alarming spike in form API failures

[Liliana_Cheng1]

We're using Marketo forms across our marketing site which is hosted on wpengine. In conjunction with our Marketo forms, we use the Marketo API to write the form values to Marketo (instead of embedding the form). When the Marketo API fails to respond to our site within four seconds, we trigger a non-Marketo form that writes information directly to Salesforce instead.

Additionally, we set up a report to monitor when the Marketo API times out at four seconds. When the number of failures hits the threshold of 80 failures within one hour, an alert is triggered to our developers to investigate the incident.

Since June 1, 2019 there have been reports of frequent spikes in incidents of API failures - including a 20-hour period of consistent API failures beginning yesterday which peaks around 300 failures several times within the past day:

Our team has not deployed any changes to the site in several days, so we don't know of any changes that would impact this. The Marketo Status page does not show issues with the API service in June - does anyone know what Marketo's thresholds are for reporting an API incident and what impacts the reliability of the Marketo API? Why are we experiencing an influx in unresponsiveness?

We created a support case and we're waiting to hear back from Marketo, but I wonder if other instances are seeing similar issues.

[Dennis_Keefer]

Not sure if this is relevant here, but we recently ran into the same issue.  We implemented a custom form embed on our website about 1.5 years ago.  We came into the office on Tuesday and discovered that none of the forms were loading on the website.  We were using REST primarily but were forced to use another endpoint from Marketo due to missing hidden field default values on the endpoint /rest/asset/v1/form/{id}/fields.json.  Instead we used https://[POD-ID].marketo.com/index.php/form/getForm?munchkinId=[MUNCHKIN-ID]&form=[FORM-ID]. For background see this post and suggestion from Sanford Whiteman: https://nation.marketo.com/thread/40995-how-to-get-default-value-of-a-form-field-through-the-rest-ap...

We traced the problem down to a captcha challenge when using cURL to retrieve the JSON response from the marketo.com/index.php endpoint above.  It looks like Marketo recently added a captcha challenge to all of their index.php endpoints via CloudFlare.

Marketo either needs to remove the captcha challenge or improve their REST API to expose all form data so that it is useful.  

[SanfordWhiteman]

There's no reCAPTCHA required for getForm: that's the endpoint used by all form embeds to get the form descriptor.

[Dennis_Keefer]

The captcha challenge only occurs when requesting the page via cURL.

[SanfordWhiteman]

You can forge the full client environment from the server. I bet it's a User-Agent filter, more like.

Same can happen with proxies in front of CloudFlare...  CF is largely vaporware security-wise, it's just a vendor lock-in grift IME.

[Dennis_Keefer]

I tried that but was unsuccessful. Any idea what headers I should try? It's beyond frustrating that Marketo would make a change like this, especially considering their REST API is incomplete.

[SanfordWhiteman]

I'll work on it and let you know.

[Dennis_Keefer]

Thank you! I tried adding a user agent to the CURLOPT_AGENT but that was unsuccessful.

This was Marketo's response when we brought the issue up with them:

We do not support the method in which these Marketo servers resources are being called. After a further review of these cURL commands & responses, I've found that it appears your teams developers have attempted to reverse engineer our embedded forms API & have scraped various URLs in an attempt to access Marketo server resources in a custom way. The URL that your team is trying to access is a private URL that will only process requests for Marketo servers, this is why you will not see it in any of our developer documentation as a possible way to access Marketo form data. These CloudFlare captcha requirements are our servers essentially letting your team know this is not a valid way to access those resources.

[SanfordWhiteman]

Given that, rather than hashing this out on the Nation message me and I'll share the workaround.

[SanfordWhiteman]

Please be specific about what endpoint you're using and what exact code you're using to call it and detect timeouts.

The forms save endpoints, if that's what you're  referring to, are not the same as the REST infrastructure. 

[Liliana_Cheng1]

Hi Sanford Whiteman‌, I was hoping you'd reply!

The timeout is on the MktoForms2.loadForm event for when MktoForms2.loadForm doesn't respond within four seconds.

Other relevant and important info: We've had these alerts in place in February 2019 and the team has fine tuned the alert to be triggered only when we've surpassed the threshold by a healthy amount. It is currently well past the threshold and has been that way for the past day. Our team has not deployed any changes on our side since last week.

[SanfordWhiteman]

I'd need to see the actual code, because the code itself (since there's no inherent timer/timeout event for it to use) may not be correctly written.