Alarming spike in form API failures

[Liliana_Cheng1]

We're using Marketo forms across our marketing site which is hosted on wpengine. In conjunction with our Marketo forms, we use the Marketo API to write the form values to Marketo (instead of embedding the form). When the Marketo API fails to respond to our site within four seconds, we trigger a non-Marketo form that writes information directly to Salesforce instead.

Additionally, we set up a report to monitor when the Marketo API times out at four seconds. When the number of failures hits the threshold of 80 failures within one hour, an alert is triggered to our developers to investigate the incident.

Since June 1, 2019 there have been reports of frequent spikes in incidents of API failures - including a 20-hour period of consistent API failures beginning yesterday which peaks around 300 failures several times within the past day:

Our team has not deployed any changes to the site in several days, so we don't know of any changes that would impact this. The Marketo Status page does not show issues with the API service in June - does anyone know what Marketo's thresholds are for reporting an API incident and what impacts the reliability of the Marketo API? Why are we experiencing an influx in unresponsiveness?

We created a support case and we're waiting to hear back from Marketo, but I wonder if other instances are seeing similar issues.

[SanfordWhiteman]

Please be specific about what endpoint you're using and what exact code you're using to call it and detect timeouts.

The forms save endpoints, if that's what you're  referring to, are not the same as the REST infrastructure. 

[Liliana_Cheng1]

Hi Sanford Whiteman‌, I was hoping you'd reply!

The timeout is on the MktoForms2.loadForm event for when MktoForms2.loadForm doesn't respond within four seconds.

Other relevant and important info: We've had these alerts in place in February 2019 and the team has fine tuned the alert to be triggered only when we've surpassed the threshold by a healthy amount. It is currently well past the threshold and has been that way for the past day. Our team has not deployed any changes on our side since last week.

[SanfordWhiteman]

I'd need to see the actual code, because the code itself (since there's no inherent timer/timeout event for it to use) may not be correctly written.

[Dennis_Keefer]

Not sure if this is relevant here, but we recently ran into the same issue.  We implemented a custom form embed on our website about 1.5 years ago.  We came into the office on Tuesday and discovered that none of the forms were loading on the website.  We were using REST primarily but were forced to use another endpoint from Marketo due to missing hidden field default values on the endpoint /rest/asset/v1/form/{id}/fields.json.  Instead we used https://[POD-ID].marketo.com/index.php/form/getForm?munchkinId=[MUNCHKIN-ID]&form=[FORM-ID]. For background see this post and suggestion from Sanford Whiteman: https://nation.marketo.com/thread/40995-how-to-get-default-value-of-a-form-field-through-the-rest-ap...

We traced the problem down to a captcha challenge when using cURL to retrieve the JSON response from the marketo.com/index.php endpoint above.  It looks like Marketo recently added a captcha challenge to all of their index.php endpoints via CloudFlare.

Marketo either needs to remove the captcha challenge or improve their REST API to expose all form data so that it is useful.  

[SanfordWhiteman]

There's no reCAPTCHA required for getForm: that's the endpoint used by all form embeds to get the form descriptor.

[Dennis_Keefer]

The captcha challenge only occurs when requesting the page via cURL.

[SanfordWhiteman]

You can forge the full client environment from the server. I bet it's a User-Agent filter, more like.

Same can happen with proxies in front of CloudFlare...  CF is largely vaporware security-wise, it's just a vendor lock-in grift IME.

[Dennis_Keefer]

I tried that but was unsuccessful. Any idea what headers I should try? It's beyond frustrating that Marketo would make a change like this, especially considering their REST API is incomplete.

[SanfordWhiteman]

I'll work on it and let you know.