Alarming spike in form API failures

Liliana_Cheng1
Level 4

Alarming spike in form API failures

We're using Marketo forms across our marketing site which is hosted on wpengine. In conjunction with our Marketo forms, we use the Marketo API to write the form values to Marketo (instead of embedding the form). When the Marketo API fails to respond to our site within four seconds, we trigger a non-Marketo form that writes information directly to Salesforce instead.

Additionally, we set up a report to monitor when the Marketo API times out at four seconds. When the number of failures hits the threshold of 80 failures within one hour, an alert is triggered to our developers to investigate the incident.

Since June 1, 2019 there have been reports of frequent spikes in incidents of API failures - including a 20-hour period of consistent API failures beginning yesterday which peaks around 300 failures several times within the past day:

Screen Shot 2019-06-26 at 10.52.26 AM.png

Our team has not deployed any changes to the site in several days, so we don't know of any changes that would impact this. The Marketo Status page does not show issues with the API service in June - does anyone know what Marketo's thresholds are for reporting an API incident and what impacts the reliability of the Marketo API? Why are we experiencing an influx in unresponsiveness?

We created a support case and we're waiting to hear back from Marketo, but I wonder if other instances are seeing similar issues.

11 REPLIES 11
SanfordWhiteman
Level 10 - Community Moderator

Re: Alarming spike in form API failures

Please be specific about what endpoint you're using and what exact code you're using to call it and detect timeouts.

The forms save endpoints, if that's what you're  referring to, are not the same as the REST infrastructure. 

Liliana_Cheng1
Level 4

Re: Alarming spike in form API failures

Hi Sanford Whiteman‌, I was hoping you'd reply!

The timeout is on the MktoForms2.loadForm event for when MktoForms2.loadForm doesn't respond within four seconds.

Other relevant and important info: We've had these alerts in place in February 2019 and the team has fine tuned the alert to be triggered only when we've surpassed the threshold by a healthy amount. It is currently well past the threshold and has been that way for the past day. Our team has not deployed any changes on our side since last week.

SanfordWhiteman
Level 10 - Community Moderator

Re: Alarming spike in form API failures

I'd need to see the actual code, because the code itself (since there's no inherent timer/timeout event for it to use) may not be correctly written.

Dennis_Keefer
Level 2

Re: Alarming spike in form API failures

Not sure if this is relevant here, but we recently ran into the same issue.  We implemented a custom form embed on our website about 1.5 years ago.  We came into the office on Tuesday and discovered that none of the forms were loading on the website.  We were using REST primarily but were forced to use another endpoint from Marketo due to missing hidden field default values on the endpoint /rest/asset/v1/form/{id}/fields.json.  Instead we used https://[POD-ID].marketo.com/index.php/form/getForm?munchkinId=[MUNCHKIN-ID]&form=[FORM-ID]. For background see this post and suggestion from Sanford Whitemanhttps://nation.marketo.com/thread/40995-how-to-get-default-value-of-a-form-field-through-the-rest-ap...

We traced the problem down to a captcha challenge when using cURL to retrieve the JSON response from the marketo.com/index.php endpoint above.  It looks like Marketo recently added a captcha challenge to all of their index.php endpoints via CloudFlare.

Marketo either needs to remove the captcha challenge or improve their REST API to expose all form data so that it is useful.  

SanfordWhiteman
Level 10 - Community Moderator

Re: Alarming spike in form API failures

There's no reCAPTCHA required for getForm: that's the endpoint used by all form embeds to get the form descriptor.

Dennis_Keefer
Level 2

Re: Alarming spike in form API failures

The captcha challenge only occurs when requesting the page via cURL.

SanfordWhiteman
Level 10 - Community Moderator

Re: Alarming spike in form API failures

You can forge the full client environment from the server. I bet it's a User-Agent filter, more like.

Same can happen with proxies in front of CloudFlare...  CF is largely vaporware security-wise, it's just a vendor lock-in grift IME.

Dennis_Keefer
Level 2

Re: Alarming spike in form API failures

I tried that but was unsuccessful. Any idea what headers I should try? It's beyond frustrating that Marketo would make a change like this, especially considering their REST API is incomplete.

SanfordWhiteman
Level 10 - Community Moderator

Re: Alarming spike in form API failures

I'll work on it and let you know.