Why are we losing 70% or more of form submissions (Drupal, Cloudflare)

[Brent_Copeland]

We've been using marketo since the beginning of the month, and this week we just  noticed that we were not getting all the form submissions.

We were using the server side submit method noted here: http://developers.marketo.com/blog/server-side-form-post/

We have checked the rate limit and turned off high traffic forms, and that does not appear to be the issue.  Even on at the highest point, we seem to only be at 7 submits per minute, well under the 30 per minute.

We have now started using a Drupal module - Marketo MA, which works flawlessly in our development box, but not at all in production.  We're trying to track down the cause of this.

The major differences between the two servers are that the production servers are using HTTPS and behind Cloudflare.  We are going to test with cloudflare off this week and see if that is somehow the culprit.

Are there other things that anyone can think of? 

Thanks for any information!

-B

[SanfordWhiteman]

Do you have an override rule on your CDN that is caching POSTs? (This should never be done per RFC, but some proxies allow it.)

You have two different hops over which you're passing form data. At which point are you "not getting" form submissions?

Although it would be interesting to know what's causing your problem, there's no server-side form relay method that can be responsibly recommended. Server-side form post has an explicit per-minute rate limit, as you know (even if your legitimate traffic is below the limit) and API-based update has an explicit, and very low, daily limit (as well as a per-20-second limit) shared with all your other integrations.  The recommended method is to Make a Marketo Form Submission in the background from the client.

[Brent_Copeland]

We are getting submissions into drupal itself.  We even can show that the data is being collected offsite in postcatchers when we set those up.

Before Marketo we had been using Eloqua for years using the server-side method and didn't have any issues.  One of the reasons we wanted to use it.  So I don't think we have anything new caching posts.  Definitely something I'll investigate today.

Main reason we didn't want to go with the "in the background" method was it appeared to be quite a bit of work to build something to generate these forms dynamically.  After all the work going into trying to debug this issue, we may have to go that route.  Though we tried one form this method and it didn't appear to be working either.

Edit:  Just wanted to add that we are using this method: https://www.drupal.org/node/2190655#comment-9190021

Thanks for the advice.

[Brent_Copeland · Accepted solution]

We have solved this issue!  and it all came down to Type Casting.

We have some javascript that creates cookies and reads cookies and we set it to a text message when it loads the first time, then it changes to a date after that.

The field in Marketo is expecting a date or numerical value, and the text data was causing the submit to fail.  In the past implementation it would drop that data in the submit and post default instead of dropping the request completely.

We were still getting success messages back from the submission.  Would be great if it would either fail or submit everything but miscast data, so you don't lose anything.  Alerting you in the admin that it's happening would be great as well.

EDIT:

I wanted to add that you can validate this by using soap with the submission data, and you will get a db error back from marketo at that point.

[SanfordWhiteman]

Not all type problems cause the entire form payload to be discarded, but some do. It doesn't depend on lossiness or exact data type match, but on whether a parsing error is thrown as opposed to silent coercion. Date parsing is likely to create an error, for example.  Boolean parsing is not.

The endpoint can't fail based on data validation, because it is essentially just a journal at that point.

[Brent_Copeland]

Sending the same form submission via soap, we were returned this error msg: Lead sync failedmktServiceExceptionsyncLead operation failed: mktDbException [PropelException]; msg = Unable to parse value of [datetime_value] as date/time

I'm assuming this is coming from the Marketo end, and not adding the lead record because it doesn't fit their db schema.

[SanfordWhiteman]

SOAP is strongly typed within the schema itself.  I'm talking about the forms endpoint, which only inserts/updates/casts/discards asynchronously.

[Brent_Copeland]

Thanks for the information! 

The endpoint didn't appear to be our issue though.  It's the db not allowing the submission, and handling of that not allowing any data submission or reporting errors.

[SanfordWhiteman]

I'm explaining that the forms endpoint is not doing real-time inserts nor validation. Thus it can't kick back an error immediately.

[Brent_Copeland]

That makes complete sense.  What do you think would be a good way to handle it?  Do you think adding db error messages in the dashboard would be possible?