Spam Records Created (Bot attack where Honeypot doesn't help)

[Keith_Nyberg2]

Hey Community,

This past week we got hit by a bot or something similar that was creating roughly 250 bad leads in our instance every minute. We caught these getting created because our Trial server crashed out with all the activity and when I investigated, 30K records had already been create in my instance over night (all with inferred IP info from China). I immediately found a common thread (phone number) and filtered these records out of our !Entry Point smart campaigns as the existing backlog of records being processed nearly brought our instance to a halt. So I started to dig in to see what could be done to stop these records from being created.

I enabled a honeypot as defined is this sweet Perkuto article (Reduce Spam Leads with a Marketo Honeypot, thanks Perkuto!) but when testing noticed that records being created were missing the new honeypot field I had added in the "Filled Out Form" activity that is logged in MKTO, where real form submissions included this new field. (Honeypot field is called "The 5th Quarter", see images below of form submit activity). We also noticed that Munchkin was not tracking any landing page visits, nor was Google Analytics. All of this leads me to believe that these records are never on our landing pages.

So my question really relates to Marketo's form API and what is required for the API call to be successful and have a record created in our instance. What validation does Marketo require to confirm that the API request is a valid form submit vs being done via another mechanism (just the form #, instance munchkin ID, LP and referrer)? Is that enough? Because in this scenario, i'm not sure if this is something that needs to be tightened on Marketo's side or if nothing can be done at all. (if nothing can be done, what is the most sensitive parameter? I would assume the munchkin ID?

Support's advice was to unapproved the existing form and swap it with the new one. Hope the attackers get a "Form Submit Failed" notification and decide to move on. I wasn't all that thrilled with this answer as it eludes to nothing being possible to stop this from occuring again in the future. And also means my "Filled Out Form" = Trial filter is no longer fully inclusive (not a big deal, but annoying).

Anyone else have this happen? Questions? Thoughts? Comments? Just really unsure what to do next... Screenshots that show valid vs invalid submits are below.

Valid Submission:

Invalid Submit:

[SanfordWhiteman]

I enabled a honeypot as defined is this sweet Perkuto article (Reduce Spam Leads with a Marketo Honeypot, thanks Perkuto!) but when testing noticed that records being created were missing the new honeypot field I had added in the "Filled Out Form" activity that is logged in MKTO, where real form submissions included this new field. We also noticed that Munchkin was not tracking any landing page visits, nor was Google Analytics. All of this leads me to believe that these records are never on our landing pages.

Correct. There's no reason for any human to ever be viewing your LP in a browser. They only need to know how to submit to your form (which can be done via a one-time automated scrape or network recording, still doesn't need to be a human behind it).

So my question really relates to Marketo's form API and what is required for the API call to be successful and have a record created in our instance. What validation does Marketo require to confirm that the API request is a valid form submit vs being done via another mechanism (just the form #, instance munchkin ID, LP and referrer)? Is that enough?

Pod URL + Munchkin ID + Form ID. Exactly what you supply in a loadForm() call -- the server can't require you to know any more than that.

It's not so much a "validation" (there is also some data type checking of fields, date fields for example) as it is a database routing lookup. Without the required config info, there's no place to put the form data or create the Filled Out Form activity.

I recommend a ReCAPTCHA on the form. The honeypot is all too easy to fool (think about it -- all you need is a trace of one successful form post that you can then emulate 1000s of times).

[Dan_Stevens_]

Pod URL + Munchkin ID + Form ID. Exactly what you supply in a loadForm() call -- the server can't require you to know any more than that.

All three of these values can be obtained from the underlying source code.  Surprised that this issue isn't more prevalent among Marketo customers.  Aside from using ReCAPTCHA, is there nothing that can done on the Marketo side to prevent these massive form insertions?

[Keith_Nyberg2]

Thank for the response guys. I have a call scheduled with Marketo support early next week to ask them what can be done here to limit this from occuring more broadly with users. If either of you have recommended or possible solutions to this I would love to bring them up on the call. Let me know!

[SanfordWhiteman]

ReCAPTCHA is the only practical answer.

Marketo enforces a quite low rate limit of form submissions per source IP (30 per minute) but that's still high enough -- if used maliciously -- to cause problems. And it's not practical to tune this too much lower, since when people share IPs (think a large company, tradeshow floor, etc.) you need to have the headroom.

Bots are a problem faced by every website with public forms... either you limit the submission rate to the point that you're losing legit leads, or you find another way to authenticate the form posts.

[SanfordWhiteman]

All three of these values can be obtained from the underlying source code.

Yep, and just watching a form post from the Network tab would give you the same values. It's not possible to hide data that's sent over HTTP (or HTTPS) from the client itself.  If encryption routines are used in the browser it's still trivial to see what's happening.

[Grégoire_Miche2]

Hi Keith,

We had the same issue. On important point to know, is that you cannot deactivate a form... It will only stop when you DELETE the form from Marketo. So the support recommendation you mention is... useless. Once the spammer has got the form endpoint and paramaters, unapproving the landing page has not effect.

I also discovered that, in addition to limiting the number of form fillout per minute, Marketo also detects when an IP address summits to many forms and blacklists this IP address (the threshold is not public information)

The problem we faced was that the attack was changing IP address every 1000 submits, and therefore we had to delete the form to stop the attack. 260000 spam leads in about 24 hours...

-Greg

[Keith_Nyberg2]

Hey Grégoire Michel​,

Thanks for the comment, and to clarify, the recommendation was to unapprove the form, not just the landing page. Are you sure that you had to actually delete the form to actually stop the submissions? Based on what I was told about the form API's they should not be submittable if the form is not approved. I'll ask support to clarify this via some testing a report back here.

In the end, there really is nothing that could be done here and the final recommendation from support was to limit people accessing our landing pages without JS enabled. Process was described as having the page load to tell the user Java must be enabled to view the page if they have Java disabled. The hope here is to make it harder for bots to find  endpoints in the first place, but this still isn't bulletproof obviously.

[SanfordWhiteman]

the final recommendation from support was to limit people accessing our landing pages without JS enabled. Process was described as having the page load to tell the user Java must be enabled to view the page if they have Java disabled. The hope here is to make it harder for bots to find endpoints in the first place, but this still isn't bulletproof obviously.

Whoever thought this was reasonable advice should be relieved of their support duties.

[Grégoire_Miche2]

the recommendation was to unapprove the form, not just the landing page

If you ever find a way to unapprove a form, let me know

So I completely agree with Sanford, whoever gave you these recommendations should be sent back to training.

-Greg