example.png

Growing Impact of Form Abuse

The concept of form abuse, AKA email bombing or list bombing, has been around for a long time. At a high level this is where addresses are added to your database through a form by someone other than the address owner. These attacks may not seem bothersome at first, what's a few misrouted emails? However, forms can be filled out programmatically using different methods increasing the scale of impact. At scale these kinds of form attacks can cause harm to the email recipients, junk leads to be added to a business's database, and overwhelm the systems behind the forms making them unusable or causing downtime.

 

I have identified the following distinct patterns from analyzing data sets tied to this behavior:

 

Distributed Denial of Service (DDoS) Attacks - The attacker actively works to overwhelm the systems supporting the form​. By loading data at a rate the form cannot keep up with the attacker can cause system failures that may cause downtime for providers. Historically, DDoS attacks were a primary method for disrupting computer systems on a network. Firewalls and other technologies have developed and continue to evolve to combat this kind of attack.

 

Targeting Individuals by Email Bombing - An email address is signed up to a large number of email lists through many different forms at the same time. This causes the individual to start receiving email at such a rate that they may not be able to use their email account. Even if the form operator has set up double opt-in on the form, the rate of emails received at one time typically overwhelms the address owner. This gained attention a couple years ago when Security Research, Brian Krebs, described his own list bombing attack.

 

This kind of attack can be unseen by a service provider, like Marketo, because the attack against the individual is often distributed across many different ESPs and senders. Marketo is made aware this is happening typically through blocklistings of the IP addresses sending the email. Typically these blocklistings are by Spamhaus, an entity that keeps a running list of known spamming operations to which many of the world’s largest Internet service providers (ISPs) subscribe. When Spamhaus lists IP addresses as a source of spam or other abusive mail, ISPs often stop accepting mail from those IP addresses. In this case Marketo's Email Delivery & Compliance Team will reach out to the customer and work with both Spamhaus and the customer to understand and resolve the issue. Spamhaus was instrumental in helping to resolve the attack on Brian Krebs mentioned above.

 

This type of attack seems to be made to punish individuals, as in the Brian Krebs incident, or to render a email account useless so an attacker can compromise other systems, like a customer's bank account, for example. I was sitting with a friend at a conference when this started to happen to him! He was receiving hundred of emails a minute, all he could do initially was sit there and watch the emails pile up. In his case he ended up discovering that one of his online accounts at a popular technology store had been compromised. The attacker appeared to be using the attack to prevent him from noticing the original account being compromised.

 

Delivering Spam Payloads - Another pattern observed leverages personalization in emails sent from form fill outs. In this case we see volumes and volumes of addresses added through a form that asks for details like First & Last Name. The malicious actor puts a spam payload in the form field that personalizes an email so instead of using your first name in a greeting, for example, there is a spam payload in it's place!

 

The email will be delivered with a spam payload where the First Name should be. The victim, whose form was attacked, sometimes has no idea that their content has been taken over like a zombie parasite.

 

Example of using name fields leveraged for personalization

example.png

 

We will see a variety of different spam payloads added to the field that is used for personalization, for example here is a list of similar payloads used in the First Name field

 

Screen Shot 2019-02-12 at 1.54.42 PM.png

It can be difficult for an ESP or MA, like Marketo, to identify these kinds of attacks when done successfully. The point of the attack is to take advantage of the form and the resulting personalized emails, not to take them down. So these attackers try to prevent overwhelming the form with requests, often posting an address once a minute or hour. This attack is more successful the longer this behavior goes undetected and more email is delivered. The most common pattern I have observed with this attack pattern is that addresses from Chinese ISPs are added to the form and in the field that the email is personalized with is filled with spam content in Chinese, often linking to gambling sites. This can become problematic when a database becomes bloated with these junk leads. The majority of new subscribers are coming from qq.com and other Chinese domains, and if you are not targeting China it can be easy to identify and resolve. If you are targeting China then this becomes more difficult to manage and the influx of junk leads and a form sending spam content can impact a sender's reputation at top Chinese domains reducing delivery rates to impacted domains.

 

How is Marketo dealing with this evolving issue?

Marketo employs a variety of defenses for these kinds of attacks and our efforts to prevent and identify them when they do occur is constantly evolving.

Rate limiting - Marketo monitors for and limits key patterns added to forms by time.

Block traffic by IP address - IP addresses that have been associated with abusive traffic are cataloged and blocked from filling out forms.

Block traffic by payload pattern - When Marketo starts to see common patterns in the payload added to a field used for personalization, rules can be built to ignore that activity.

Honey pot - A form field that is hidden via styling or other means. People don’t fill out form fields they don’t see but unsophisticated bots fill out all form fields, including hidden ones. If there is a value in the honey pot, Marketo won’t create a lead record.

Monitoring and Alerting to internal teams with defined mitigation actions - early warning has allowed Marketo to respond before systems are overwhelmed.

 

Additional workarounds implemented by customers:

  • Set up rules that the form only allows entries from approved geo-locations
  • Additional honey pots via forms
  • Additional validation & data cleansing using partners
  • CATPCHA via webhooks
  • Clone and replace the form when abuse is observed - The honey pots are sometimes identified by more sophisticated actors, then the form is cataloged and a script built to attack the form. If the form is being attacked clone, replace, and delete the old form. This can sometimes buy some time while other solutions are put in place because the attacker sometimes has to start over.
  • Remove the personalization from the email that is sent after the form is filled out since that may be what is attracting the abusers.

 

Because this attack vector is ever evolving, so is Marketo's approach to how to manage this abuse so there are some features on the product roadmap* are focused on strengthening form security.

 

*Can't commit to specific release for these features at this time, stay tuned!

5710
10
10 Comments
Dory_Viscoglio
Level 10

Kiersti Esparza​ we've had this happen before, so we've cloned and replaced the form but we continued to see the fillouts, now coming from an 'unknown' form ID. They were also coming in from a single form at a volume of ~10k/hour which exceeded the 'limits' put in place by the rate limiting you mentioned. This was January of 2018 so maybe things are better now, but this is certainly a widespread problem that we appreciate Marketo tackling!

Vlada_Prasolova
Level 5

We've had a payload attack recently and having tried most of what you mention i can say that:

1. Honeypot. does not work. It takes seconds for someone to detect your honeypot requirement and then set their bot to post 1000s upon 1000s of forms that meet the requirement (happened in our case).

2. Changing forms is not working, bc if the bots go directly into the database via form submission link they don't care about the actual form ID

3. CAPTCHA. we've tried regular. See #2

i'm currently researching this problem in more detail but for now i see 2 possible options:

1. drop using marketo forms

2. employ a team of developers to implement some means of protecting the form from backend

both options will require considerable time, $$ and energy

SanfordWhiteman
Level 10 - Community Moderator

Re: 2: You must turn on the corresponding Treasure Chest option, then an active form ID is necessary for a post to be processed. Yes, the attacker can still get or guess a valid ID, but it's not true that any ID will work.

Re: 3: reCAPTCHA works quite well. You can't expect it to stop the form post completely in this environment. Instead, you tombstone all failing leads prior to any other processing, then delete them. Managing the order of operations is essential.

Honeypots are indeed a joke.

Vlada_Prasolova
Level 5

Thank you!

i did not know about the treasure chest option, this might be helpful.

As for the reCAPTCHA, it did work for us partially, but my IT department warned me that this won't solve the problem entirely

SanfordWhiteman
Level 10 - Community Moderator

but my IT department warned me that this won't solve the problem entirely

I don't know what they're referring to, because if "the" problem is automated posts that don't pass reCAPTCHA, and you are able to use reCAPTCHA for your audience, then it does solve the problem. No, it doesn't stop the leads from being initially created, but you don't get charged for leads that are deleted promptly.

Vlada_Prasolova
Level 5

well, it doesn't stop fake leads from creating and that's the main problem i was reffering to

SanfordWhiteman
Level 10 - Community Moderator

If a lead is created, validated, and deleted I don't understand how that disturbs your operating environment.

Vlada_Prasolova
Level 5

for example, we've had recent case with a webinar connected via GTW. Bc of the multiple spam submits per second the connection fell down so the real people were not able to register.

i can't say it's a disaster but obviously it's not something i'd like to solve on a Saturday night

all in all, i can live with t but i would prefer not to if there was a way

SanfordWhiteman
Level 10 - Community Moderator

Did that happen because you were automatically adding them to a program before validating the reCAPTCHA?  The idea is the reCAPTCHA validation must happen before all other campaigns -- the lead should be sorted into any other flows/programs/lists if they may have failed the test.

Vlada_Prasolova
Level 5

well, looks like we haven't validated it from Marketo

i've found your step by step guide and trying to get it up and running

thank you for this!

i should say you are 100000000000% more helpful and knowledgeable that marketo support