December 14, 2020 Global Gmail Incorrect Hard Bounce Self-Help Document

[j_briscoe]

Issue Description 

A global issue at Gmail resulted in emails sent to valid Gmail email addresses being incorrectly hard bounced as Invalid Email Addresses by Gmail servers.  Per standard bounce handling logic, Marketo Engage automatically set these leads’ “Email Invalid” field to be true along with “Email Invalid Cause” with the error message from Google “550-5.1.1 The email account that you tried to reach does not exist.” 

You will not be able to send emails to these leads via Marketo Engage until the “Email Invalid” field is set back to false as documented here.   

Impact Timeframe 

Although Google stated the Email Bounced/Invalid issue occurred on 12/14 between 4:20 pm PST to 5:30 pm PST on this page, we noticed the issue started as early as 12/14 at 6:00 am PST until 12/15 at 3:51 pm PST.  Updates were posted here.  The majority of the bounces occurred on 12/14 between 2:00 pm PST and 5:00 pm PST and again between 1:00 pm PST and 4:00 pm PST on 12/15.  We saw slightly elevated bounces outside of this timeframe. 

How to correct the affected leads? 

The leads would have an “Email Bounced” activity as shown in the screenshot below with SubCategory=2003 and Category=2. 

 

In addition, you would see the following “Change Data Value” activities that were system triggered due to the Bounce Event.  The “Email Invalid” field was set to true and the “Email Invalid Cause” was set to the Bounce Error Message from Gmail. 

Since we have the affected timeframe and the Category/Subcategory values, we can construct a Smart Campaign similar to the screenshot below in order to identify and correct the affected leads.  

Please be aware of the following important notes: 

• It’s possible the smart list would qualify truly “Email Invalid” leads since the Bounce Error Message from Google is the same for both legitimate and illegitimate bounce.  We expect this to be minimally risky since the truly “Email Invalid” leads will bounce again the next time an email is being sent to them. 
• Since we can’t use a precise timestamp in the Date of Activity Constraint, you could potentially qualify leads that were not affected where some leads were truly “Email Invalid”.  But same as above, the emails to these leads will be bounced again. 

 

In addition to the above correction logic, you might have customized trigger campaigns that add leads to an Exclusion List and other logic that needs to be reverted.  Please add the necessary flows to correct those items if you have any custom logic. 

[Jon_Wu]

Thank you for writing this up! Really helps make damage control easier on the Marketo side. I've been going through all our email sending platforms all day trying to clear out denylists, and this is the easiest one.

One small tip - you may not want to filter by gmail.com. This issue affected Google Workspace (formerly G Suite, Google Apps) domains too, so it's impossible to know exactly what email addresses would be affected. I'd rather unflag a few that will rebounce vs never send a real person an email b/c they were not on the list, but everybody's requirements may vary.

As mentioned below, Marketo only observed issues with gmail.com addresses. Google's issue summary also confirms that other domains should not have been affected.

[Carmi_Lopez-Jon]

Thanks for the feedback @Jon_Wu.  This is an important consideration.  In fact, we analyzed our bounce logs and we saw minimal impact to Google hosted domains outside of Gmail.com.  As a result, we were intentional in not recommending inclusion beyond Gmail.com.  The bounce codes cited, Cat 2, Subcategory 2003 will apply broadly to many domains, beyond Gmail and Google hosted domains.  The exclusion of a filter for "@gmail.com" domains could result in resurrecting bounces beyond the impacted Gmail bounces and would be discouraged as there could be a negative reputational impact.  Cheers and Happy Holidays!  

Carmi

[MitchellH]

Thanks for posting this - I had a campaign go out this week that had a massive 65% bounce rate and was wondering what on earth caused it. This explains it. I have followed the steps here and will re-send the campaign to those who bounced.

[Lucas_Metherall]

Thank you for the communication, very helpful and I would actually have missed this.

[David_Klein]

Just a note, the above smart campaign accounts for records that were hard bounced. But if your marketo instance has duplicates by email and one of the duplicates bounced it will not reverse the hard bounce propagation that marketo applies. For example, let's suppose you have 2 email addresses: marketo id 1 with email address of a@b.com and marketo id 2 with email address of a@b.com. When bounce occurred Marketo propagates hard bounce to both records. But smart campaign above will only capture the record that actually has a bounce activity associated with it. So you will also need to reverse Marketo's system process of propagating  hard bounces.