SOLVED

Email Outage?

Go to solution
Nora_Slattery3
Level 4

Email Outage?

Hi guys!

I have logged a support ticket about this but being a weekend, I'm hoping the community would be able to get back to me sooner.

A client contacted me Friday with a Marketo Program that didn't appear to be working.

On Friday afternoon (Australian Time) I was testied the program but was not receiving emails to one email address/domain (it was actually my client's domain) the smart campaign was saying it was sent but nothing to follow (i.e delivered/bounced).

I then tested with 2 email addresses on my  own work domain and both of those were blocked as spam and the lead in MKTO changed to Email suspended.

I then created a new hotmail (@outlook.com) account to test and none of those emails made it through until 9am Saturday when I received all emails that were sent to my @outlook account but my work email addresses are still set to email suspended in mkto.

Does anyone know if this is standard behaviour? Was there an outage on Friday? Has anyone had something similar happen?

Any help would be very welcome and greatly appreciated!

Warm regards,
Nora

Tags (1)
1 ACCEPTED SOLUTION

Accepted Solutions
Kiersti_Esparz1
Level 7

Re: Email Outage?

On 6/28, Friday morning, part of Marketo's infrastructure was briefly listed with Spamhaus.  Spamhaus provides a blacklist that email networks can use to help judge the reputation of an email.  Email providers can use a listing with a blacklist like Spamhaus to help determine whether they should block mail delivery.  

The listing was triggered by a customer mailing to a high volume of Spamhaus spamtrap email addresses.  According to Spamhaus "spamtraps are email addresses which do not belong to real users. A spamtrap either never belonged to a real user, or did but was closed and rejected email for a significant period before being repurposed."

A listing like this is rare and Marketo reacted swiftly, resolving the listing within couple of hours.  We continue to work very closely with the customer who triggered the listing to ensure this does not happen again.   Even though they are rare, listings like this can occasionally occur in a high volume email sending environment and Marketo has alerting and monitoring set up for blacklistings and email blocking.  Marketo maintains a good working relationship with Spamhaus and other blacklist providers to quickly resolve issues if they occur and we work with customers to maintain email best practices to avoid delivery impacting events.

Some customers who were sending from the impacted infrastructure may have seen some mail delivery delayed or not delivered because of the brief listing.  If Marketo received a bounce indicating a delivery problem in most cases our servers would have retried delivering the mail for up to 24 hours.  By retrying to deliver the mail at a later time, when the listing had time to be addressed, most impacted email was delivered instead of failing out right.  The email addresses that did bounce when the listing was active are still mailable for future email campaigns.  They bounces with a Spam Block Hard Bounce which does not set the Email=Invalid flag.

If you are interested in identifying the impacted leads in your database you can create a Smart List for email campaign sent Friday, 6/28.

SMART LIST
Filter: Email Bounced
(set constraints as needed, by time range or by email)
Constraint: Category is 1
   This constraint identifies emails that bounces with a Spam Block Hard Bounce.

Once the leads are pulled you can add them to a new email campaign if you'd like to retarget them.  [Leads Tab > Lead Actions > Flow Actions]

This did not impact all Marketo customers, only a portion of Marketo's infrastructure was briefly impacted.  I apologize if you were impacted by this event.
Kiersti Esparza

View solution in original post

6 REPLIES 6
Anonymous
Not applicable

Re: Email Outage?

This is not standard behavior and our email has been adversly impacted as well.  We sent out an email at 10:44 AM EDT on 9/28 to our customers.  Some had a delivery confirmation within a minute of sending the email including one of our internal users, but many of the email confirmations didn't occur until after 4:00 PM EDT and as of 9/30 @ 12:30 AM EDT, the delivery rate is ony 87.5% when emails to this segment typically have delivery rates over 99%.   I would like to know what is going on as well.
Anonymous
Not applicable

Re: Email Outage?

I was getting 404 error this morning when I  tried to login. They may be doing some server maintenance. I wish they would tell us beforehand.
Anonymous
Not applicable

Re: Email Outage?

Hi all,

I will follow up with support on this and get back to you asap. Sorry for the inconvenience. and thank you for your patience.

Best,
Liz
Anonymous
Not applicable

Re: Email Outage?

@hitoshi, the login error does not have to do with this issue, so please log a support ticket if you are still having trouble. And let me know if I can assist. 

Thanks
Liz
Kiersti_Esparz1
Level 7

Re: Email Outage?

On 6/28, Friday morning, part of Marketo's infrastructure was briefly listed with Spamhaus.  Spamhaus provides a blacklist that email networks can use to help judge the reputation of an email.  Email providers can use a listing with a blacklist like Spamhaus to help determine whether they should block mail delivery.  

The listing was triggered by a customer mailing to a high volume of Spamhaus spamtrap email addresses.  According to Spamhaus "spamtraps are email addresses which do not belong to real users. A spamtrap either never belonged to a real user, or did but was closed and rejected email for a significant period before being repurposed."

A listing like this is rare and Marketo reacted swiftly, resolving the listing within couple of hours.  We continue to work very closely with the customer who triggered the listing to ensure this does not happen again.   Even though they are rare, listings like this can occasionally occur in a high volume email sending environment and Marketo has alerting and monitoring set up for blacklistings and email blocking.  Marketo maintains a good working relationship with Spamhaus and other blacklist providers to quickly resolve issues if they occur and we work with customers to maintain email best practices to avoid delivery impacting events.

Some customers who were sending from the impacted infrastructure may have seen some mail delivery delayed or not delivered because of the brief listing.  If Marketo received a bounce indicating a delivery problem in most cases our servers would have retried delivering the mail for up to 24 hours.  By retrying to deliver the mail at a later time, when the listing had time to be addressed, most impacted email was delivered instead of failing out right.  The email addresses that did bounce when the listing was active are still mailable for future email campaigns.  They bounces with a Spam Block Hard Bounce which does not set the Email=Invalid flag.

If you are interested in identifying the impacted leads in your database you can create a Smart List for email campaign sent Friday, 6/28.

SMART LIST
Filter: Email Bounced
(set constraints as needed, by time range or by email)
Constraint: Category is 1
   This constraint identifies emails that bounces with a Spam Block Hard Bounce.

Once the leads are pulled you can add them to a new email campaign if you'd like to retarget them.  [Leads Tab > Lead Actions > Flow Actions]

This did not impact all Marketo customers, only a portion of Marketo's infrastructure was briefly impacted.  I apologize if you were impacted by this event.
Kiersti Esparza
Nora_Slattery3
Level 4

Re: Email Outage?

Thank you for getting back to me Liz and Kiersti!