San Jose Datacenter Email Send Interruption

Document created by Mike Reynolds Employee on Sep 13, 2018Last modified by Mike Reynolds Employee on Sep 13, 2018
Version 7Show Document
  • View in full screen mode

Summary

Customers based in our San Jose datacenter experienced a service issue causing an interruption in email sends beginning on Tuesday, September 11, 2018. No customers outside of the San Jose datacenter were impacted. This issue has been mitigated and emails are now sending properly.

 

During the impacted timeframe, some email sends were lost and will not be delivered. This documentation is for customers based in the San Jose datacenter and provides details on how to identify the impacted campaigns.

 

We sincerely apologize for this issue and are committed to helping our customers identify impacted campaigns and ensuring proper safeguards are put in place.

 

Impact

During the recovery process, we reinitialized the Active Message Queue (AMQ), which is used to process emails during the sending process.

The restarts of the AMQ had the following impact:

  • Emails sent by Trigger campaigns during this timeframe (and those waiting in queue beforehand) could not be processed and were lost.
  • Emails that were sent by Batch campaigns during this time or just prior may show that they were sent, but do not show a delivered activity. There are two ways that these could appear:
    1. Some Batch campaigns will show every email as sent, but zero delivered. If there are zero emails showing delivered, these Batch campaigns will not send those emails.
    2. Other Batch campaigns will show a mix of some emails sent but not delivered and some that show they were successfully delivered. If there are any deliveries showing in the campaign, the rest of the emails sent should send normally.

 

 

Timeframe

  • The email sending delays began on the morning (PDT) of Tuesday, September 11, 2018.
  • The AMQ reinitialization caused emails to not be sent during the following timeframes:
    1. September 11, 2018, 12:30 PM PDT – 1:00 PM PDT
    2. September 12, 2018, 9:15 AM PDT – 9:50 AM PDT
  • The problem was mitigated at approximately 10:30 AM PDT on Wednesday, September 12, 2018.

 

 

Identifying Impacted Campaigns

 

Smart Lists

There are two methods to identify which campaigns had lost emails by using Smart Lists; one for Trigger campaigns, and the other for Batch campaigns.

 

Identifying Trigger Campaigns

Emails that were sent by Trigger campaigns while the AMQ was being reinitialized and all emails sent by Trigger campaigns that were waiting in queue to be processed by the AMQ when it was reinitialized were lost.

 

The Smart List below will give you a list of affected leads that had an email sent by a Trigger campaign, but the email was lost before sending.

 

Smart List Filter Rules:   ALL filters

 

Filter #1:

Member of Smart Campaign

Campaign is "<YOUR_TRIGGERED_CAMPAIGN_NAME>"

 

Filter #2:

Was Sent Email

Email is "<YOUR_TRIGGERED_CAMPAIGN_EMAIL_NAME>"
Date of Activity is 09/12/2018

Filter #3:

Not Was Delivered Email
Email is "<YOUR_TRIGGERED_CAMPAIGN_EMAIL_NAME>"
Date of Activity is 09/12/2018

Filter #4:

Not Email Bounced Soft
Email is "<YOUR_TRIGGERED_CAMPAIGN_EMAIL_NAME>"
Date of Activity is 09/12/2018

Filter #5:

Not Email Bounced
Email is "<YOUR_TRIGGERED_CAMPAIGN_EMAIL_NAME>"
Date of Activity is 09/12/2018

 

Once complete, your Smart List will look like this:

 

Trigger Campaign Identification SL.png

 

 

 

Identifying Batch Campaigns

Emails that were sent by Batch campaigns while the AMQ was being reinitialized, or just prior, may show that they were sent, but do not show a delivered activity. There are two ways that these could appear:

  1. Some Batch campaigns will show a mix of some emails sent but not delivered, and then some that show they were successfully delivered. If there are any deliveries showing in the campaign, the rest of the emails sent should send as normal and these campaigns were not impacted.
  2. Other Batch campaigns will show every email as sent, but zero delivered. If there are zero emails showing delivered, these Batch campaigns will not send those emails. These are the Batch campaigns that were impacted.

 

The Smart List below will show you how to identify whether a specific Batch campaign was impacted.

 

Advanced Smart List Filter Rules: 1 and 2 and (3 or 4 or 5)

Filter #1:

Member of Smart Campaign

Campaign is "<YOUR_BATCH_CAMPAIGN_NAME>"

Filter #2:

Was Sent Email
Email is "<YOUR_BATCH_CAMPAIGN_EMAIL>"
Date of Activity is 09/12/2018

Filter #3:

Was Delivered Email
Email is "<YOUR_BATCH_CAMPAIGN_EMAIL>"
Date of Activity is 09/12/2018


Filter #4:

Email Bounced Soft
Email is "<YOUR_BATCH_CAMPAIGN_EMAIL>"
Date of Activity is 09/12/2018

Filter #5:

Email Bounced
Email is "<YOUR_BATCH_CAMPAIGN_EMAIL>"
Date of Activity is 09/12/2018

 

Once complete, your Smart List will look like this:

 

Batch Campaign Identification SL.png

 

 

Date Constraint.png

 

Campaign Email Results

The Summary page in Smart Campaigns has a tab called Email that shows the results of any email sent by that campaign.

 

campaign email tab.png

 

 

When trying to identify campaigns that were impacted, you will be looking for campaigns that had emails sent but had zero emails delivered or bounced. All emails will be in Pending. Here’s an example of what to look for:

 

Campaign Email Results.png

 

If you see email results showing that all emails that were sent are in pending, this is a Batch campaign that was impacted. No emails from this campaign will be sent.

 

 

Email Performance Report

The Email Performance Report will show you the stats for how your emails have performed. When trying to identify impacted campaigns, check an Email Performance Report for emails on the specific date you need. If you see the same behavior of a large volume of emails sent but all of them in Pending, you’ll be able to track which email sends were impacted.

 

Email Performance Report.png

 

Performance report note.png

 

Additional Help

If you would like any additional help identifying impacted campaigns, please Contact Marketo Support

Attachments

    Outcomes