San Jose Datacenter Email Send Interruption

Version 7

    Summary

    Customers based in our San Jose datacenter experienced a service issue causing an interruption in email sends beginning on Tuesday, September 11, 2018. No customers outside of the San Jose datacenter were impacted. This issue has been mitigated and emails are now sending properly.

     

    During the impacted timeframe, some email sends were lost and will not be delivered. This documentation is for customers based in the San Jose datacenter and provides details on how to identify the impacted campaigns.

     

    We sincerely apologize for this issue and are committed to helping our customers identify impacted campaigns and ensuring proper safeguards are put in place.

     

    Impact

    During the recovery process, we reinitialized the Active Message Queue (AMQ), which is used to process emails during the sending process.

    The restarts of the AMQ had the following impact:

    • Emails sent by Trigger campaigns during this timeframe (and those waiting in queue beforehand) could not be processed and were lost.
    • Emails that were sent by Batch campaigns during this time or just prior may show that they were sent, but do not show a delivered activity. There are two ways that these could appear:
      1. Some Batch campaigns will show every email as sent, but zero delivered. If there are zero emails showing delivered, these Batch campaigns will not send those emails.
      2. Other Batch campaigns will show a mix of some emails sent but not delivered and some that show they were successfully delivered. If there are any deliveries showing in the campaign, the rest of the emails sent should send normally.

     

     

    Timeframe

    • The email sending delays began on the morning (PDT) of Tuesday, September 11, 2018.
    • The AMQ reinitialization caused emails to not be sent during the following timeframes:
      1. September 11, 2018, 12:30 PM PDT – 1:00 PM PDT
      2. September 12, 2018, 9:15 AM PDT – 9:50 AM PDT
    • The problem was mitigated at approximately 10:30 AM PDT on Wednesday, September 12, 2018.

     

     

    Identifying Impacted Campaigns

     

    Smart Lists

    There are two methods to identify which campaigns had lost emails by using Smart Lists; one for Trigger campaigns, and the other for Batch campaigns.

     

    Identifying Trigger Campaigns

    Emails that were sent by Trigger campaigns while the AMQ was being reinitialized and all emails sent by Trigger campaigns that were waiting in queue to be processed by the AMQ when it was reinitialized were lost.

     

    The Smart List below will give you a list of affected leads that had an email sent by a Trigger campaign, but the email was lost before sending.

     

    Smart List Filter Rules:   ALL filters

     

    Filter #1:

    Member of Smart Campaign

    Campaign is "<YOUR_TRIGGERED_CAMPAIGN_NAME>"

     

    Filter #2:

    Was Sent Email

    Email is "<YOUR_TRIGGERED_CAMPAIGN_EMAIL_NAME>"
    Date of Activity is 09/12/2018

    Filter #3:

    Not Was Delivered Email
    Email is "<YOUR_TRIGGERED_CAMPAIGN_EMAIL_NAME>"
    Date of Activity is 09/12/2018

    Filter #4:

    Not Email Bounced Soft
    Email is "<YOUR_TRIGGERED_CAMPAIGN_EMAIL_NAME>"
    Date of Activity is 09/12/2018

    Filter #5:

    Not Email Bounced
    Email is "<YOUR_TRIGGERED_CAMPAIGN_EMAIL_NAME>"
    Date of Activity is 09/12/2018

     

    Once complete, your Smart List will look like this:

     

    Trigger Campaign Identification SL.png

     

     

     

    Identifying Batch Campaigns

    Emails that were sent by Batch campaigns while the AMQ was being reinitialized, or just prior, may show that they were sent, but do not show a delivered activity. There are two ways that these could appear:

    1. Some Batch campaigns will show a mix of some emails sent but not delivered, and then some that show they were successfully delivered. If there are any deliveries showing in the campaign, the rest of the emails sent should send as normal and these campaigns were not impacted.
    2. Other Batch campaigns will show every email as sent, but zero delivered. If there are zero emails showing delivered, these Batch campaigns will not send those emails. These are the Batch campaigns that were impacted.

     

    The Smart List below will show you how to identify whether a specific Batch campaign was impacted.

     

    Advanced Smart List Filter Rules: 1 and 2 and (3 or 4 or 5)

    Filter #1:

    Member of Smart Campaign

    Campaign is "<YOUR_BATCH_CAMPAIGN_NAME>"

    Filter #2:

    Was Sent Email
    Email is "<YOUR_BATCH_CAMPAIGN_EMAIL>"
    Date of Activity is 09/12/2018

    Filter #3:

    Was Delivered Email
    Email is "<YOUR_BATCH_CAMPAIGN_EMAIL>"
    Date of Activity is 09/12/2018


    Filter #4:

    Email Bounced Soft
    Email is "<YOUR_BATCH_CAMPAIGN_EMAIL>"
    Date of Activity is 09/12/2018

    Filter #5:

    Email Bounced
    Email is "<YOUR_BATCH_CAMPAIGN_EMAIL>"
    Date of Activity is 09/12/2018

     

    Once complete, your Smart List will look like this:

     

    Batch Campaign Identification SL.png

     

     

    Date Constraint.png

     

    Campaign Email Results

    The Summary page in Smart Campaigns has a tab called Email that shows the results of any email sent by that campaign.

     

    campaign email tab.png

     

     

    When trying to identify campaigns that were impacted, you will be looking for campaigns that had emails sent but had zero emails delivered or bounced. All emails will be in Pending. Here’s an example of what to look for:

     

    Campaign Email Results.png

     

    If you see email results showing that all emails that were sent are in pending, this is a Batch campaign that was impacted. No emails from this campaign will be sent.

     

     

    Email Performance Report

    The Email Performance Report will show you the stats for how your emails have performed. When trying to identify impacted campaigns, check an Email Performance Report for emails on the specific date you need. If you see the same behavior of a large volume of emails sent but all of them in Pending, you’ll be able to track which email sends were impacted.

     

    Email Performance Report.png

     

    Performance report note.png

     

    Additional Help

    If you would like any additional help identifying impacted campaigns, please Contact Marketo Support