Summary
Customers based in our San Jose datacenter experienced a service issue causing an interruption in email sends beginning on Tuesday, September 11, 2018. No customers outside of the San Jose datacenter were impacted. This issue has been mitigated and emails are now sending properly.
During the impacted timeframe, some email sends were lost and will not be delivered. This documentation is for customers based in the San Jose datacenter and provides details on how to identify the impacted campaigns.
We sincerely apologize for this issue and are committed to helping our customers identify impacted campaigns and ensuring proper safeguards are put in place.
Impact
During the recovery process, we reinitialized the Active Message Queue (AMQ), which is used to process emails during the sending process.
The restarts of the AMQ had the following impact:
- Emails sent by Trigger campaigns during this timeframe (and those waiting in queue beforehand) could not be processed and were lost.
- Emails that were sent by Batch campaigns during this time or just prior may show that they were sent, but do not show a delivered activity. There are two ways that these could appear:
- Some Batch campaigns will show every email as sent, but zero delivered. If there are zero emails showing delivered, these Batch campaigns will not send those emails.
- Other Batch campaigns will show a mix of some emails sent but not delivered and some that show they were successfully delivered. If there are any deliveries showing in the campaign, the rest of the emails sent should send normally.
Timeframe
- The email sending delays began on the morning (PDT) of Tuesday, September 11, 2018.
- The AMQ reinitialization caused emails to not be sent during the following timeframes:
- September 11, 2018, 12:30 PM PDT – 1:00 PM PDT
- September 12, 2018, 9:15 AM PDT – 9:50 AM PDT
- The problem was mitigated at approximately 10:30 AM PDT on Wednesday, September 12, 2018.
Identifying Impacted Campaigns
Smart Lists
There are two methods to identify which campaigns had lost emails by using Smart Lists; one for Trigger campaigns, and the other for Batch campaigns.
Identifying Trigger Campaigns
Emails that were sent by Trigger campaigns while the AMQ was being reinitialized and all emails sent by Trigger campaigns that were waiting in queue to be processed by the AMQ when it was reinitialized were lost.
The Smart List below will give you a list of affected leads that had an email sent by a Trigger campaign, but the email was lost before sending.
Smart List Filter Rules: ALL filters
Filter #1:
Member of Smart Campaign
Campaign is "<YOUR_TRIGGERED_CAMPAIGN_NAME>"
Filter #2:
Was Sent Email
Email is "<YOUR_TRIGGERED_CAMPAIGN_EMAIL_NAME>"
Date of Activity is 09/12/2018
Filter #3:
Not Was Delivered Email
Email is "<YOUR_TRIGGERED_CAMPAIGN_EMAIL_NAME>"
Date of Activity is 09/12/2018
Filter #4:
Not Email Bounced Soft
Email is "<YOUR_TRIGGERED_CAMPAIGN_EMAIL_NAME>"
Date of Activity is 09/12/2018
Filter #5:
Not Email Bounced
Email is "<YOUR_TRIGGERED_CAMPAIGN_EMAIL_NAME>"
Date of Activity is 09/12/2018
Once complete, your Smart List will look like this:
Identifying Batch Campaigns
Emails that were sent by Batch campaigns while the AMQ was being reinitialized, or just prior, may show that they were sent, but do not show a delivered activity. There are two ways that these could appear:
- Some Batch campaigns will show a mix of some emails sent but not delivered, and then some that show they were successfully delivered. If there are any deliveries showing in the campaign, the rest of the emails sent should send as normal and these campaigns were not impacted.
- Other Batch campaigns will show every email as sent, but zero delivered. If there are zero emails showing delivered, these Batch campaigns will not send those emails. These are the Batch campaigns that were impacted.
The Smart List below will show you how to identify whether a specific Batch campaign was impacted.
Advanced Smart List Filter Rules: 1 and 2 and (3 or 4 or 5)
Filter #1:
Member of Smart Campaign
Campaign is "<YOUR_BATCH_CAMPAIGN_NAME>"
Filter #2:
Was Sent Email
Email is "<YOUR_BATCH_CAMPAIGN_EMAIL>"
Date of Activity is 09/12/2018
Filter #3:
Was Delivered Email
Email is "<YOUR_BATCH_CAMPAIGN_EMAIL>"
Date of Activity is 09/12/2018
Filter #4:
Email Bounced Soft
Email is "<YOUR_BATCH_CAMPAIGN_EMAIL>"
Date of Activity is 09/12/2018
Filter #5:
Email Bounced
Email is "<YOUR_BATCH_CAMPAIGN_EMAIL>"
Date of Activity is 09/12/2018
Once complete, your Smart List will look like this:
Campaign Email Results
The Summary page in Smart Campaigns has a tab called Email that shows the results of any email sent by that campaign.
When trying to identify campaigns that were impacted, you will be looking for campaigns that had emails sent but had zero emails delivered or bounced. All emails will be in Pending. Here’s an example of what to look for:
If you see email results showing that all emails that were sent are in pending, this is a Batch campaign that was impacted. No emails from this campaign will be sent.
Email Performance Report
The Email Performance Report will show you the stats for how your emails have performed. When trying to identify impacted campaigns, check an Email Performance Report for emails on the specific date you need. If you see the same behavior of a large volume of emails sent but all of them in Pending, you’ll be able to track which email sends were impacted.
Additional Help
If you would like any additional help identifying impacted campaigns, please Contact Marketo Support
.png)