This document provides an overview of the recent import service issue that impacted a subset of customers across our data centers beginning on April 24, 2019. All customers that were impacted were emailed details of the incident on May 15, 2019. If you are unsure if your instance was impacted, please reach out to Marketo support at https://support.marketo.com.
Start time:
London data center, April 24, 2019, 4:30 PM PDT
Sydney data center, April 26, 2019, 2:00 PM PDT
San Jose and Ashburn data centers, April 26, 2019, 5:30 PM PDT
How to identify what data center your instance is located in:
You can deduce the server locations by looking at the URL of your instance. The CNAME=the pod or server your instance is on.
The example above indicates that this instance is in our Ashburn data center.
The abbreviations for all our data centers are as follows.
lon = London
sn = Sydney
sj = San Jose
ab = Ashburn
End time:
Patch was pushed May 8, 2019, 4:25 PM PDT
Service affected:
During the impacted timeframe, customers may have noticed that the Email Invalid field for records was inaccurately set to False during list imports. If a record had the Email Invalid field set to True, and a list without the Email Invalid column was imported containing that record, the Email Invalid value would subsequently revert to False. Please note that your list import would not have been impacted if your list contains the Email Invalid column. No other fields were impacted by this issue, including Unsubscribe, Marketing Suspended, Blacklisted, etc. When working as expected, the Email Invalid field is set to True when a certain type of hard bounce occurs such as when the recipient doesn’t exist. Emailing invalid email addresses could trigger another bounce that would revert the Email Invalid field to true if the correct bounce information is returned to our mail systems. For additional information on hard bounces and their classifications, visit our Marketo Docs, resources.
What happened:
This issue originated from a patch deployed during a recent release. The patch was designed to update the database schema for valid email addresses that are imported through a list. This did not produce the correct behavior and inadvertently caused the Email Invalid field to be set to False.
Remediation:
Once the cause was identified, our team developed and deployed a patch on May 8, 2019, to revert the changes that introduced this issue. To correct the data that was affected during the impacted timeframe, our team is scoping and designing a bulk data fix process with the highest urgency. We expect this process to take two weeks to complete, with an anticipated end date of May 28, 2019. You can also choose to manually fix the affected data by creating the smart campaign outlined below. If this option is chosen, the data fix being done by our team will not interfere with the corrected values.
Email Invalid Data Fix Smart Campaign:
Smart List:
*Change the date range based on the data center your instance is located in.
Sources to copy and paste into the Source multiple value chooser menu:
List import
Web service API
Untrusted List import
Web service API - Import to list
Flow:
Schedule:
Uncommon Use Cases to be aware of:
1. The Webservice API source includes BOTH (a) Bulk API Import WITHOUT a "listId" parameter in the REST call and (b) Regular API calls. If a customer uses Regular API Calls ex. (https://developers.marketo.com/rest-api/lead-database/leads/#create_and_update) to update the Email Invalid field to False (during the impacted timeframe) making the lead Marketable, the manual fix above would accidentally set the Email Invalid field back to True making the lead NOT Marketable again.
2. If a customer has a real use case which updated a subset of the affected leads to Email Invalid is False, the manual smart campaign data fix will incorrectly mark them as Email Invalid is True. Note that Marketo does not generate a Change Data Value activity if a field value hasn’t changed. An example sequence of events is provided below;
The list import issue inadvertently changed a lead's Email Invalid field from True to False. A Change Data Value activity in the lead's activity log would have been generated.
If a customer finds that the lead has a valid email address from another source and updates the leads Email Invalid field to False (No Change Data Value will be generated since no change to the value was made), the data fix will identify such leads as problematic, and reset their Email Invalid field back to True. In summary, there is a chance this process will mark a Marketable lead as Unmarketable via the Email Invalid field.
If you have additional questions or experience any issues, please contact Marketo support at https://support.marketo.com, or through any of the methods listed here.
.png)