Munchkin Activities Recording Issue – Trigger Replay Overview

This document provides an overview of the recent Munchkin Activities Recording Issue that impacted a subset of customers across our data centers beginning on April 24, 2019. All customers that were impacted were emailed details of the incident on May 1, 2019. If you are unsure if your instance was impacted, please reach out to Marketo support at https://support.marketo.com.

Start time:

  • London data center, April 24, 2019, 4:30 PM PDT
  • Sydney data center, April 26, 2019, 2:00 PM PDT
  • San Jose and Ashburn data centers, April 26, 2019, 5:30 PM PDT

How to identify what data center your instance is located in:

You can deduce the server locations by looking at the URL of your instance. The CNAME=the pod or server your instance is on.

Instance URL.PNG

The example above indicates that this instance is in our Ashburn data center.

The abbreviations for all our data centers are as follows.

lon = London

sn = Sydney

sj = San Jose

ab = Ashburn

End time:

  • Updated patch was pushed April 30, 2019, 3:40 PM PDT

Service affected:

A subset of Marketo customers may have experienced Munchkin activities coming from web pages (clicks, visits) that were not being logged in activity logs. Since the activities were not immediately registered, any Trigger Campaigns set to initiate when those activities occurred would not have been triggered.

What happened:

This issue originated from a patch deployed during a recent release. The patch was designed to update the process used to determine if a subscription’s web traffic should be retained. An error in the patch code caused one of the evaluation parameters to be missed, resulting in web traffic activity not being logged.

Remediation:

Once the cause was identified, our team developed and deployed a patch on April 30, 2019, to correct the piece of code that introduced this issue. To backfill the activity data that was not recorded during the impacted timeframe, our team has developed a script to restore the data back into our activity logs, so no data will be lost. We expect this data fix to be complete on May 24, 2019.

Campaigns that would have been initiated by the activities that occurred during the impacted timeframe will not be replayed. If you need assistance to identify which smart campaigns were active during this incident, our Campaign inspector is a great tool to assist you. You can use this to search for active campaigns that use specific triggers (“visits web page” or “clicks link on web page”).

Replay Impacted Smart Campaign:

After you have identified the smart campaigns that may have been impacted, create a new smart campaign using the following smart list filters.

For smart campaigns triggered by clicking a link on a web page, use the following filters. Select the applicable link and timeframe based off what data center your instance is located in. The second filter is to prevent activities that were not impacted by this incident.

sc1.png

For smart campaigns triggered by a web page visit, use the following filters. Select the applicable web page and timeframe based off what data center your instance is located in. The second filter is to prevent activities that were not impacted by this incident.

sc2.png

After the smart list is set up, the flow steps should mirror the flow steps of the original smart campaign that you wish to replay.

If you experience any issues with replaying the impacted smart campaigns, please contact Marketo support at https://support.marketo.com, or through any of the methods listed here.