Velocity Loading with Duplicate Records

[Wyatt_Bales4]

As we continue to try and roll out Related Contact functionality, we still have an issue with duplicates in Marketo--mostly leads and contacts with matching email addresses. Normally for basic batch campaigns this isn't a big concern (besides some attribution and reporting issues); however, for email scripting it's causing a bit of a headache.

Problem: Email scripts inconsistently load (in preview, sample, nor actual send) when there is more than one record in the database with a matching email address.

Possible Culprit: When Marketo tries to understand which record to qualify in a duplicate scenario, batch campaigns qualify the first created record while trigger campaigns qualify the last updated record. I believe Velocity pulls in the most recently updated record which if doesn't match with what the smart campaign brought in, the script won't load.

I understand there are multiple ways of ridding duplicates from Marketo but focusing on this issue, I'm trying to understand if there's anything which can be done inside the email script itself.

Curious how Sanford Whiteman​ or other savvy email scriptors compensate for duplicates while using email scripting.

[Wyatt_Bales4]

Justin Cooperman​ are you aware of any tricks to make smart campaigns ONLY pull in the record with the matching criteria and not its duplicate? (e.g. If there's a Contact and Lead with the same email address but only the Contact has a Custom Object relationship, make sure the smart campaign pulls in the Contact for the email script to reference)

Appreciate any Product insights you may have outside of enabling Purposeful Duplicates for the instance.

[Justin_Cooperm2]

For certain COs it is possible to add that as the criteria for smart list qualification. Think of something like "Justin" AND "has product X". But, can you give more specifics about the custom objects you're referencing. Are they Marketo COs or CRM COs? Are you guaranteed that only one will have a CO relationship?

[Wyatt_Bales4]

Thanks for the reply, Justin Cooperman​. This isn't necessarily just focused on Custom Objects but how smart campaigns qualify duplicates.

Use Case: Load a roll-up table of Uber Eats transaction data for a McDonald's location (e.g. Display 10 metrics for how each of the Franchise Owner's locations performed yesterday). 

Solution: An email script loads tables in an email showing Uber Eats transaction history at 1 or multiple McDonald's locations. It's a simple #foreach loop.

Smart List Criteria: Has McDonalds Order Summary and Country. This "should" only qualify Contact: wyatt.bales@mcds.com.

Issue: There is a duplicate wyatt.bales@mcds.com (Lead) and since it's the first created the record, the batch campaign qualifies it instead (even though the Lead doesn't have a Custom Object relationship).

So what we're seeing is even though the smart list criteria "should" only qualify the Contact, the Lead is processed because it's the first created record. Thus Velocity references the Lead which doesn't have a custom object relationship. From what we've tested, uploading custom object data through the GUI ties it to the last updated record--not the first created record. Thus creating kind of a gap between trying to append custom object data and have it referenced in a smart campaign (only in the scenario of duplicates).

We now have an automated API upsert but besides trying to trick the upsert to append only to the first created record, I'm unaware of any other tricks.

[Justin_Cooperm2]

Hi Wyatt,

Thank you for the added detail, I better understand the issue you're seeing. Let me take a look at that and see what's happening there. I think you're correct that, regardless of creation date, the duplicate Lead record should not even be part of the Smart List qualification since it doesn't have the CO relationship you're specifying.

Feel free to shoot me an email so we can discuss further.

Justin

[Dan_Stevens_]

Hi Wyatt - once you get to the bottom of the this, let’s be sure to update this master table of duplicate scenarios found here: If duplicate leads are an issue, be aware of how Marketo determines which lead is active

[SanfordWhiteman]

Velocity runs in the context of the same lead that receives the email (you can verify this by outputting ${mktmail.id}, which is the Marketo ID).  So I'm not sure where you're seeing crossover -- there shouldn't be any way for lead X to have access to lead Y's attached objects, even if you wanted it to! Can you describe what you mean by inconsistently? Is it that you want to see data across leads?

[Wyatt_Bales4]

Thanks for the response, Sanford. It's not a matter of crossover but a matter of not populating when there's a duplicate.

For example, if I have a simple $object.attribute in an email pulling from a Custom Object, it will not display any value in the email if there is a duplicate record in the database (I'll explain more on how this isn't 100% of the time as it's dependent on which record was created first).

Because the smart campaign could be pulling in the Lead duplicate when in fact it's the Contact which has the Custom Object (which is confusing considering you're using a custom object smart list filter), the email script will render null because it doesn't see the Custom Object relationship. This has to due with how Marketo smart campaigns process duplicate records. Even though you provide very specific Contact-related smart list filters, the duplicate Lead can still qualify because it has a matching Email Address and was created first.

A simple test would be to create a lead in Marketo, sync over a duplicate Contact, and within an email script, reference a Custom Object field which is tied to the Contact. Running a batch campaign will qualify the first created record (i.e. the Lead) and thus the email script won't see the Custom Object.

It's the only common theme across the records which all pass through the same token. Support has confirmed duplicate records are the root cause but I'm still trying to research how to mitigate Velocity from showing null. I understand the root cause is qualification (even before it gets to Velocity) so I guess this question is more focused on smart campaign qualification/attribution than Velocity.

[SanfordWhiteman]

Right, as you say in your last paragraph, this isn't a Velocity issue.  It can be frustrating to try to build SL qualifications by CO (another example is when you add multiple CO filters and they don't apply to the same exact CO record).  Reducing duplicates is the key, or at least making sure only Lead or Contact, but not both, can ever have COs.

[Wyatt_Bales4]

Thanks, Sanford Whiteman​. Considering how prevalent duplicates are, I'm surprised more customers are not complaining about email scripts not loading values. Even if you reference normal fields on the Lead record, duplicates will cause the script to not load. We send over login details by concatenating different fields together and if there is already a record with a matching email address, there's no guarantee it will load the right details (e.g. we use an Account email address field for Restaurants to login but if the Lead is pulled in, the script won't load).

[SanfordWhiteman]

"Scripts won't load" doesn't seem like the right way to put it, they'll load whatever they can for the lead that Marketo chooses to qualify.

[Wyatt_Bales4]

"Incorrect smart campaign qualification due to duplicate records causing Email Script to load incorrect values."

Understood. I'll try to work on phrasing it's just unfortunately Velocity is the only "visible" indicator the wrong record has qualified.

[Grégoire_Miche2]

Hi Wyatt,

Can you please provide some more information about the data schema you are using (what custom objet) and also the velocity code you have developed ?

-Greg

[Wyatt_Bales4]

Thanks for the response, Greg. We use various custom objects ranging from transactional nature (e.g. order history) to a reference table (e.g. onboarding status). The dedupe fields are usually a custom row ID with the link field either being email address or SFDC ID.

As for the scripts, some are #foreach loops while the majority are just a simple $object.attribute reference. All mimic the same behavior regardless of the schema or scripting logic.