If duplicate leads are an issue, be aware of how Marketo determines which lead is active

[Dan_Stevens_]

Many of us deal with duplicate leads on a regular basis.  Today, I just learned of something that's been causing issues for us, as we were always under the impression that Marketo considered the lead that was "updated last/most recent" as the active lead - and the lead that would be affected for things like list uploads, email sends, etc.  That's not the case.  This could have significant implications if you have a lot of duplicates in your instance. I just posted an idea that hopefully will get enough attention and change the way Marketo deals with duplicate leads:

NOTE: if you head over to the Idea page ( ), you'll note that a serious flaw has been identified - specifically around the way Marketo re-uses old LeadIDs (when the original lead is purged from Marketo) and allows new leads to take on these old IDs.  This could have serious implications, for example, for those of us that leverage the API in our environments.

[Nicholas_Manojl]

The use case isn't really a one-size-fits-all approach though.

You have a very complete lead record that was updated 6 months ago.

You have a duplicate lead that was created 1 minute ago, but it contains just an email address.

In your scenario, you want to email the lead record that only has an email address.

I think the problem is much harder than this.. in reality we need a way to methodically judge the quality of a lead record. But how? It's a very difficult problem.

[Dan_Stevens_]

Totally get that.  The premise of the Idea was to have consistency across the entire environment - for batch/trigger email sends, list imports, form submits, etc.  But that's not what I'm referring to here in my most recent post.

If you look at the Idea comments/replies - beginning with Aaron's post on January 29, 2018 - he uncovered a pretty serious flaw in the system where Marketo will reuse leadIDs (when leads are deleted/purged from the system) and allow new leads to assume those older IDs.  This could have some significant implications for those of us that rely heavily on the API.  As Sanford put it:

"This is far and away the most fatal and mind-boggling technical detail -- if indeed it's true -- I've heard about w/Marketo, ever."

[Nicholas_Manojl]

well, I haven't really touched the API much in recent times. But my understanding was the leadID was on the way out and the GUID was on the way in. But I think broadly speaking you make some really excellent points, so I definitely support you on that.

[SanfordWhiteman]

marketoGUID is for Activities.

[Dan_Stevens_]

Yeah, here's the thread where we discussed that change about a year ago:

Re: An Important Change is Coming to Marketo's REST and SOAP APIs

...and the link to the formal update on the developer site:

http://developers.marketo.com/blog/important-change-activity-records-marketo-apis/

[Dan_Stevens_]

Niranjan Kumbi [PM], near the bottom of this thread ( ), you mentioned that you would investigate internally to determine if this is an issue.  Do you have any updates for us?

Sanford Whiteman​

Aaron Dear​

[Anonymous]

Thanks, everyone.  Very helpful.

[Gerard_Donnell4]

Thanks for sharing this info! It definitely answers a few questions.

[Dan_Stevens_]

As I posted within the idea I created ( ), one way to remedy this would be to offer an option in the Qualification rules of a batch smart campaign (this last option does not exist - this is just a mock-up):

[Jenn_DiMaria2]

Wow. Thanks for posting this. I always thought it defaulted to whatever was in SFDC (because that's what it always seemed like when I'd go to merge them). Learn something new every day

[Dan_Stevens_]

Jenn - do you automatically sync all leads from Marketo to SFDC?  If so, then your assumption may still be true.  The issue arises when you have duplicate leads/contacts in SFDC (which, of course, also exist in Marketo).

[Jenn_DiMaria2]

SFDC is our master and we don't sync all Marketo leads to it, but we still have lots of issues with duplicates. Namely because our main contact form on the website is an SFDC web-to-lead form and many of our customers think they can reach Support this way. Of course, that then creates duplicates, which makes my life super fun

In many cases with those duplicates, we then have a Contact on an Account and then a Lead. If they've merely submitted the W2L form, those are easy to merge, but I remember when we first migrated our database, there was some best practice from SFDC about holding a Lead separately from a Contact, despite being the same information. I wish I remembered the details, but it's such a data pain.

[Dan_Stevens_]

Not to take this discussion too far off-topic, but for our contact-us form on our website, nothing is even considered a lead unless the submitter selects one of the appropriate values from the "Relationship with Avanade" field:

In the trigger campaign, we classify "current customer" and "potential customer" different from the others - and based on this program status, these leads proceed through our "fast-track to MQL" workflows (actually they're in an "Evaluation" stage until the marketer physically qualifies it in CRM.

We understand that other values - like "other" - can be potential leads for us as well.  We have workflows in place to fast track these types of requests, just like the other ones.

[Grégoire_Miche2]

Hi Jenn,

I really feel sorry for you to have to bear web-to-lead, which is a 15 years old technology that creates headaches to Marketers all around the planet.

It comes from a time when the online form had just been invented. The precedent tool for targeting was phone directories. So, being able to call only people who had filled out a form seemed a tremendous progress, compared to calling every one in the city or the region

The underlying process was also quite straightforward: when a lead would enter the DB, it was qualified ASAP. If it was good, it was converted and if it was not not good for the sales, it was discarded. So duplicates were not really an issue...

What is really an impressive success from SFDC is that, 15 years later, companies are still using it, although the underlying concept is completely obsolete.

-Greg

[Jenn_DiMaria2]

It really is painful and stems from the fact we wanted to use all of SFDC's features; we also didn't have Marketo at the time of our website's implementation. We've looked into switching to Marketo forms, which I'm hoping happens soon as I use them for every other form on our website and it's a beautiful thing! It's a shame Marketo wasn't included in our site's redesign, as the main concern now is the forms won't fit in as seamlessly as we may like (nor be as responsive).

That said, I honestly can't believe W2L doesn't automatically de-dupe anyway, especially now they have Pardot/the technology to do so.

[Grégoire_Miche2]

Hi Jenn,

As I wrote, W2L is a very old technology that was never updated and has absolutely nothing to do with Pardot. In fact, Pardot is not there to enhance W2L but to replace it, as Marketo does.

-Greg

[Dan_Stevens_]

For your reference, here is a detailed breakdown of Marketo's duplicate rules processing:

Sending Batch Emails

If there are duplicates within your email list, only one email will go out. It will be the lowest Marketo ID (created first). This record is also used to populate tokens.  Example:

Created	Updated	Marketo ID	First name	Email
3/13/2010	4/13/2010	2749	Elizabeth	beth@company.com
12/15/2012	3/15/2016	10936	Beth	beth@company.com

In this case, the email will be sent to the first record with Marketo ID 2749. It will say “Dear Elizabeth” if you use a token for first name. Also, the activity entries will be attached to that record (sent, delivered, opened, clicked). When the email click goes to a page with Munchkin code in it, any web visits will also be associated with this record.

[EDIT 1/30/2018]: apparently this is not the case.  As you can see in the latest comments, here in the Idea related to this: , Aaron Dear​ uncovered a serious issue/bug - especially if you have a lot duplicates in your instance (in some cases, these duplicates are intentional).  Basically, there is no defined logic to determine which lead record is used when evaluating duplicates as part of a batch email campaign.

List Uploads

When a lead is included in a list import and the lead already exists in Marketo, it will affect the most recently updated record in Marketo (updated last) - in this case, the record with the first name "Beth".  If you're intention was to ensure Beth's record had the most up-to-date information so that some of that info could be used in an email send, you're out of luck.  Marketo will use "Elizabeth's" record for that send.

Sending Trigger Emails

While a batch email de-duplicates, trigger emails do not de-dupe because triggers “batch each record individually”. However, it is uncommon that trigger emails are used for batch emails, so this is not usually a problem. Also, it is rare that duplicate records will qualify for a trigger campaign at the same time.

Form Fill-outs

If someone fills out a form and their email address is in Marketo multiple times, the form submissions will be applied to the most recently updated record (by last modified date in the “updated” field). The data value changes in Marketo will only affect one record if there are duplicates; they will not affect the duplicate records. For example, if a record changes their title, the change will not occur for the duplicate records.

Assuming that the Marketo cookie information is included in the form (standard for Marketo landing pages or embedded Marketo forms), the web page visits will now be tracked on the most recently updated record. Luckily, the most recently updated record is often the record that is most up to date. (which is why the the way Marketo treats batch email sends is an issue).

Note: If there are duplicate records in Marketo and one of the records is cookied, the cookied record will be associated to the form fillout regardless if a duplicate record was updated last. As noted above, the form fill-out activity may not be associated to a record which was sent an email directing them to the form fill-out.

Web Activity

Web activity will only be associated with one record if there are duplicates. If a record has a duplicate record and he/she goes on the website, the munchkin code will only track activity for the record that most recently was cookied by submitting a form. If the visitor’s cookie is reset and visits the site again and fills out a form, the record may be associated to a different record in Marketo. Therefore, all web activity that is recorded may be spread across the duplicate records.

Merging Leads

When merging two leads, Marketo will always use the most recent one to define the "Created On" date.  This is undesirable since most would prefer to keep the initial/first date of the older lead as the "Created On" date.

Thanks to Kristen Carmean​ for surfacing this comprehensive list with us!

[Mark_Wallace1]

Hi Dan,

Im now having a number of issues with dupes and seems your in the same boat.  Quick question re the above.  In our instance where dupes are allowed any form fill creates a new duplicate.  So for the above how do you assign a form fill out to an existing version of the lead?

I have an approval process where sales fill a form that updates the lead record and triggers some emails to go to the customer.  But now when they fill the form it just creates a dupe which needs a manual merge to put the lead through the flow.

With duplicates allows how can you assign a form fill out to an existing lead?

thanks

Mark

[Dan_Stevens_]

Hi Mark - Marketo dedupes (on email address) by default on a form submit - and the "form fill-outs" scenario above would apply.  Do you have a custom setup where a purposeful duplicate is created on form submit?

[Mark_Wallace1]

Hello

Unfortunately we do, its a checking if the content of the product interest field changes and deciding if a dupe needs to be created.  Its a function of the marketo standard connector and MS dynamics.

Its starting to cause issues in more ways than one.

thanks

Mark