All about Lead Deduplication and Merge

[Raakshasi]

All of us know Marketo automatically Merges (De-duplicates) records when created with the same email ID. But then, there are scenarios where the leads will be created as duplicates and are not merged even though they have the same Email- address.

Let’s check out the scenarios and how to merge them.

Scenario 1: The leads coming through Form fill out.

Now for example: when a lead is filling out the form and has some issues with the submit button or network issues or maybe an issue with the form, when the lead tries to click on submit button multiple times continuously in the same time, no matter how many times the lead clicked on submit button, the same number of leads are created in the instance and Marketo won't be able to identify the original lead because of this “Race Condition”. Hence the duplicates are created and should be merged Manually.

If the duplicates are high in number then, unfortunately, there is nothing can be done from our end, you can recommend them to professional services.

Scenario 2: The leads coming through SFDC.

We saw many scenarios where SFDC creates duplicates. But why? Because Salesforce creates a new person based on SFDC ID, and when Marketo sees the new lead, it looks at the ID, not the email address. So, if a Lead exists in Marketo and then a Lead with the same email address is created in Salesforce and syncs down, Marketo will create a new duplicate Lead.

For example, You have a lead in Marketo from Form fill out and the same lead already exists in SFDC and synced to Marketo but then created two records when sync happened because Marketo lead doesn’t have SFDC ID because they didn’t sync to SFDC to have an ID. The best way to do so is to create a lead via a form fill and then sync it to Salesforce. If that cannot be avoided, then the records must be merged Manually or through Easy Merge (through Marketo Services).

(As a best practice, always merge the fields in SFDC and sync down to Marketo)

Scenario 3: The leads coming through Webservice API or Rest API.

We all know that Marketo does de-duplicate leads (based on email addresses) from web forms and list imports, it does not necessarily de-dupe leads coming in via the REST API. It depends on which REST operation is used. When performing lead upsert requests concurrently or in quick succession, duplicate records may result when making multiple requests with the same key value if a subsequent call with the same value is made before the first returns. This can be avoided either by using the createOnly, or updateOnly as appropriate or by queuing calls and waiting for your call to return before making subsequent upsert calls with the same key.

To Merge the leads, there is a paid service called Marketo Automerge, which you can purchase from your customer success manager.

You can find more information on the following link: https://developers.marketo.com/rest-api/endpoint-reference/lead-database-endpoint-reference/#!/Leads...

Note: The best possible way to avoid duplicates from different sources is to have a secondary deduplication field. You can ask the customer to add the custom dedupe fields on the below functions of Marketo.

1)Web Form Fillout
2)Munchkin API
3)Webservice API
4)List Import
5)Web Service API - List Import

Once the rule is applied Marketo will not allow the duplicates to create in Marketo from the above possible ways.

I hope this helps.

[Jwestover]

How does the custom dedupe fiedl work? You create it, what kind of field? How is it supposed to function to prevent duplicates.

[SanfordWhiteman]

---

How does the custom dedupe fiedl work? You create it, what kind of field? How is it supposed to function to prevent duplicates.

---

You're right to wonder!

A custom dedupe key isn't a single field. It's a compound key consisting of the Email Address and another (typically string) field.

It helps tune your instance to your business's definition of a duplicate. For example, in your instance, 2 people with the same Email Address but different Phone Numbers may not be considered duplicates, as they are marketed to separately.

It wouldn't say this "prevents" duplicates, but rather it more accurately defines duplicates.

Most Marketo experts are not in favor of compound/custom dedupe keys because they cause more problems than they solve.

[Jashan · Accepted solution]

Lead duplication occurs when multiple entries for the same lead exist in a database or CRM system. This can happen when different sales or marketing teams input the same lead into the system or when a lead provides their information multiple times through different channels. 

Data Duplication is one of the major costs organizations are incurring every year and badly hitting the productivity levels of the sales and marketing organizations. 

There are numerous reasons why the issue of data inconsistency occurs:

• Lack of unique identifiers at various entry points. 
• Lack of valid patterns and integrity constraints
• Data heterogeneity causes difficulty in data comparison. .

Having multiple records in your CRM results in a lack of single customer view.

From a marketing perspective, sending emails to the same customer over and over results in ruining the brand image. 

However there is a solution to solve this problem, data deduplication. This process helps in removing the replicating data in your software and making it spick and span for future campaigns and data analysis.

There are various tools and techniques that can be used to identify and merge duplicate leads in a database or CRM/MA systems. These can include automated checks, that leverage algorithms to detect duplicates, or hiring third-party services to help with the process. It is important to regularly review and clean up your database or CRM system to avoid the negative impacts of lead duplication and ensure that your team is working with accurate and complete information

In case you are using Marketo, de-duping through the inbuilt feature is not enough. The feature comes with a lot of manual work wherein it does not let you configure and define criteria for deduping the leads.  

There are numerous deduplication solutions like M-Clean, Cloundingo, , DemandTool etc available in the market that can be easily integrated with Marketo.