Merging Automatically with Zapier

Tyron_Pretorius · ‎10-12-2021

Your duplicate list becoming ever more daunting? Stop the rot by learning how to build a zap in Zapier that will merge duplicate leads as soon as they are created according to custom rules you specify.

Of course, the optimum approach to dealing with duplicates is to address the root causes that are causing duplicates to be created. The zap is only intended to be a patch while you address the underlying issues creating duplicates.

If you are instead looking to deal with that daunting pile of duplicates in your instance and merge them all at once in bulk then take a look at the Merging Leads in Bulk using the API post to see how you can tackle this backlog using Python and the API.

Also if you are looking for a course for learning the Marketo API then DM me 🙂

All the Python code used in the “Code by Zapier” actions below can be found in this Github folder.

Adding Duplicates to Static List

A workaround is needed to be able to trigger a Zap when duplicates are created in Marketo. As shown in the “Lead Added to Duplicates List Trigger” section below the “New Lead” trigger event of the Marketo app in Zapier is used along with the constraint that the newly created lead must be a member of the “Possible Duplicates” static list.

Therefore in Marketo, every time a duplicate lead is created a smart campaign is used to transfer this lead to the “Possible Duplicates” static list and thereby trigger the Zap to run.

Smart campaign trigger for newly created duplicates

Smart campaign flow to add to the "Possible Duplicates" list

Lead Added to Duplicates List Trigger

Once the “New Lead” trigger event has been selected in the Marketo app in Zapier then add the “Possible Duplicates” static list to the “List” field.

New lead added to list trigger in Zapier

Get Access Token

The “Code by Zapier” action is used along with Python code (get_marketo_token.py) to make a request to the authentication endpoint to retrieve the access token needed to make subsequent requests.

"Code by Zapier" action using Python code to retrieve the access token

Get Duplicate Leads

The “Lead Added to Duplicates List” trigger will only bring in one of the duplicate leads for a certain email address. This means that the leads endpoint of the API must be used to obtain all people in Marketo who share an email address (get_duplicate_leads.py).

The fields you want to compare between leads to choose winning values (Step 4: Select Winning Field Values) must be brought in in this request so make sure to modify the field_names string to include the fields you care about when merging e.g. leadSource, unsubscribed.

"Code by Zapier" action using Python to retrieve all leads with the same email address

Select Winning Field Values

Once all duplicate leads sharing the same email address have been retrieved along with their field values for the fields you want to compare between them then the logic in the “Select Winning Field Values” Python code will decide which field values should win in the case of a difference (select_winning_values.py).

In general, if there are differences between field values Marketo’s inbuilt merging logic will choose the value of the winning lead provided that this value is not null. However, sometimes there are cases when you would like values on losing leads to be kept and this is where Python code can be used to specify your own custom rules to determine the field values that the merged lead should have.

Here are some examples of the logic implemented in this Python code:

The lead with an sfdcLeadID will be chosen as the winning lead over leads without an sfdcLeadID
The earliest createdAt date will win
A false value for the unsubscribed field will win over a true value (because you do not want to email this person if they have unsubscribed through one of the duplicates)
leadSource field values of Advertising, Paid Search, and Organic will be prioritized over values such as Sales Generated and Referral.

N.B. If you want more examples of prioritization rules you can use to pick winning values for different fields take a look at the “Rules for Winning Field Values” section of the Merging Leads in Bulk using the API post.

You can modify the Python code in this step to prioritize values for each of the fields you care about when merging. Once the code has chosen winning field values these values are then stored and will be used in Step 6:Update Winning Lead to update these same fields on the winning lead to have these winning field values.

N.B. If you want to make sure that the Python logic has selected the correct winning lead id and winning field values before proceeding with the merging step then you can use the Zapier approval action to add a manual review before progressing.

Python code used to determine the winning values for each of the fields being compared on the duplicate leads

Merging

A request is made to the endpoint to merge the duplicate leads with the winning lead determined from Step 4: Select Winning Field Values being specified as the winning id (marketo_merge.py).

To handle the scenario where there might be more than 2 duplicate leads sharing the same email address a for loop is used to successively merge the winning lead with each of the losers in turn. This is necessary because when the mergeinCRM parameter is set to true in the request to the merge endpoint only 2 leads can be merged at once.

N.B. Zapier has a 10-second timeout limit on all actions. When merging leads via request it takes more than 10 seconds for the merge to complete and the successful response to be sent to Zapier. Therefore, this merge action will most likely be marked as failed by Zapier even though the merge was successful. This is a limitation of Zapier and unfortunately, there is nothing to be done here.

"Code by Zapier" action using Python code to merge leads using the API

Update Winning Lead

Once all the duplicates have been merged and the winning lead is left standing, this lead needs to have its fields updated with the winning values stored from Step 4: Select Winning Field Values. Ideally, we would check that the log returned from Step 5: Merge to see that the merge was successful before proceeding, however, as explained above, Zapier’s 10-second timeout on actions means that the merge step will most likely be marked as failed.

Therefore, Step 6:Update Winning Lead proceeds with the assumption that the merge was successful. If you want to ensure that the merge was successful before attempting an update then you can query the leads endpoint for each of the people involved in the merge to ensure that only the winning lead still exists.

A request is made to the leads endpoint to update the winning id designated from Step 4: Select Winning Field Values (update_winner.py). If the update is successful then the script and the zap both end.

However, sometimes Marketo’s inbuilt merging logic might have chosen a different winning id than that specified in Step 4: Select Winning Field Values. This can happen in the case that the one of the leads being merged is a contact in SFDC and the others are leads. This inbuilt merging logic will always choose the contact to be the winner in this case.

If the Python logic used in Step 4: Select Winning Field Values selects an SFDC lead as the winner then Marketo will override this choice and the SFDC contact will be the actual winner. Therefore, when attempting to update the incorrectly labeled winning id from Step 4: Select Winning Field Values this lead will no longer exist and the keyword “skipped” will be in the update response.

This is why a while loop is used to cycle through the other ids that were part of the merge until the actual winner is found and updated.

You can avoid the need for this while loop and ensure that the winner selected in your Python logic always matches that of Marketo’s inbuilt logic. This can be done by modifying Step 3: Get Duplicate Leads to bring in the sfdcType field and then modifying the Python code in Step 4: Select Winning Field Values to ensure that an SFDC contact will always be selected as the winner if the other people to be merged are SFDC leads.

"Code by Zapier" action to update the winning lead with the winning field values

Next Up

Now that you have mastered how to use Zapier and the merge endpoint to merge leads automatically upon creation, what do you do with that huge backlog of duplicates?

I have just the post for you! The Merging Leads in Bulk using the API post will show you how to use the exact same endpoint to merge leads in bulk using Python to make the request and to set custom rules for how you want to prioritize conflicting values for the same fields on different leads.

Also if you want to merge with more confidence take a look at the Zapier Approval Workflow Example with Slack Alerts post (DM me for the link) to see how to add a manual review step to this zap to ensure that the correct winning lead id and field values have been selected before merging.

The content in this blog has been reviewed by the Community Manager to ensure that it is following Marketing Nation Community guidelines. If you have concerns or questions, please reach out to @Jon_Chen or comment down below.

BiancaTimely · ‎12-11-2022

@Tyron_Pretorius - Thank you soo much for sharing this! I have limited dev resource so I have to somehow do this myself and this article helped me soo much and saved me a lot of headache. It works beautifully so far for my refer a friend programme and I'm super stoked.

One thing I'd comment is to add a way to decode the email addresses.

'&filterValues='+urllib.parse.quote(input['lookup_value'])+'&fields='+input['field_names']
print(input['field_names'])

Tyron_Pretorius · ‎01-03-2023

@BiancaTimely I am glad this was useful for you!

I do not follow what you mean by adding a way to decode email addresses. Can you please expand?

SanfordWhiteman · ‎01-03-2023

@Tyron_Pretorius meaning encoding (not actually decoding) query parameters sent to the REST API endpoint.

Params always need to be URL-encoded, a.k.a. percent encoded, to (a) avoid conflicts with reserved characters in URLs and (b) encode characters outside of the ASCII printable range.

Tyron_Pretorius · ‎01-09-2023

import urllib.parse

encoded_name = urllib.parse.quote(input['name'])

url = input['base_url'] + '/rest/asset/v1/program/byName.json?name='+encoded_name

In Python URL encoding can be done using the following example

I have gotten away without needing it for the most part until last week actually when I noticed some of the team included "+" in program names (reserved characters as Sanford mentioned above)