Re: Unknown cause of lead merge (leaddb) - Expert knowledge required!

Sean_Richards · ‎03-25-2019

I just converted a lead to a contact in Salesforce, and then the record triggered a score update:

This is despite having a criteria on our score triggers (which are global, not campaign specific) to exclude the "Lead merge" reason.

Funny enough as I write this I've noticed that the Reason value in this latest "Merge" has a different string to the existing exclude criteria, although I was sure that the exclude string I had in place was the correct one to exclude Merges.

Can anyone confirm that this new reason value "Merge (leaddb)" is the value for a conversion, rather than a merge?

I feel we're deep in the Marketo dark arts here, so please shed some light!!

Thanks to all in advance.

Sean

Sean Richards

Sean_Richards · ‎07-08-2019

Hi Matt,

I've been holding off bothering to reply to this thread. Yes, I found a solution. After my ticket spent months in Marketo's top tier support channels, they didn't have a solution. Well, not entirely true. They did, but it was not the actual solution. Here was there response.

-----------------------------------------------------

Hi Sean,

Thank you for your patience with this one. Our Engineering team looked into this case and found logs for the affected records. One such is below:

Logs show operations of data pull:
Lead Pull conversion SOQL:
Mar 28 17:42:31

Contact Record created in sfdc at:
sfdc_created_date: 2019-03-28 17:42:32

Contact pull-updates SOQL:
Mar 28 17:42:46

During a Lead to Contact conversion. When the standard conversion process is performed in SFDC, a new contact is created in SFDC and the data mapped between lead and contact is used to populate the fields of the contact in SFDC. The lead in turn becomes a "shell" of a lead. This conversion is pulled during the Pull Conversion operation in the sync. New contact records or contact updates are both pulled in the Contact Pull-Updates operation in the sync.

The lead was converted to a contact during the short window of time between the conversions were pulled and the start of the updates being pulled. The new contact was pulled over to Marketo by the pull-updates operation rather than the pull-conversions. Hence the creation of the new contact in Marketo - a duplicate. During the coming sync cycle after that, the pull-conversions operation merged the duplicates to complete the conversion.

Our engineers have confirmed that when this scenario occurs (a conversion happened in that small time gap), this behavior is expected to occur.

-----------------------------------------------------

See, I already knew this. That the API calls run in a specific order (there is a post on this somewhere) and that these got out of sync. But why? There is no answer to that, nor why my records caused this to occur dozens of times.

Well. I can now report that in our case, this was not due to RingLead. It was actually caused through the combination of having a lead validation rule in SF, plus a small setting enabled in SF.

In SF, navigate to Setup then search for "Lead Settings". Locate the "Require Validation for Converted Leads" checkbox and Disable this setting. This setting was causing field validation and workflows to execute on conversion, and this was the cause for breaking the sync API execution timing.

Prior to figuring this out, we also setup a great Marketo program that "captured" these irregular merge records. I highly recommend creating this as an operational program so that you stamp these errors as they occur so that you can clean them up.

Basically, the activity log registers two "Person is Created" records when this bug occurs. So you can capture that and add the record to a static list. My list currently has 18 people in it.

I wanted to let support try and come up with this solution or at least tell us to check that setting in SF, but unfortunately they didn't have many cases of this error occurring and never once pointed to this SF setting as a possible cause of the issue.

I feel pretty confident that if you disable that setting in SF, you two will see these issues cease to occur.

Funny thing that having that program to stamp them in place, one person (like 1 week ago) triggered the issue to reoccur. But I guess that's acceptable and is what you would consider "regular" occurrence of this de-sync timing issue.

I hope that this information helps others now and in the future. I lost maybe 40 hours to this issue.

Finally, shout out to Veronica Holmes‌, it was an issue in Salesforce Nice work haha.

Cheers,

Sean

Sean Richards

View solution in original post

Sean_Richards · ‎03-27-2019

There is more information Justin Norris added here, which also confirms the issue: https://nation.marketo.com/message/200295-re-converting-lead-to-a-contact#comment-200295

Disabling the sync is not a solution however, as I tested the conversion being done from RingLead in batch, Ringlead, single records one by one, and Manual conversion within SF and was able to trigger the issue in all three scenarios.

Sean Richards

Jay_Jiang · ‎03-26-2019

Quite possibly Marketo changed the default reasons at one point in time. But yes, some in deep dive documentation into lead merge would have been nice.

Steven_Vanderb3 · ‎03-26-2019

Did the record's activity log record a Merge activity? When two records are merged in Salesforce Marketo will sum their scores together so this really looks like a merge rather than a conversion...

Sean_Richards · ‎03-26-2019

Hi Steven,

That's what I'm trying to say. The activity logged a "Convert Person", AND through that conversion, it appears that it also Merged a two records. So you, this makes sense that it then added the scores together, and also re-triggered the score campaigns, as the exclude reason didn't match the string we had in place.

However, I am unsure as to why the Convert person, triggered a merge. The two records had different emails (even though they were the same person), so surely Marketo would not have auto-merged them. The Lead should simply have become a Contact under the Account and no merge should have been triggered.

Here is the detail from the Activity from the Convert:

Assign To:	Sean Richards
SFDC Type:	Contact
Person ID:	1167198

Here is the detail from the Activity from the merge (I've remove some of the details):

Merge IDs:	1117921
Merge Fields:	Account Owner Email Address: sean.richards@alaress.com.au Account Owner First Name: Sean Account Owner Last Name: Richards Accounts: 1 Billing Country: Australia Billing Country Code: AU Billing State: Queensland Billing State/Province Code: QLD Global Region (A): Australia and New Zealand Main Phone: (07) 5508 2929 Open New Business Opportunities: 0 Organisation Type: Organisation Record Type ID: 01290000001IbuEAAS SFDC Account Type: Partner SFDC Campaigns: Lead Lifecycle Processing; WE - [18.09] Schoolbox + Digistorm - An Integrated Mobile Experience; NSE:MU [19.02] [Meetup]: SBMu19 Anonymous IP: 61.69.104.170 Behavior Score: 122 Unsubscribed Cause: System flow action for 'Change Data Value' triggered Mon, 27 Aug 2018 17:28:33 -0500 Lead Score: 268 Id: 1167198
Master Updated:	true
Merged in Sales:	false
Merge Source:	leaddb
Person ID:	1167198

OK, so as I understand it, the scoring campaigns didn't get triggered, and that the score's were added together through the merge.

But why would a merge have occurred through the convert?

Sean Richards

Veronica_Holme4 · ‎03-26-2019

You're 100% sure that when you converted the lead in Salesforce you didn't convert INTO an existing Contact in the Account? That's a merge.

Sean_Richards · ‎03-26-2019

Hi Veronica.

As far as I can tell. I only did a Convert. However there is another factor at play. The convert was not done in SF manually.

I performed the Convert through RingLead, which is configured to "Convert Lead to Contact at Matched Account". Which (as far as I know) doesn't do a merge, even if there was a dupe.

But I'm sure there are no dupe emails at all in the instance. There was an existing Contact with the same name etc, but I have to assume that RingLead would not have done a merge on this action, that task is not configured at all to merge on dupe, just to perform the convert.

I can't figure out what triggered the merge. Marketo would only merge on a dupe email. But the Lead was already in Marketo, so there is no de-dupe trigger I can think of occurring from Marketo.

On the Contact in SF, under the Contact History, there is only one record which says "Created by lead convert".

When I did the Convert, Marketo has logged an Activity against the existing Lead as "New Person" which also has reprocessed the lead through our Lead Lifecycle (as we must need to also add the exclude reason to that trigger to stop them reflowing through after Merges). It's also recalled our global scoring triggers as it's re-added the new contact to the campaigns, triggering these score events to fire again.

The "Change Score" activity details are as follows:

Score ID:	146
Score Name:	Person Score
Change Value:	+124
Old Value:	144
New Value:	268
Reason:	Merge (leaddb)
Urgency:	144
Relative Urgency:	3
Relative Score:	2
Priority:	140
Person ID:	1167198

Which is obviously triggered from this mysterious merge trigger.

I did another test and converted another Lead to an Account through RingLead and this did not cause any Merge, even when there was a contact on the matched account with a dupe email.

This proves that a convert does not trigger the "New Person" activity? The Merge triggers it. But why did the merge occur.

There must be some edge case here. I just can't see it.

Sean Richards

Sean_Richards · ‎03-26-2019

Can anyone explain this

Merge IDs:

1117921

Merge Fields:

Account Owner Email Address: sean.richards@alaress.com.au
Account Owner First Name: Sean
Account Owner Last Name: Richards
Accounts: 1
Billing Country: Australia
Billing Country Code: AU
Billing State: Queensland
Billing State/Province Code: QLD
Global Region (A): Australia and New Zealand
Main Phone: (07) 5508 2929
Open New Business Opportunities: 0
Organisation Type: Organisation
Record Type ID: 01290000001IbuEAAS
SFDC Account Type: Partner
SFDC Campaigns: Lead Lifecycle Processing; WE - [18.09] Schoolbox + Digistorm - An Integrated Mobile Experience; NSE:MU [19.02] [Meetup]: SBMu19
Anonymous IP: 61.69.104.170
Behavior Score: 122
Unsubscribed Cause: System flow action for 'Change Data Value' triggered Mon, 27 Aug 2018 17:28:33 -0500
Lead Score: 268
Id: 1167198

Master Updated:

true

Merged in Sales:

false

Merge Source:

leaddb

Person ID:

1167198

Sean Richards

Sean_Richards · ‎03-26-2019

I think that this is an edge case and for some unknown reason, Marketo has performed a Merge. There is literally one mention online of this "leaddb" merge reason here: How to identify leads manually merged in SFDC

This is completely undocumented of course.

I'll log a support ticket and struggle to find someone expert enough to tell me what this is. Wish me luck.

Sean Richards

Sean_Richards · ‎03-26-2019

This article (Understanding Merges in Marketo ) makes reference to the reason "Lead merge" which we added to our score programs which as far as I am aware, catches the manually performed lead merges . But it's unclear what causes the "leaddb" reason.

Sean Richards

Denise_Greenb12 · ‎03-26-2019

Hi Sean,

Found it! Have a read through this thread: Marketo / Salesforce Sync Limits & Priority Info

Hope it helps!

Denise

Sean_Richards · ‎03-26-2019

This was so helpful. Thanks again Denise. I've updated my support ticket with this information and requested someone senior calls me to explain what multi-batch queue does on an instance, the pros and cons and what is required to have this feature disabled on our instance.

Do you have any idea of any caveats to disabling this feature?

If I had to guess, this is probably enabled on shared instances by default as it saves on bandwidth/CPU resources and most basic customers don't ever notice the issues it can cause.

Sean Richards

Denise_Greenb12 · ‎03-27-2019

Hi Sean,

Sorry for the delayed response. The only reason we had multi-batch queuing enabled was to increase the sync speed to clear a backlog (Marketo Support had turned off our sync while troubleshooting a problem and had forgotten to turn it back on!). Last time I checked Marketo engineering had not produced a fix for conversions coming in out of order with multi-batch queuing enabled so I would not recommend using it. It's not turned on my default.

The out of order conversions can occur with the regular sync but are rarer and Marketo has some ability so adjust the sync order to minimize the odds when you are NOT using multibatch queuing.

One other thing to mention since you said you were using LeanData - when I was working in another client's Marketo instance a year or so ago they did an initial mass lead to account mapping (70K records) and conversion process with LeanData and it caused huge problems - including the out of order conversions - because the sync was overwhelmed. If memory serves, Marketo Support advised us to limit the size of those batch conversions to 5K records at a time and put an hour in between. I think we got Marketo to temporarily raise the number of records that can be processed in a single cycle to 10K for this project. But don't hold me to the exact numbers.

Denise

Sean_Richards · ‎03-27-2019

Just a correction, we are using RingLead, however I don't feel that disabling the sync is a solution, as I tested the conversion being done from RingLead in batch, Ringlead, single records one by one, and Manual conversion within SF one by one, and was able to trigger the issue in all three scenarios.

I hope like you said that "Marketo has some ability so adjust the sync order to minimize the odds when you are NOT using multibatch queuing" might be able to help reduce/fix the issue.

Sean Richards

Denise_Greenb12 · ‎03-27-2019

Hey Sean -

Just to clarify (and maybe you mistyped) - I was NOT suggesting disabling the sync. Marketo disabled our sync while troubleshooting - unbeknownst to us and it was supposed to be for a few minutes - and forgot to turn it back on. This built up a backlog because it was days before we figured out what was going on - and they enabled multi-batch queuing to fix it - and then they forgot to turn THAT off.

If multibatch queuing is in use it doesn't matter whether you do the conversions one at a time or in bulk. The conversions can come in out of order because there is no control over the order in which things sync when.

(The LeanData issue had nothing to do with multi-batch queuing vs regular queuing).

Denise

Sean_Richards · ‎03-27-2019

Yeah I wrote that out of context, I did understand what you were saying.

There is more information Justin Norris added here, which also confirms the issue: https://nation.marketo.com/message/200295-re-converting-lead-to-a-contact#comment-200295

In his comment, he said Marketo support said to disable the sync when doing bulk conversions. But why I said that this won't work as a solution for us is that I have confirmed the issue is occurring (randomal) even on a single conversion done manually in SF.

Sean Richards

Veronica_Holme4 · ‎03-27-2019

Since this isn't the batch conversion issue and Justin has correctly identified (I've seen that one too) I feel like we are back to perhaps a non-standard conversion process in Salesforce we need to identify. If Marketo can't see the lead through it's conversion to a contact (even for a little while) it will believe it needs to create a new record, which turns out to be a duplicate. I've worked with literally hundreds of synced instances, and every time I have seen this issue it has turned out to be a non-standard conversion process, a permissions issue with the Marketo sync user, or something else that means Marketo loses track of the original lead on conversion. The fix is always in Salesforce. We just need to nail it down.

Sean_Richards · ‎03-27-2019

OK, I'll play that game

My Sync user is a custom profile cloned from the system admin profile in SF. It has basically full access to the SF org, bar some objects that are not relevant to it.

Is there anything specific to audit on this user? It certainly has never displayed a permission issue to date.

If it was not a permission issue, when you say "non-standard conversion process", what does that mean?

When you say "something else that means Marketo loses track of the original lead on conversion", what could cause this?

Thanks Veronica!

Sean Richards

Matt_Reilly1 · ‎07-08-2019

Sean, were you able to find the solution to this? I'm seeing the same leaddb issues happening on my end. We're using Leandata to run our conversion, but are transitioning to Ringlead soon. I thought Ringlead would be the fix, but am distressed that you're having issues with Ringlead conversions as well.

Sean_Richards · ‎07-08-2019