Webhook to create purposeful duplicates in Marketo is unreliable

[Dan_Stevens_]

As part of our purposeful duplicate process (when CRM contacts are created in MS Dynamics) a basic webhook is called to create a fresh new lead:

We're finding that this is working very inconsistently for us.  Sometimes it works, sometimes it doesn't.  What's odd is that we always receive a common error type "Duplicate Lead" Failed. Server Returned code 302" - even when the webhook runs successfully.  When I asked Support about this, they came back with the following:

Engineering confirmed that the HTTP 302 message is not an error. Instead, it is the response code for a page redirect.

HTTP 302:

https://en.wikipedia.org/wiki/HTTP_302

With that being said; I've asked if Engineering can modify the wording being written in the activity log, to prevent any future confusion. I also pointed out that you are not aware of why the redirect is taking place in the first place, and asked if they could explain that as well.

But this doesn't explain why the webhook is working so inconsistently.  I know Sanford Whiteman​ commented last year that the loopback post is broken on some instances (The 3 Coolest Webhooks I've seen (and or) Implemented ).  Could this be the issue?  Any others experience this inconsistent behavior?

[SanfordWhiteman]

302 is the expected response from the legacy /save endpoint, though.

If it weren't, then old + noscript forms wouldn't work: the /save endpoint has to allow for browsers without JS to redirect to a follow-up page, and the only way to do that is via an HTTP redirect (should be HTTP 307 these days, but that's another matter). The new /save2 endpoint doesn't do this, because it requires JS support in the browser in the first place.

So I wouldn't be concerned about the 302.  I'd be looking at whether the call is actually successful. If it's failing, are you sure you're not going over the limit of 1 form post every 2 seconds?

[Dan_Stevens_]

We're still trying to find out what this issue is here.  But when doing some further investigating, we came across another issue: when the webhook creates the duplicate lead, it will populate the email address with the email-address-temp (since you cannot create duplicate by populating the email address immediately for obvious reasons), but then the email address only exists when looking at the lead detail view.  In the lead grid, the email address is blank.  And the lead - which now resides in one of the country lead partitions - isn't even retrievable (e.g., via smart list) in that partition.  Even when searching by last name.  We're not sure of the extent of how large this issue is just yet.

As for ensuring we're not going over the limit of 1 form post every 2 seconds, how can we delay this?  This would be most problemsome when a batch of contacts are added to CRM - which would trigger the webhook call.  I don't think a wait step would affect this here.

[SanfordWhiteman]

but then the email address only exists when looking at the lead detail view. In the lead grid, the email address is blank

Even after running it through the flow to copy Temp Email → Email? That doesn't make much sense, I mean, data sent to /save2 is literally the same path taken by a true client-side data.

However, there is something in your flow that I wouldn't do, and that is post the data without an Email. I'd send {{lead.id}}-{{system.datetime}} as the Email. You're going to change it anyway, and this creates a nice little audit trail while ensuring uniqueness.

As for ensuring we're not going over the limit of 1 form post every 2 seconds, how can we delay this?

You can't directly delay it -- as you point out, a Wait Step would not help, since the event of a flood of new contacts you'd just be delaying the whole flood by a certain amount of time.

If you can't batch up these 'hooks (which would require major rearchitecture) you can use the method I describe here: https://nation.marketo.com/message/154866-re-throttle-web-hook-requests#comment-154866

[Dan_Stevens_]

So here's a sample record where the email address exists, but does not show up in the lead grid, nor the company lead partition:

And here are her records in the lead grid (the highlighted one with the missing email address in the record above):

And when searching her record (by last name) in the Switzerland lead partition, there is no record to be found:

[SanfordWhiteman]

This particular case (regardless of the cause) is starting to shape up like a Support case. Needless to say, it shouldn't be possible to have an invisible lead.

I wonder what would happen if you searched with the API... ?

Anyway, I recommend you use my method of constructing a unique temp email address. See if that helps.

[Dan_Stevens_]

For the record, this has been with support since last week.  They have no idea why this is happening.

I do like your little audit trail enhancement.

[Trish_Keenan1]

Hi Dan. We have the same issue with a Marketo/Dynamics integration. I had a support ticket in months ago about it, and was reassured that everything was working fine... but it's not. We're finding inconsistencies (without any apparent reason) as to the sync working or not, and have the exact same issue with regard to the email address being blank in Dynamics after sync. We're also digging in deeper on this end to determine if there's any logic to what happens when. But you're not alone in this!! Let's stay in touch.

[Dan_Stevens_]

We just heard back from Support on this:

Our escalations team took a look at your webhook, and confirmed with Product Management that this is not a supported method of creating duplicates. A couple of alternative suggestions for this are to set up a service to proxy requests back, or manage duplicate creation on the front end.

Seems odd as this was originally architected by our Marketo consultant and professional services team (Kristen Carmean and John Mattos).

[Trish_Keenan1]

This is a frustrating reply to say the least. Kristen Carmean was also involved with this solution for us. And what does "manage duplicate creation on the front end" mean?  I don't really understand the "set up a service to proxy requests back" option, either.

As far as integration with Dynamics is concerned, the webhook solution is straightforward. If the person is already a contact in Dynamics, create a new lead record. Are they recommending we need a third-party solution to address a pretty routine issue?

I think this needs to be addressed more thoroughly, as it's a common consideration for those of us with Dynamics. That response isn't satisfactory.