26 Replies Latest reply on Jul 13, 2017 10:25 AM by Dan Stevens

    Webhook to create purposeful duplicates in Marketo is unreliable

    Dan Stevens

      As part of our purposeful duplicate process (when CRM contacts are created in MS Dynamics) a basic webhook is called to create a fresh new lead:

       

       

       

      We're finding that this is working very inconsistently for us.  Sometimes it works, sometimes it doesn't.  What's odd is that we always receive a common error type "Duplicate Lead" Failed. Server Returned code 302" - even when the webhook runs successfully.  When I asked Support about this, they came back with the following:

       

      Engineering confirmed that the HTTP 302 message is not an error. Instead, it is the response code for a page redirect.

       

      HTTP 302:

      https://en.wikipedia.org/wiki/HTTP_302

       

      With that being said; I've asked if Engineering can modify the wording being written in the activity log, to prevent any future confusion. I also pointed out that you are not aware of why the redirect is taking place in the first place, and asked if they could explain that as well.

       

      But this doesn't explain why the webhook is working so inconsistently.  I know Sanford Whiteman commented last year that the loopback post is broken on some instances (The 3 Coolest Webhooks I've seen (and or) Implemented ).  Could this be the issue?  Any others experience this inconsistent behavior?

        • Re: Webhook to create purposeful duplicates in Marketo is unreliable
          Sanford Whiteman

          302 is the expected response from the legacy /save endpoint, though.

           

          If it weren't, then old + noscript forms wouldn't work: the /save endpoint has to allow for browsers without JS to redirect to a follow-up page, and the only way to do that is via an HTTP redirect (should be HTTP 307 these days, but that's another matter). The new /save2 endpoint doesn't do this, because it requires JS support in the browser in the first place.

           

          So I wouldn't be concerned about the 302.  I'd be looking at whether the call is actually successful. If it's failing, are you sure you're not going over the limit of 1 form post every 2 seconds?

          1 of 1 people found this helpful
            • Re: Webhook to create purposeful duplicates in Marketo is unreliable
              Dan Stevens

              We're still trying to find out what this issue is here.  But when doing some further investigating, we came across another issue: when the webhook creates the duplicate lead, it will populate the email address with the email-address-temp (since you cannot create duplicate by populating the email address immediately for obvious reasons), but then the email address only exists when looking at the lead detail view.  In the lead grid, the email address is blank.  And the lead - which now resides in one of the country lead partitions - isn't even retrievable (e.g., via smart list) in that partition.  Even when searching by last name.  We're not sure of the extent of how large this issue is just yet.

               

              As for ensuring we're not going over the limit of 1 form post every 2 seconds, how can we delay this?  This would be most problemsome when a batch of contacts are added to CRM - which would trigger the webhook call.  I don't think a wait step would affect this here.

              1 of 1 people found this helpful
                • Re: Webhook to create purposeful duplicates in Marketo is unreliable
                  Sanford Whiteman

                  but then the email address only exists when looking at the lead detail view. In the lead grid, the email address is blank

                  Even after running it through the flow to copy Temp Email → Email? That doesn't make much sense, I mean, data sent to /save2 is literally the same path taken by a true client-side data.

                   

                  However, there is something in your flow that I wouldn't do, and that is post the data without an Email. I'd send {{lead.id}}-{{system.datetime}} as the Email. You're going to change it anyway, and this creates a nice little audit trail while ensuring uniqueness.

                   

                  As for ensuring we're not going over the limit of 1 form post every 2 seconds, how can we delay this?

                  You can't directly delay it -- as you point out, a Wait Step would not help, since the event of a flood of new contacts you'd just be delaying the whole flood by a certain amount of time.

                   

                  If you can't batch up these 'hooks (which would require major rearchitecture) you can use the method I describe here: https://nation.marketo.com/message/154866-re-throttle-web-hook-requests#comment-154866

                    • Re: Webhook to create purposeful duplicates in Marketo is unreliable
                      Dan Stevens

                      So here's a sample record where the email address exists, but does not show up in the lead grid, nor the company lead partition:

                       

                      And here are her records in the lead grid (the highlighted one with the missing email address in the record above):

                       

                      And when searching her record (by last name) in the Switzerland lead partition, there is no record to be found:

                       

                        • Re: Webhook to create purposeful duplicates in Marketo is unreliable
                          Sanford Whiteman

                          This particular case (regardless of the cause) is starting to shape up like a Support case. Needless to say, it shouldn't be possible to have an invisible lead.

                           

                          I wonder what would happen if you searched with the API... ?

                           

                          Anyway, I recommend you use my method of constructing a unique temp email address. See if that helps.

                            • Re: Webhook to create purposeful duplicates in Marketo is unreliable
                              Dan Stevens

                              For the record, this has been with support since last week.  They have no idea why this is happening.

                               

                              I do like your little audit trail enhancement.

                                • Re: Webhook to create purposeful duplicates in Marketo is unreliable
                                  Trish Keenan

                                  Hi Dan. We have the same issue with a Marketo/Dynamics integration. I had a support ticket in months ago about it, and was reassured that everything was working fine... but it's not. We're finding inconsistencies (without any apparent reason) as to the sync working or not, and have the exact same issue with regard to the email address being blank in Dynamics after sync. We're also digging in deeper on this end to determine if there's any logic to what happens when. But you're not alone in this!! Let's stay in touch.

                                  • Re: Webhook to create purposeful duplicates in Marketo is unreliable
                                    Dan Stevens

                                    We just heard back from Support on this:

                                     

                                    Our escalations team took a look at your webhook, and confirmed with Product Management that this is not a supported method of creating duplicates. A couple of alternative suggestions for this are to set up a service to proxy requests back, or manage duplicate creation on the front end.

                                     

                                    Seems odd as this was originally architected by our Marketo consultant and professional services team (Kristen Carmean and John Mattos).

                                      • Re: Webhook to create purposeful duplicates in Marketo is unreliable
                                        Trish Keenan

                                        This is a frustrating reply to say the least. Kristen Carmean was also involved with this solution for us. And what does "manage duplicate creation on the front end" mean?  I don't really understand the "set up a service to proxy requests back" option, either.

                                         

                                        As far as integration with Dynamics is concerned, the webhook solution is straightforward. If the person is already a contact in Dynamics, create a new lead record. Are they recommending we need a third-party solution to address a pretty routine issue?

                                         

                                        I think this needs to be addressed more thoroughly, as it's a common consideration for those of us with Dynamics. That response isn't satisfactory.

                                          • Re: Webhook to create purposeful duplicates in Marketo is unreliable
                                            Dan Stevens

                                            I hear you, Trish - very frustrating.  It almost sounds like a cop-out reply since they can't figure out why this is happening.  I'm hoping Kristen or John can chime in here.  If anything, this is the most basic type of webhook - nothing fancy here.  I replied back to the support ticket asking what they mean by "manage duplicate creation on the front end".

                                             

                                            Paul Wilson - do you use this approach with any of your customers?

                                              • Re: Webhook to create purposeful duplicates in Marketo is unreliable
                                                Trish Keenan

                                                Hi Dan - Kristen left Marketo last year, unfortunately. I agree that this just isn't a good enough answer. Can you ask that it be escalated further? (Thank you!)

                                                • Re: Webhook to create purposeful duplicates in Marketo is unreliable
                                                  Sanford Whiteman

                                                  Let me explain why something like this would be unreliable unless specifically part of their infrastructure plan (as opposed to a coincidental byproduct of certain infrastructure choices, which is why it seems to work sporadically).

                                                   

                                                  Webhooks originate from servers inside Marketo's network. Those servers may or may not be the same pod that your instance is on.  Before the webhook connects, it resolves the hostname using DNS.  In most organizations, one or more of the following is true:

                                                   

                                                  • a public hostname (like app-zz.marketo.com) does not resolve internally at all, so the DNS lookup fails
                                                  • a public hostname resolves to a private IP address from inside the network (i.e. using internal DNS servers) but that IP addy doesn't accept inbound connections and/or is a different actual server from the intended destination
                                                  • a public hostname resolves to the usual public IP address, but due to firewall rules that IP address isn't translated to a private IP or isn't accessible from inside. (Some firewalls technically can't make connections that go both in and out of the same internal port, other cases they're manually set up to block such connections.)

                                                   

                                                  The sum of all the above is that you can't literally connect to "app-zz.marketo.com" from servers inside Marketo because of a combination of DNS and TCP/IP routing problems.

                                                   

                                                  But you can try to connect to the same server that's referred to by "app-zz.marketo.com" but using a different name. Let's say the canonical name of that server is DANSERVER and its private addresses are 1.2.3.4 and 127.0.0.1 (the second is the loopback address present on all machines).

                                                   

                                                  We customers wouldn't know the internal address 1.2.3.4 (Marketo wouldn't publish this), and we wouldn't even know the official name is DANSERVER, but we would know that if we were making an outbound connection from DANSERVER to DANSERVER then "localhost" would resolve to 127.0.0.1 and 127.0.0.1 is one of the local IPs. This is knowable because it's an industry standard that's true 99.999% of the time. For example, on your personal PC or Mac, "localhost" points to your machine. Think of it as the universal hostname.

                                                   

                                                  But this only works if you going from a machine, to that same machine.  We know we want this webhook to connect to your pod (that's where a form listener is to make these duplicates). If the webhooks are also going from your pod, then connecting to "http://localhost/index.php/leadCapture/save" is going both from and to that same server, so it should work.

                                                   

                                                  But there's no guarantee that webhooks will be sent from your pod server itself -- nor could there be, if Marketo's architecture is going to scale. It stands to reason that some or all webhooks will be sent from other servers. On those servers, "localhost" is no longer your pod, it's the specialized webhook server itself, which doesn't do form processing. Hence the call would fail, since it would be connecting toi a server that can't do what you want it to do.

                                                   

                                                  Other aspects which can confound connections to localhost are

                                                  • when the local server requires SSL for all connections: unless the server is very deliberately set up to accommodate this scenario, the secure connection will fail, since a public SSL cert can't cover the name "localhost"
                                                  • when the local webserver doesn't bind (a.k.a. attach/listen) to 127.0.0.1 at all, deliberately excluding it, or if the service bound to 127.0.0.1 isn't the one you want.

                                                   

                                                  When Marketo talks about using a proxy, they mean a server outside Marketo that connects back in using your public pod name (and public IP address). In contrast to the localhost connection, this kind of connection will always work, since from Marketo's perspective it's just a form post from some outside address. The syntax of the webhook is exactly the same, it just goes to a special outside hostname. We actually have several such proxies already in place for different pods. If you tell me which pods you guys are on you can just use our proxy.

                                                  3 of 3 people found this helpful
                                • Re: Webhook to create purposeful duplicates in Marketo is unreliable
                                  Jep Castelein

                                  Hi Dan,

                                   

                                  I created the 'localhost' duplicate solution you are using in 2011 and it has always been an unsupported solution. However, it has worked successfully ever since. The '302' response is the expected response. It is because the Form is trying to redirect you to the thank you page. There may be a delay in the new record showing up in the search due to the indexing delay. It is fine to use this solution as long as it works, but indeed, if anything goes wrong, support won't be able to help with this.

                                   

                                  It would be a better solution solution to use a Webhook to call a proxy service outside of Marketo, which then creates the new Lead record in Marketo using the REST API. Azure Functions would be an ideal environment to build such a service (Azure Functions—Serverless Architecture | Microsoft Azure ). Potentially even better is to add the Lead to a Static List and have the external service poll the List. This is more scalable and allows for better error handling and retries.

                                   

                                  Nevertheless, based on the write-up above I am not sure that anything is going wrong in your setup. The 302 message is expected and the Lead not showing up in the search could be due to indexing delay, If there really is a Lead visibility issue, it should also occur with Leads created via normal Form submissions.

                                   

                                  Hope this helps.

                                  Jep

                                    • Re: Webhook to create purposeful duplicates in Marketo is unreliable
                                      Dan Stevens

                                      Hi Jep - appreciate your reply here.  We learned a while back that the 302 message was not an actual error.  But what we found was that there were certain leads that didn't get duplicated - even though they qualified for the trigger campaign that kicks off the process.  Who knows - maybe it was one of the issues with trigger campaigns - and not the webhook - but nevertheless, the duplicate didn't get created.  There was no pattern or timeframe - completely random/inconsistent.   

                                       

                                      To be on the safe side, it's probably best that we develop a more scalable, reliable and supported approach.  We have some professional services hours that we need to use before the end of August.  Do you have the time (and be willing) to help build this solution?  I have a call with our internal team next week - to see if it's something they can support.  But they may not be familiar with Marketo or working with the API.  Send me an email if you want to discuss further (daniel.r.stevens@avanade.com).