Marketo Webhooks With Marketo API

[Anonymous]

Odd question but I am wondering if its possible to use a marketo webhook to call and return data with either the soap or rest api for the same instance in marketo.

I have a unique use case where I want to merge two leads but need to do it in a smart campaign and I'm wondering if I can do that all contained within marketo or if I will need to use an external script to accomplish this.

If anyone has code examples that would be super as well I'm not very proficient with api syntax and webhooks.

[Anonymous]

We wound up taking another route that was more scalable.  Thanks everyone for your help.  After consulting with Marketo experts we found out that webhooks do not operate in a batch function so large imports would take too long.

[John_M]

Actually, you cannot do this as far as I'm aware. Specifically, you cannot call a specific instance of Marketo using SOAP or REST using the webhook in that same instance.

[Anonymous]

Up until very recently (like, middle of last week) I was actually using exactly this structure and it was flawless.

We get custom events from our SaaS platform sent via Segment.io, and they show up in our lead activity log as "visited web page: domain.com/events/Name-of-Custom-Event". I then used a smart campaign triggered by the visit to that web address to call a webhook which submitted form in our instance using a POST request. So the lead activity record would subsequently have a corresponding "filled out form: capture - name of custom event" line item.

We did this in order to store events longer term, as web page visits are archived after 90 days. And of course we could still filter via date, number of occurrences, etc as needed. Then starting last week (with no warning or heads up from Marketo, I might add) they disabled this as an option. Now of course we're scrambling to find an alternative solution.

Danny Essner​ I believe it may be possible to pass data between instances still using only webhooks - You'd need to get the app id and munchkin id for the Marketo instance you want to send data into, but I've done this in the past (please let me know if you'd like more info).

[SanfordWhiteman]

I don't understand what you're saying stopped working.  Unless there's something buggy in your Marketo instance in particular, a loopback form post from a Marketo webhook to the same instance still works just fine.

This isn't, of course, actually using an API call (that was the open question). You're just using the standard forms endpoint (http://localhost/index.php/leadCapture/save) not an API.  And, like I said, still works.

[Anonymous]

Ok, quick update (I know David Horne​ found a different solution but this may help other people in the same situation. Big thanks to Sanford Whiteman​ for walking through this with me.

The following structure for a webhook appears to work (at least for now):

URL: set to http://localhost/index.php/leadCapture/save

Request type: POST

Template: include required parameters (munchkinId, formid) and any values or tokens for other fields that you want to include in the POST

Encoding: set to Form/URL

[Anonymous]

The response I got from Marketo Support:

I had a chance to speak with one of the Senior Engineers and was informed that using webhooks to POST into a Marketo Form in this fashion is no longer supported. Using a webhook in this fashion has the potential to create an infinite loop of activity and can become resource intensive.

So either it's truly no longer supported to webhook to your own instance, or they disabled it for us. Realistically I can think of a couple of ways to take an infinite loop of activity within Marketo without involving a webhook so I'm not sure why that would be a concern. I realize that it's an unsupported feature but "unsupported" shouldn't mean "kill it at random with no warning" which is what happened in our case. My ticket is still open though so maybe I'll hear more from them.

[SanfordWhiteman]

Also, is it possible that there was something in your implementation that never should've worked in the first place?  I ask because you weren't using localhost in the the snippet you posted in another thread.  In this case, maybe you're seeing a coincidence: Marketo locked down communication from inside their network to the external IP of your pod/instance and thus broke your webhook, but it would still work with actual loopback.  Just an idea.

Speaking of ideas, though, why not use an Interesting Moment for this?  Or a datetime field on the lead LastCustomEvent{EventName}?

[SanfordWhiteman]

So either it's truly no longer supported to webhook to your own instance, or they disabled it for us.

That's overly punitive, if so.  Almost bizarre given that such webhooks have not only been used in the wild for a while, but were endorsed in a blog post from 2 months ago that's still up! I know I was at a client last month and saw one that someone had implemented in the past.  I just take it as a matter of course that people still do this, even if I don't generally use it.

Realistically I can think of a couple of ways to take an infinite loop of activity within Marketo without involving a webhook so I'm not sure why that would be a concern.

I agree, I find that justification unconvincing.  But more important, all that's being addressed (if it has been disabled for your instance, it hasn't for us) is a webhook that literally uses the loopback address to post without leaving the Marketo network.  This says nothing about leaving the network and echoing the request back in, which would be extremely easy to set up.  In 5 minutes, you could put up a reverse proxy that does just that.  You can PM me about it if you want.

[Anonymous]

Hi John,

Is it possible to exchange lead information between two separate Marketo instances using webhooks and APIs?

thanks,

Danny

[SanfordWhiteman]

I've heard (via a friend) that Professional Services had a secret recipe for the same instance. But you could just pass through a dumb reverse proxy in either case, so that's not the problem. The problem is still gonna be managing it all using only Flows... I'm sure you'll agree.

[Elliott_Lowe1]

A Marketo Professional Services consultant did this with me a couple of years ago, but I haven't had a chance to implement this yet.

[John_M]

If you Do get it working, I'll be happy to be prove wrong! Be sure to post how you did it!

[John_M]

Im in professional services on the enterprise team, and my co workers and I are unaware of his currently being possible. You can try it out simply enough. Youll get a timeout.

[Josh_Hill13]

I believe there is an example on RingLead's site if you look for their webhook guide.

What precisely do you want to do? Marketo dedupes emails before it sends any emails out. What you want may already exist with Ringlead or another tool.

[Anonymous]

Hi Josh thanks for your comment.

I am creating a contact record in salesforce via the rest api and using the data from a new lead in marketo to generate it.  Once the interval sync occurs that will cause there to be two leads in marketo with the exact same information as marketo doesn't allow me to force a relationship between those records at the point of creation.

So I need a programmatic way to cause a merge of the original marketo lead that triggered the contact creation webhook and the new marketo lead that was generated from the interval sync so I don't have duplicate records in marketo.  Does that help clarify? Our company creates contacts first in salesforce which is confusing I know but this would be a workaround that would allow that to happen.

Thanks,

Dave.

[SanfordWhiteman]

David, what are the conditions that prohibit you from waiting < 5m for the Marketo lead to sync to SFDC?

Not to say I haven't seen a similar situation -- in fact, we are dealing with one now.  But that's because we have a form creating users in SFDC and we can't switch that form to be a Mkto form.

I'd like to know the particulars of your situation because there may be a better approach.

While it is possible, after a fashion, to call Mkto APIs from Marketo 'hooks, it is clumsy, not particularly scalable, and if the flows are not managed tightly will lead to infinite loops.

[Jon_Wu]

Sanford, I have this exact issue, and with some help from Marketo, we solved this in a rather brute force way. I'm wondering if you have any more efficient or reliable ways to handle this.

We create leads in Salesforce and Marketo simultaneously in many cases from our e-commerce site. Any time you enter your email on our site (usually not in a Marketo form), we add you to Salesforce and our other account systems for CRM and tracking, and call associateLead in Munchkin so that all the user's activity is associated with them immediately in all systems.

Of course, since Marketo doesn't deduplicate Salesforce synced leads, this will result in a duplicate within 5 minutes when the Salesforce lead is synced back to Marketo.

To ditch the duplicate as quickly as possible to avoid conflicts in behavioral scoring or unsubscribe preferences, or other fields, we use a Smart Campaign to fire a webhook to our own endpoint, which deduplicates the lead by finding all leads with the same email, and merging them all into the Salesforce one. This in theory works decently (not really, see below), but it still uses 2 API calls and isn't ideal from that perspective, because we'll hit API limits if we suddenly appear on national TV or have some unexpectedly large event, but perhaps the limits issue is just unavoidable with the way Marketo limits work.

The worst part is, this doesn't actually work reliably due to data consistency bugs on Marketo's side. When we first implemented this, our find dupes, then merge behavior wasn't working consistently. After talking with support, we determined that the API to find leads didn't consistently return all the leads with the same email address if the leads were recently created. To get around this, we fire a deduplication webhook as normal, but then wait 3 minutes and do it again, in case the first one didn't work.

In rarer cases, a lead may appear in Salesforce, but not appear in Marketo, so we also have to deduplicate all new leads that appear in Marketo, that aren't from Salesforce. Again, we have the same consistency bug with the API not returning all leads, so we try once, wait 3 minutes, then try again.

In all, this means every lead that gets created results in 4 API calls to find duplicates (1 for the lead starting in Marketo, 1 for the Salesforce lead synced into Marketo, and double those for retries), and 1 API call to merge them. It's been working fairly reliably (except when we hit API concurrency limits), but the number of API calls is way too many.

While we could always wait 3 minutes, this doesn't bode well for users on multiple devices doing things really quickly in real time. The longer a lead is a duplicate, the more behavioral and scoring data will be lost during the merge.

There's got to be a better way... Any suggestions?

[SanfordWhiteman]

Jon, there's certainly a more efficient way! You're in a not-rare -- not to say "common," I guess -- situation, where the lead is known to SFDC first, but you want to associate the Munchkin session. This happens with Web-2-Lead and 3rd-party forms and other environments as well.

The answer is: don't call associateLead from the client (nor from your server). Let the sync create the person in Marketo.  But include on the lead a field that stores the Munchkin cookie value (_mkto_trk). When the lead is created or when this field changes, use a webhook from Marketo to associate the cookie with the lead. Takes one API call total.

We actually have a field that holds multiple Munchkin cookies (in case the person goes in from more than one device before the next sync, unlikely but possible). But you can work with just one at a time in most cases.

[Jon_Wu]

Hi Sanford,

That's great advice. I almost had something like that working with an old localhost trick on the Marketo end, but I think that stopped working way back and it seemed error-prone as if we were being too sneaky. I'm glad you mentioned multiple cookies since I had this concern as well.

If I'm understanding correctly, you'd have a long string lead field that could hold multiple cookies and you can just delimit them by something and accumulate them in the Salesforce lead. Then, when the lead is created or the field changes in Marketo, you'd use a webhook from Marketo to call a custom API that we could build, that would call the REST API (http://developers.marketo.com/rest-api/endpoint-reference/lead-database-endpoint-reference/#!/Leads/...​) for each cookie value seen in the lead. Is that the right idea? Then, you could remove all those cookies from the Salesforce lead so the field doesn't fill up and we know that those cookie values have been associated.

If I'm on the right track so far, what happens in these 2 cases?

1. Visitors come back to the site on a new device, but we want to associate their tracking cookie

If we go through the whole process again and wait for the cookie value to change, we end up using 1 API call for every new visitor. Is this would you had in mind?

It seems like we could get around this by having our Salesforce lead creation API let us know if the lead was already processed, synced, and associated into Marketo, so we'd know it'd be safe to call associateLead from Munchkin without creating a duplicate. This still uses 1 Salesforce API call per visit, but there's not a big way around that, and adding capacity there is fairly cheap. However, I wanted to see if you had other ideas.

2. What happens if somebody is filling out a Marketo form? It seems like there's no way to avoid them getting associated and created in Marketo immediately, unless I basically don't use Marketo forms.

We just started using Marketo forms and it's causing a lot of headache with sync conflicts. However, unless we don't actually use a Marketo form eve,  and just build our own forms and APIs that insert into Salesforce directly, I wasn't sure how to avoid creating leads in Marketo and Salesforce at the same time. In this case we also create the Salesforce lead immediately because we load a promotion into their Salesforce / e-commerce account once they fill out the form.

Thanks so much for your help!

Jon

[SanfordWhiteman]

That's right, Jon, you have the idea. I use a JSON array because I'm like that, but could be a semicolon-delimited list (since these are cookies, you can't have a delimiter collision w/semicolons).

Then I put the already-processed cookies in a history field so they don't run through again.

Like you, I'm not convinced the loopback form post is here to stay. Not that it's deprecated exactly, but that it's not documented enough internally and that it could go away due to network/DNS changes, etc.  But you can do the same thing (and ensure that it'll still work) by bouncing off a reverse proxy and back in. Again, if you wanted to go that route, which does have a certain appeal since it doesn't use API calls.

To the 2 cases you mention:

Jon Wu wrote:

1. Visitors come back to the site on a new device, but we want to associate their tracking cookie

If we go through the whole process again and wait for the cookie value to change, we end up using 1 API call for every new visitor. Is this would you had in mind?

It seems like we could get around this by having our Salesforce lead creation API let us know if the lead was already processed, synced, and associated into Marketo, so we'd know it'd be safe to call associateLead from Munchkin without creating a duplicate. This still uses 1 Salesforce API call per visit, but there's not a big way around that, and adding capacity there is fairly cheap. However, I wanted to see if you had other ideas.

Like you said, don't see a way around this without using at least one SFDC API call, since that's the only way you can then save on Mkto calls. Check to see if the lead has made the roundtrip into Marketo and back to SFDC.  Otherwise, pass the cookie again over to Marketo.

2. What happens if somebody is filling out a Marketo form? It seems like there's no way to avoid them getting associated and created in Marketo immediately, unless I basically don't use Marketo forms.

We just started using Marketo forms and it's causing a lot of headache with sync conflicts. However, unless we don't actually use a Marketo form eve, and just build our own forms and APIs that insert into Salesforce directly, I wasn't sure how to avoid creating leads in Marketo and Salesforce at the same time. In this case we also create the Salesforce lead immediately because we load a promotion into their Salesforce / e-commerce account once they fill out the form.

You can use a Marketo form merely as a transport mechanism. That is, you pass a random GUID as the Email and put the real Email (along with everything else) in custom fields. Then a webhook calls the SFDC API to create the actual lead first over there, and you delete the temporary lead in Marketo.  It's... messy but works.