Lead enrichment - how many webhook calls is too many?

[Marc_Asmus]

I'm interested in enriching every lead in my Marketo database daily with data from a different tool within our tech stack. 

I'm running into an issue with the native integration between this tool and Marketo that's causing duplicates to be created within Marketo. So, I'm exploring alternatives. 

One idea I have is to configure Marketo to "ask" the tool for the data about each lead. Because Marketo would only ask for data about leads who already exist in Marketo, I believe I'd eliminate the risk of duplicates. I'm looking at leveraging Marketo webhooks to make this happen. Going this route would mean sending tens of thousands of webhooks from Marketo to the tool every day. I'd schedule so that this process ran overnight or on the weekends when there's less happening within our Marketo instance. 

• How many webhooks are too many? What are the limitations?
• Is this a bad idea? Why?
• Ideas for a better way to solve?

[SanfordWhiteman · Accepted solution]

There's no hard limit. It depends on how fast each call returns (up to the max of 30s) because only 20 connections can be open simultaneously.

If your calls respond in under 5s I wouldn't worry about making tens of thousands per day. Gets a little hairy above 50,000.

[Marc_Asmus]

Thanks for the reply @SanfordWhiteman!

Does Marketo provide visibility into how fast calls return?

I ran a few tests through Postman that suggest these calls take <1 second.

Can you describe what gets hairy above 50k calls? Is it that Marketo needs time to process all the calls and could delay subsequent flow steps or other processes?

Assuming 1 second/call...

100k calls * 1 sec/call = 100k seconds = ~1,667 minutes = ~28 hours

Does 20 simultaneous connections mean I could divide that ~28 hours by 20 to estimate those 100k calls might take ~1.4 hours?

Related:

https://developers.marketo.com/webhooks/

Webhook execution time is at the mercy of the response time of the service being used and can result in long campaign execution delays.  Even if a service only takes 50ms to execute, that is 1.5 hours when executed 100,000 times.

[SanfordWhiteman]

Yes, divide by 20!

The problems arise when you're running people through a daily campaign and the campaign takes nearly a day (let alone more) to finish! You want to keep your runs under ~4h, whatever they do, to make sure you can deliver usable results every day.