Webhook SSL Error - unable to install cert provided, need alternate solution

[ma_mgibbons]

We have a set of 3 Marketo webhooks which have been working fine since September 2024.

The webhooks connect to a data service we screen some marketo records against.

That data service has now required the installation of an SSL cert (provided by the service) on the server executing the webhooks.

Marketo does not install certs on behalf of clients.

As a result, we are currently stuck and unable to run the data screenings.

Does anyone have a similar situation they have solved for?

Any creative solutions, maybe?

Possibly adding an integration layer/middleman server to handle the call to the data service? Just spit-balling at this point.

Any help is greatly appreciated as we have limited resources and are at a standstill.

[SanfordWhiteman]

We use an intermediate service to connect to any SSL endpoint that requires client certs.

Extremely easy to build. We use AWS Lambda, though any platform with an HTTP client, including php_curl or whatever, will do. But you do need someone familiar with such things.

[ma_mgibbons]

Hi Sanford -

Thanks for your reply, this is very helpful

We set up a call with some external developers tomorrow to explore our options and will share this info with them

[SanfordWhiteman]

Cool! For the record, if they quote more than 2h (or 1h really) to write the code itself,* you should prob look elsewhere. This is simple as can be, unless there’s something you haven’t mentioned.

* Figuring out where to host your li’l API gateway app is
as big a question as writing it. Hopefully they already have

a public cloud or hosting provider where they can drop it in.

[ma_mgibbons]

Waiting on the quote now so we'll see what they come back with.

I am admittedly out of my depth with some of this stuff, my strengths are in other areas.

I don't know if you can shed light on this part, or if someone at Marketo can chime in but an engineer with the source we're connecting to via the webhooks had the following to say:

Marketo is likely using a version of Java that does not include the latest available root certificates for SSL connections. Java is updated periodically to include global root certs which are used to validate encrypted connections, when they connect to grid and receive our certificate, the java app will use the root certificate on file to verify the certificate chain. As it appears their server does not have the root certificate required, they are unable to validate, and thus the connection fails to our services.

Whitelisting IP addresses will not change anything as all connections must be encrypted still.

The root certificates required were generated in 2022, but we do see that some java versions (example https://bugs.openjdk.org/browse/JDK-8341057) only began including the root cert in September 2024. Marketo would likely need to update to a java build version post this date to resolve or manually include the latest root certificates on their servers.
---
I've also sent a ticket to support to see if they know if they're on a recent java build as it pertains to this. Honestly not sure if it matters, we likely need to setup the new solution regardless. Thanks again for your time and input on this.
 

[SanfordWhiteman]

OK, this is describing a very different scenario (though as you’ve noted, one that still requires a custom solution if you need it fixed ASAP).

Your original post mentioned a client certificate. In this setup, a cert specific to your account is generated on the server side and needs to be preshared with you — for example, downloaded via a password-protected page. Then that cert is included with all connections. AFAIK, client certs aren’t supported by any product’s generic webhook feature, so Marketo isn’t alone. It can be worked around by using an intermediate gateway as described above.

Your latest post describes a missing CA cert in the CA bundle used by the webhook’s HTTP library. That’s the database of well-known public certificates included in browsers, OSes, runtimes like the Java runtime engine, custom HTTP libraries, etc. Sometimes apps will use the OS-level bundle, other times they have their own internal bundle. And as the engineer notes, they have to be kept up to date, which includes both adding new CAs and removing defunct CAs.

The problem of a missing CA cert is even easier to solve using an intermediate gateway. There you’re talking a 15-minute setup of an AWS CloudFront distribution, for example.

[ma_mgibbons]

You are truly a source of unlimited wisdom! Haha

Thank you again for your insights.

There are a series of internal issues adding complexities to the situation but hopefully we will have this resolved sooner rather than later and your input has been invaluable.