Web Services Error 1003 & 606 - Same time every day

[Anonymous]

We receive a Web Services error 1003 every morning at 10:06am.  More often than not, there is also a 606 error at the same time, as well.

I've contacted support, and they were able to tell me the definition of the errors, but unable to explain in non-developer speak or help me identify where the error is coming from. 

I've searched through the batch campaigns, and nothing is scheduled around 10am.  Does the time of the notification even indicate the time the error actually occurs?

In addition, there are 3 Launchpoint applications that I installed, but this error has been happening prior to those being installed. 

I'm not sure how to troubleshoot this from here.  Does anyone have any experience successfully finding the origination of these errors?

[Ryan_Vong7]

Lisa,

The alerts are definitely not when it's happening. I've mentioned this to Marketo since we use another tool to monitor these and no answers yet as to why it would need to wait 24hrs to let someone know that something wrong happened.

Sanford probably has the best approach in figuring out which is the culprit. ..."create new service definitions for Socedo, Knak, and WP and delete the old ones. That will ensure that no other app is borrowing those credentials. Also, turn off SOAP by creating a new Encryption Key"

At least this helps you isolate the specific apps using apis. We have a tool that will look at this info and which api user by the hour. Provides some pretty good clues and so far and what I've seen are a lot of apps doing things in the middle of the night. Seems like everyone thinks the best time to hit apis are between 1 and 4 in the morning. (I think there's some comedy skit about 4am.)

[Anonymous]

ryan - may I ask what tool you are using to monitor your API usage?

[Ryan_Vong7]

Hi Lori,

OMEGA – Automation Intelligence

[Anonymous]

I just discovered a DataFox integration buried in the depths of a data management campaign that I'm pursuing now.  It has some field mappings that I'm thinking may not be set right - mapping fields with an integer or currency type - maybe the data coming over from DataFox isn't the right data type and it's throwing the 1003 error.  It's the best I have to go on at the moment. 

[Steven_Vanderb3]

The errors aren't happening at 10:06 every day; that notification is a roll up of all the errors you received in the past 24 hours.  Support can also tell you the time the errors are occurring and the source of the call, but we can't tell you what exactly is wrong with the call to cause that error without you providing in the ticket a log of the body of the call.  If they aren't/having been providing you a bare minimum of time and user to help you troubleshoot internally then I would escalate that ticket.

1003 error means there's something wrong with the data being sent in a call (for instance, a non-date value being submitted into a date field).  606 error means too many calls are being made in too short a period of time.

[Anonymous]

Thank you - I asked the support rep to provide the time of the errors and source of the call.  If they are able to give me a time, is the next step to go through the results of each batch or triggered campaign and hunt for errors?

[SanfordWhiteman]

Not every API call has results that would show up anywhere else. For example, if I overwhelm limits by fetching the Activity Log, that's not part of a Campaign.

Also, if you go over limits, by definition you won't see corresponding activities over the limit (though you may see a trend before then).

If you're saying this error occurred without any integrations, that doesn't make sense. Something is using your API calls, and it isn't SFDC. I'd assume something like a WP integration getting hit by a bot attack (which is exactly why those integrations shouldn't be used -- they aren't professionally written to deal with of real-world traffic).

[Anonymous]

Apologies, I don't have a deep technical understanding of APIs so a lot of what you're saying I don't quite understand.  I'm trying to troubleshoot based on what I know, which is:

1.  I inherited a Marketo instance in late May, and do not have visibility in to what has been plugged in to it in the past.

2. These web services error have happened every day, beginning at some point in time prior to my coming on board.

3.  In June/July, I have installed Knak, Wordpress (so that our Wordpress guy could embed our web forms), and Socedo.

4.  There were two older integrations in Launchpoint that I removed in late July.  I had suspicions that they were the cause of the web services error, but the errors persist, so that wasn't it.

I understand that SFDC isn't the cause. But I need to figure out what is. Based on what I have access to in my instance, is there a way I can search for what is causing the daily error? 

[SanfordWhiteman]

...I don't have a deep technical understanding of APIs so a lot of what you're saying I don't quite understand.

Point I'm making above is that there's no reason to expect an API call to leave a trace in the logs -- anywhere.

API endpoints like ​Get Lead​ or ​Get Activities ​are simply fetching data from your Marketo db, not altering it, so there is no activity associated with the call, nor any campaign. There's no activity like "Was read via API" (nor should there be, as it has no significance within the db). But you can just as easily use up your API limits with those calls.

The other point I was making is that even with calls that are normally logged, like updating a lead or requesting a Campaign, if you're over limits, naturally nothing would be logged. The call was blocked, and "Attempted to change field but exceeded rate limit" is not an Activity.

Based on what I have access to in my instance, is there a way I can search for what is causing the daily error?

Search, no. But you can create new service definitions for Socedo, Knak, and WP and delete the old ones. That will ensure that no other app is borrowing those credentials.

Also, turn off SOAP by creating a new Encryption Key:

Use a site like Strong Random Password Generator to gen a new key.

[Anonymous]

Thanks - I appreciate your help - Thanks for taking the time to explain it further.  I have a lead on a webhook I was unaware of writing to lead records.  If that's not it, I'll go through the other steps.

[SanfordWhiteman]

Webhook calls don't use API calls. (But if the webhook itself loops back in to the Mkto API it will.)

[Dan_Stevens_]

Here are the error codes and what they mean - for your reference: http://developers.marketo.com/rest-api/error-codes/

[Josh_Hill13]

Those are usually related to the API. What other tools have access or are you using to send out of Marketo?

[Anonymous]

As far as I know, just Salesforce.

[SanfordWhiteman]

You can't have just Salesforce (which doesn't use Mkto API calls) and also have 3 LaunchPoint integrations! (That's a contradiction.)

I assume you're sharing API credentials so you can't tell which app is bursting over the limit. So try disabling one LaunchPoint defined service at a time.

As for the timing, yeah, the time of the alert isn't the time of the infraction.

[Anonymous]

Hi Sanford Whiteman, Josh Hill

How can I see the launchpoint apps?. I have just taken over a new Marketo instance a few days ago and I got this error 2 times.

I see in integrations SFDC and Sales insights.

Furthermore to this my Marketo instance is being terribly slow in campaign execution. Does that have something to do with this?

Thanks

Sara

[Anonymous]

Sanford Whiteman, Josh Hill

WE have installed:

- Google adwords

- Go to webinar

- Marketo facebook lead ads

- My rest service

- Marketo facebook

I will appreciate your feedback on the matter as I have seen you have quite some knowledge

Thanks

Sara

[SanfordWhiteman]

A few HTTP 1003 errors aren't slowing down your instance (606 errors, by contrast, spell overuse of the API and can correspond to instance slowness).

You mention "My REST service" but do you not have logs from this service to tell if it has seen the 1003?

[Anonymous]

Many thanks for your answer .Does that look normal to you??          Sanford Whiteman

Nay recommendations? Our Marketo queue is huge and leads take ages to get push to SFDC

[Anonymous]

I thought he meant in addition to the 3 Launchpoint apps.  Those are Knak, Wordpress, and Socedo - which were all installed (by me) after this error started happening.  If it's not Salesforce, then I'm not sure how to tell what else could be causing it.