Web Services Error 1003 & 606 - Same time every day

[Steven_Vanderb3]

The errors aren't happening at 10:06 every day; that notification is a roll up of all the errors you received in the past 24 hours.  Support can also tell you the time the errors are occurring and the source of the call, but we can't tell you what exactly is wrong with the call to cause that error without you providing in the ticket a log of the body of the call.  If they aren't/having been providing you a bare minimum of time and user to help you troubleshoot internally then I would escalate that ticket.

1003 error means there's something wrong with the data being sent in a call (for instance, a non-date value being submitted into a date field).  606 error means too many calls are being made in too short a period of time.

[Anonymous]

Thank you - I asked the support rep to provide the time of the errors and source of the call.  If they are able to give me a time, is the next step to go through the results of each batch or triggered campaign and hunt for errors?

[SanfordWhiteman]

Not every API call has results that would show up anywhere else. For example, if I overwhelm limits by fetching the Activity Log, that's not part of a Campaign.

Also, if you go over limits, by definition you won't see corresponding activities over the limit (though you may see a trend before then).

If you're saying this error occurred without any integrations, that doesn't make sense. Something is using your API calls, and it isn't SFDC. I'd assume something like a WP integration getting hit by a bot attack (which is exactly why those integrations shouldn't be used -- they aren't professionally written to deal with of real-world traffic).

[Anonymous]

Apologies, I don't have a deep technical understanding of APIs so a lot of what you're saying I don't quite understand.  I'm trying to troubleshoot based on what I know, which is:

1.  I inherited a Marketo instance in late May, and do not have visibility in to what has been plugged in to it in the past.

2. These web services error have happened every day, beginning at some point in time prior to my coming on board.

3.  In June/July, I have installed Knak, Wordpress (so that our Wordpress guy could embed our web forms), and Socedo.

4.  There were two older integrations in Launchpoint that I removed in late July.  I had suspicions that they were the cause of the web services error, but the errors persist, so that wasn't it.

I understand that SFDC isn't the cause. But I need to figure out what is. Based on what I have access to in my instance, is there a way I can search for what is causing the daily error? 

[SanfordWhiteman]

...I don't have a deep technical understanding of APIs so a lot of what you're saying I don't quite understand.

Point I'm making above is that there's no reason to expect an API call to leave a trace in the logs -- anywhere.

API endpoints like ​Get Lead​ or ​Get Activities ​are simply fetching data from your Marketo db, not altering it, so there is no activity associated with the call, nor any campaign. There's no activity like "Was read via API" (nor should there be, as it has no significance within the db). But you can just as easily use up your API limits with those calls.

The other point I was making is that even with calls that are normally logged, like updating a lead or requesting a Campaign, if you're over limits, naturally nothing would be logged. The call was blocked, and "Attempted to change field but exceeded rate limit" is not an Activity.

Based on what I have access to in my instance, is there a way I can search for what is causing the daily error?

Search, no. But you can create new service definitions for Socedo, Knak, and WP and delete the old ones. That will ensure that no other app is borrowing those credentials.

Also, turn off SOAP by creating a new Encryption Key:

Use a site like Strong Random Password Generator to gen a new key.

[Anonymous]

Thanks - I appreciate your help - Thanks for taking the time to explain it further.  I have a lead on a webhook I was unaware of writing to lead records.  If that's not it, I'll go through the other steps.

[SanfordWhiteman]

Webhook calls don't use API calls. (But if the webhook itself loops back in to the Mkto API it will.)

[Anonymous]

I just discovered a DataFox integration buried in the depths of a data management campaign that I'm pursuing now.  It has some field mappings that I'm thinking may not be set right - mapping fields with an integer or currency type - maybe the data coming over from DataFox isn't the right data type and it's throwing the 1003 error.  It's the best I have to go on at the moment. 

[Ryan_Vong7]

Lisa,

The alerts are definitely not when it's happening. I've mentioned this to Marketo since we use another tool to monitor these and no answers yet as to why it would need to wait 24hrs to let someone know that something wrong happened.

Sanford probably has the best approach in figuring out which is the culprit. ..."create new service definitions for Socedo, Knak, and WP and delete the old ones. That will ensure that no other app is borrowing those credentials. Also, turn off SOAP by creating a new Encryption Key"

At least this helps you isolate the specific apps using apis. We have a tool that will look at this info and which api user by the hour. Provides some pretty good clues and so far and what I've seen are a lot of apps doing things in the middle of the night. Seems like everyone thinks the best time to hit apis are between 1 and 4 in the morning. (I think there's some comedy skit about 4am.)

[Anonymous]

ryan - may I ask what tool you are using to monitor your API usage?