Bulk API - 500M limits and error messages

[Kurt_Koller]

First, I wanted to say thanks to Kenny etc at Marketo for shipping the Bulk API, right now you're hearing/dealing with launch issues but I want to step back a moment and say I'm VERY HAPPY there's a Bulk API at all, and I'm hopeful that this will let us take our integrations to a new level.

Queuing/processing seems to work reliably now, however as of an hour ago our lead exports all seem to fail for us (I've had one of about 20 work, supplied the same query again that worked and it failed). We will try again tomorrow.

I moved on to Activity Export. On this instance, exporting one day of activities (52M) works, exporting a week (should be around 350M) seems to fail. The biggest file we received was 205M of activities for 4 days. Did that 2x and 52M lead export, the next query I submitted I received

        {

            "code": "1029",

            "message": "Export daily quota 500MB exceeded."

        }

How does that work exactly? If I'm at 463M I can't submit a query that the system thinks will take me over 500M? Or any job submitted after that will fail if it's more than 36M? I may have calculated this wrong, but I'm wondering as the documentation is light on this, it says 500M max per day but how does that work exactly.

If this limit is disk-based, and you have 5GB allocated per instance say for 10 days * 500M and that is the primary concern, it would be great if we could issue a DELETE once we pick up a file and continue to work to some other higher transfer cap if necessary. If we are working with an instance that can have 100M or more for a day of activities, there are some issues. I don't see a way to filter out large values in attributes (and sometimes those can be massive), so it would be nice if there was an option to manage disk if that's the underlying concern. This is for scenarios like an initial load of say 2-3 months of data into an analysis product, for instance. If we want to analyze/manage 3 months of activities, once we're caught up there's no issue daily but getting the initial data is tricky depending on how this limit works.

Also, is there any possibility to get errors other than "System error" to help us troubleshoot/work around on our own?

Thanks again to Kenny and team for getting Bulk API shipped!

[Josh_Hill13]

This sounds like a Support ticket.

[Kurt_Koller]

Ok. Apologies, as a programmer I've become accustomed to forums and communities like Stack and github, where people discuss pretty much everything publicly which means other people later can see those discussions and it can help them if they are having similar issues.

I've done more testing and I'll ask someone to submit a case/support ticket about multi-day activity and lead export not working on any instance I've tried so far. Thanks for your reply.

[Anonymous]

No apologies Please share your findings, because it will save me a lot of time. The limit is not clearly documented and it's not reported by the usage endpoint, so it will be hard for us to build the retry logic. For example, if we exhausted the quota for the day, we need to see which time range we missed and sync that part again.

On a side note I also want to say thank you to the team working on the bulk API. This feature exactly addresses our use case.

[Dave_Everly · Accepted solution]

We have added addition information to the Developers Site to help explain file size daily quota.  Hope this helps!

http://developers.marketo.com/rest-api/bulk-extract/#file_size

[Kurt_Koller]

Thanks Dave! It would be great if we could manage the files in daily quota/delete them for more quota. There are some instances where it's 250M-500M a day in activities and getting any kind of history is a very long and arduous process.

Thanks again