5 Replies Latest reply on Aug 3, 2017 10:50 AM by Kurt Koller

    Bulk API - 500M limits and error messages

    Kurt Koller

      First, I wanted to say thanks to Kenny etc at Marketo for shipping the Bulk API, right now you're hearing/dealing with launch issues but I want to step back a moment and say I'm VERY HAPPY there's a Bulk API at all, and I'm hopeful that this will let us take our integrations to a new level.

       

      Queuing/processing seems to work reliably now, however as of an hour ago our lead exports all seem to fail for us (I've had one of about 20 work, supplied the same query again that worked and it failed). We will try again tomorrow.

       

      I moved on to Activity Export. On this instance, exporting one day of activities (52M) works, exporting a week (should be around 350M) seems to fail. The biggest file we received was 205M of activities for 4 days. Did that 2x and 52M lead export, the next query I submitted I received

              {

                  "code": "1029",

                  "message": "Export daily quota 500MB exceeded."

              }

       

      How does that work exactly? If I'm at 463M I can't submit a query that the system thinks will take me over 500M? Or any job submitted after that will fail if it's more than 36M? I may have calculated this wrong, but I'm wondering as the documentation is light on this, it says 500M max per day but how does that work exactly.

       

      If this limit is disk-based, and you have 5GB allocated per instance say for 10 days * 500M and that is the primary concern, it would be great if we could issue a DELETE once we pick up a file and continue to work to some other higher transfer cap if necessary. If we are working with an instance that can have 100M or more for a day of activities, there are some issues. I don't see a way to filter out large values in attributes (and sometimes those can be massive), so it would be nice if there was an option to manage disk if that's the underlying concern. This is for scenarios like an initial load of say 2-3 months of data into an analysis product, for instance. If we want to analyze/manage 3 months of activities, once we're caught up there's no issue daily but getting the initial data is tricky depending on how this limit works.

       

      Also, is there any possibility to get errors other than "System error" to help us troubleshoot/work around on our own?

       

      Thanks again to Kenny and team for getting Bulk API shipped!