6 Replies Latest reply on Nov 21, 2017 5:16 AM by Jason Ferrell

    Rest API - Bulk Import Status

    Jason Ferrell

      I'm having an issue tracking the status of bulk imports and I'm curious if anyone else has had the same issue. 

      When I create a bulk import and add to a static list, it appears that the batch id that is assigned to the action is the same as the list id.  I think that the bulk import has a queue of up to 10 jobs and can run 2 simultaneously.  However, since all the jobs get the same batchid, when I try to check the progress, I get odd results.  I can generally work around this for large imports that require multiple jobs.  It does create an issue when I try to check the status and it gives me the status of the job from yesterday.

      /bulk/v1/leads.json?access_token=123&format=csv&listId=22038&lookupField=email

        • Re: Rest API - Bulk Import Status
          Kenny Elkington

          Do you have any examples demonstrating this?

            • Re: Rest API - Bulk Import Status
              Jason Ferrell

              I'm not exactly sure what you're looking for.  When you upload a CSV to the bulk import using the Rest API link shown above, the result is a JSON object that contains the requestId and three elements of the result:  batchId, importId, and status.  When uploading to a static list, the batchId and the importId are always the same number as the static list id.  For example, above I'm uploading to static list 22038.  When I create upload jobs for this list, both the batchId and the importId are always returned as 22038. 

              This causes the status to be imperfect, since I don't get a unique job number for each upload.  There appear to be no parameters in the API to force a unique batchId for this type of upload.

                • Re: Rest API - Bulk Import Status
                  Kenny Elkington

                  Sorry if I was unclear, but I'd like to see some of the example requests and responses where you're observing this behavior, rather than just a description.

                    • Re: Rest API - Bulk Import Status
                      Jason Ferrell

                      Here is a set of actions via R...attempt to upload file "01", then attempt to upload file "02"...appears to fail indicating that queue is not work...then a successful upload of file 02, which results in teh same batchId (equal to the listid).

                       

                      This is the url string I'm using:

                      https://947-IOX-305.mktorest.com/bulk/v1/leads.json?access_token=ABCDEF&format=csv&listId=22038&lookupField=email

                      > postresult<-POST(url,body=list(file=upload_file(filename)))

                      > batchId<-content(postresult)$result[[1]]$batchId

                      > content(postresult)

                      $requestId

                      [1] "f0b5#15fb6ed37a8"

                      $result

                      $result[[1]]

                      $result[[1]]$batchId

                      [1] 22038

                      $result[[1]]$importId

                      [1] "22038"

                      $result[[1]]$status

                      [1] "Queued"

                      $success

                      [1] TRUE

                      > filename<-paste0(filepath,"discovery_integration_02.csv")

                      > postresult<-POST(url,body=list(file=upload_file(filename)))

                      > batchId<-content(postresult)$result[[1]]$batchId

                      > content(postresult)

                      $requestId

                      [1] "7784#15fb6edeb75"

                      $success

                      [1] FALSE

                      $errors

                      $errors[[1]]

                      $errors[[1]]$code

                      [1] "1019"

                      $errors[[1]]$message

                      [1] "Import is in progress"

                      > postresult<-POST(url,body=list(file=upload_file(filename)))

                      > batchId<-content(postresult)$result[[1]]$batchId

                      > content(postresult)

                      $requestId

                      [1] "82e#15fb6ef9ffd"

                      $result

                      $result[[1]]

                      $result[[1]]$batchId

                      [1] 22038

                      $result[[1]]$importId

                      [1] "22038"

                      $result[[1]]$status

                      [1] "Queued"

                      $success

                      [1] TRUE

                        • Re: Rest API - Bulk Import Status
                          Kenny Elkington

                          Do you have the real requests and responses?  I have trust issues with variable dumps.

                            • Re: Rest API - Bulk Import Status
                              Jason Ferrell

                              I'm having trouble getting the response in a non-variable format, but it's just a JSON object.  Here is the request, in part, with the 1000 lead CSV data omitted from the middle. 

                              -> POST /bulk/v1/leads.json?access_token=123&format=csv&listId=22038&lookupField=email HTTP/1.1

                              -> Host: 947-IOX-305.mktorest.com

                              -> User-Agent: libcurl/7.54.1 r-curl/2.8.1 httr/1.3.1

                              -> Accept-Encoding: gzip, deflate

                              -> Accept: application/json, text/xml, application/xml, */*

                              -> Content-Length: 396883

                              -> Content-Type: multipart/form-data; boundary=------------------------d16b91702e24668e

                              ->

                              >> --------------------------d16b91702e24668e

                              >> Content-Disposition: form-data; name="file"; filename="discovery_integration_01.csv"

                              >> Content-Type: text/csv

                              >>

                               

                               

                              >> "accountnumber","company",

                              ...

                              >>

                              >> --------------------------d16b91702e24668e--

                               

                               

                              <- HTTP/1.1 200 OK

                              <- Server: nginx

                              <- Date: Tue, 14 Nov 2017 15:23:13 GMT

                              <- Content-Type: application/json;charset=UTF-8

                              <- Content-Length: 113

                              <- Connection: keep-alive

                              <-

                        • Re: Rest API - Bulk Import Status
                          Jason Ferrell

                          Just wanted to see if anyone else has any thoughts about this issue.  Fundamentally, the batch id that is returned by the Bulk Lead Import API cannot be the static list id because it makes it difficult to track status of the job.