Bulk Data extract

[jay2]

I have raised a support ticket to extract bulk records of approx 1M+ data from Marketo and Adobe support suggested to extract data via the bulk API as per the product limitation issue can not be extracted bulk data. (considering the schema file format contains 500+ fields names in the header)  

https://developers.marketo.com/rest-api/bulk-extract/bulk-lead-extract/ 

https://developers.marketo.com/rest-api/bulk-extract/ 

When I tried extracting bulk data via API the job status endpoints return a checksum in the fileChecksum attribute when status is “Completed”.

The checksum is a SHA-256 hash of the exported file. https://sha256sum.com/ 

I was wondering how to extract the data in such a case. I have not used any SHA-256 coding in the past. is there any workaround to download this checksum file?

[Jo_Pitts1]

@jay2 ,

which language are you developing in?

You shouldn't really need to worry about the checksum, but regardless knowing your dev stack will help immensely.

Regards

Jo

[SanfordWhiteman · Accepted solution]

---

I was wondering how to extract the data in such a case. I have not used any SHA-256 coding in the past. is there any workaround to download this checksum file?

---

The checksum is so you can be sure you downloaded the entire file.

This is usually not worth worrying about, but with very large files the chance of a network error (on either side) rises. And some HTTP clients can’t/won’t tell you if the connection was dropped prematurely, or they may not care if the length of the downloaded content matches the Content-Length header, etc.

If you run a checksum over the saved file, it must match the checksum on the server side exactly or else it’s a different file.

Note your app might be changing line breaks or perhaps adding/removing a BOM, in which case the files and thus checksums will be different, but at the business level the file is OK.