Write and store Bulk Export files gzipped

[Anonymous]

Currently Marketo Leads and Activities Bulk Export API writes and stores Bulk Export files in plain text which in some cases result in producing huge files.

In our case a month worth of activity data produces a ~50Gb files or daily leads export files up to 2Gb in size.

It's wasteful and unnecessarily requires additional storage on Marketo side and forces end users like us pay for increasing daily volume API limits.

Instead export files can be written to storage in a gzipped format in a streaming fashion.

All major technology stacks provide a way to implement it fairly easily.

[Kenny_Elkington]

>forces end users like us pay for increasing daily volume API limits.

This is not true, Peter.  Bulk Export is metered by size-on-disk, and not by data transfer size, so this does not affect your quota utilization..

[Anonymous]

It is in fact true, Kenny.

If you write a gzipped file on disk instead of plain text you end up having a much smaller file. Therefore metered size-on-disk would be less and as a result quota utilization would be less.