10 Replies Latest reply on Feb 14, 2018 1:53 AM by Arun Pachauri

    Compression on Bulk Export APIs?

    Kurt Koller

      For Bulk Activity Export - exporting a file it is both stored as plain text and transmitted without gzip compression.

       

      This data is repetitive and compresses very well. In my analysis of files with all activity types, ~90%+ with gzip -6 settings. It's a little better if you're asking for the same activity type.

       

      If we ask for 205M worth of activities, and if the data were stored compressed, it would be 20.5M versus 205M. Even if just gzip were turned on in nginx, at least that would save 90% of transmission time. Ideally this data would be stored compressed to not count toward the quota.

       

      At a minimum can compression be turned on for the file download so that part is quicker? The server currently doesn't seem to send gzipped data even when asked.