On 2015-09-23 12:29, Colin Johnston wrote:
maybe compress the data info on the probes before it is sent and save to backend storage with the data already compress’d ?
just a thought
Colin
It may be interesting to know, although not at all surprising: compressing individual results achieves almost nothing (and has the drawback of having to unpack/repack while the result moves in the pipeline). Compressing small chunks, say dozens of related results (e.g. from the same measurement) achieves 40-50% saving (a lot depends on what kind of result it is), while compressing large blocks (multi-megabyte chunks) can achieve 90-95% in most cases. It makes sense to compress data once it's at rest, especially if it's not often accessed. Therefore our processing pipeline is constructed such that it applies compression once a large enough batch has been consumed. As Romeo explained, this part was stretched a lot on Friday. Cheers, Robert