id summary reporter owner description type status priority milestone component version resolution keywords cc cpu platform 2750 LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed sprice grass-dev@… "I've added the ability for reading/writing raster rows in compression formats LZ4, LZ4HC, & ZSTD (in addition to the existing RLE & ZLIB.) These new algorithms are extremely fast (an order of magnitude faster than ZLIB) and are a better fit for modern, fast, hard drives & SSDs. I've attached a .tgz file with the necessary added & changed files. The new algorithms can be enabled/disabled with environment vars the same way as ZLIB, as described in the r.compress documentation: https://grass.osgeo.org/grass70/manuals/r.compress.html Algorithm summary: LZ4 produces a slightly worse compression ratio than ZLIB, but it compresses about an order of magnitude faster than ZLIB. It decompresses even faster. LZ4-HC is supposed to produce a compression ratio similar to ZLIB, and at about the same speed as ZLIB. It decompresses as fast as the regular LZ4 (>2 GB/s). Unfortunately, the improved compression ratio doesn't show in my tests, probably due to the fact we're compressing each row individually. This may change against floating point data if someone wants to test it. ZSTD is a new algorithm by the author of LZ4 that is intended to replace ZLIB. It compresses and decompresses extremely quickly, while maintaining a similar compression ratio as ZLIB. Unfortunately, it's still in beta. They are all under the BSD license. Links below show more info & performance numbers. https://github.com/Cyan4973/zstd https://github.com/Cyan4973/lz4 http://www.lz4.org I recommend incorporating the attached changes into GRASS, and leaving it as optional for users. At some point in the future, after testing, GRASS should move to using ZSTD as default. Power users who want the best performance '''now''' (and have the disk space) can use LZ4 immediately. It would be trivial to further alter get_row.c & put_row.c to use LZ4 for floating point compression. (And I would recommend someone do that.) Note: I've decided to use LZ4_decompress_fast() instead of LZ4_decompress_safe(). In my test, it was noticeably faster. According to the documentation, it leaves LZ4 open to a malicious attack. If this is a serious concern in the GRASS GIS internal data structures, change the commenting in get_row.c to use the safer code. On my computer, I've better than halved (!) the runtime of a r.mapcalc identity operation when using LZ4. Below are my tests while working with a RapidEye scene. {{{ > time r.mapcalc expression=""out_test_zlib=out_test_lz4hc"" --overwrite 100% real 3m33.503s user 3m25.451s sys 0m6.750s > time r.mapcalc expression=""out_test_zlib=out_test_lz4hc"" --overwrite 100% real 3m34.398s user 3m26.684s sys 0m6.138s > export GRASS_INT_LZ4=1 > time r.mapcalc expression=""out_test_lz4=out_test_lz4hc"" --overwrite 100% real 1m31.222s user 1m25.379s sys 0m5.035s > time r.mapcalc expression=""out_test_lz4=out_test_lz4hc"" --overwrite 100% real 1m29.792s user 1m24.029s sys 0m4.858s > unset GRASS_INT_LZ4 > export GRASS_INT_LZ4HC=1 > time r.mapcalc expression=""out_test_lz4hc2=out_test_lz4hc"" --overwrite 100% real 3m5.332s user 2m58.610s sys 0m5.603s > time r.mapcalc expression=""out_test_lz4hc2=out_test_lz4hc"" --overwrite 100% real 3m3.710s user 2m56.606s sys 0m5.858s > unset GRASS_INT_LZ4HC > export GRASS_INT_ZSTD=1 > time r.mapcalc expression=""out_test_zstd=out_test_lz4hc"" --overwrite 100% real 1m38.322s user 1m32.654s sys 0m4.897s > time r.mapcalc expression=""out_test_zstd=out_test_lz4hc"" --overwrite 100% real 1m42.370s user 1m35.487s sys 0m5.282s > unset GRASS_INT_ZSTD > ls -l vrt_test/PERMANENT/cell/out_test_* -rw-r--r-- 1 sprice staff 4080217012 Sep 26 14:01 vrt_test/PERMANENT/cell/out_test_lz4 -rw-r--r-- 1 sprice staff 4069728048 Sep 26 13:34 vrt_test/PERMANENT/cell/out_test_lz4hc -rw-r--r-- 1 sprice staff 4069728048 Sep 26 14:08 vrt_test/PERMANENT/cell/out_test_lz4hc2 -rw-r--r-- 1 sprice staff 3737100577 Sep 26 13:57 vrt_test/PERMANENT/cell/out_test_zlib -rw-r--r-- 1 sprice staff 3811356101 Sep 26 14:12 vrt_test/PERMANENT/cell/out_test_zstd > time r.univar out_test_zlib 100% total null and non-null cells: 3526771952 total null cells: 1502448926 Of the non-null cells: ---------------------- n: 2024323026 minimum: 807 maximum: 32767 range: 31960 mean: 9385.79 mean of absolute values: 9385.79 standard deviation: 6620.52 variance: 4.38312e+07 variation coefficient: 70.5377 % sum: 18999862195879 real 1m29.980s user 1m27.111s sys 0m2.589s > time r.univar out_test_lz4 100% total null and non-null cells: 3526771952 total null cells: 1502448926 Of the non-null cells: ---------------------- n: 2024323026 minimum: 807 maximum: 32767 range: 31960 mean: 9385.79 mean of absolute values: 9385.79 standard deviation: 6620.52 variance: 4.38312e+07 variation coefficient: 70.5377 % sum: 18999862195879 real 1m9.883s user 1m7.559s sys 0m2.210s > time r.univar out_test_lz4hc 100% total null and non-null cells: 3526771952 total null cells: 1502448926 Of the non-null cells: ---------------------- n: 2024323026 minimum: 807 maximum: 32767 range: 31960 mean: 9385.79 mean of absolute values: 9385.79 standard deviation: 6620.52 variance: 4.38312e+07 variation coefficient: 70.5377 % sum: 18999862195879 real 1m10.199s user 1m7.902s sys 0m2.173s > time r.univar out_test_zstd 100% total null and non-null cells: 3526771952 total null cells: 1502448926 Of the non-null cells: ---------------------- n: 2024323026 minimum: 807 maximum: 32767 range: 31960 mean: 9385.79 mean of absolute values: 9385.79 standard deviation: 6620.52 variance: 4.38312e+07 variation coefficient: 70.5377 % sum: 18999862195879 real 1m25.206s user 1m21.351s sys 0m2.518s }}}" enhancement closed normal 7.6.0 Raster svn-releasebranch74 fixed ZLIB LZ4 ZSTD All All