#6980 closed defect (invalid)
gdal_translate ignore CFL_TMPDIR and fills up /tmp
Reported by: | jsegu | Owned by: | warmerdam |
---|---|---|---|
Priority: | normal | Milestone: | |
Component: | GDAL_Raster | Version: | 2.2.1 |
Severity: | normal | Keywords: | gdal_translate, tmp, CFL_TMPDIR |
Cc: |
Description
When trying to patch and compose large RGB images from multiple single-band tiles (via vrt files), gdal_translate systematically fails with this enigmatic error:
$ gdal_translate -co PHOTOMETRIC=rgb in.vrt out.tif write component failed Input file size is 4000, 3000 0...10...20...30write component failed error: cannot decode code stream ERROR 1: in.jp2, band 1: IReadBlock failed at X offset 0, Y offset 0
I later figured out this was related to the /tmp filesystem filling up:
$ df -h /tmp Filesystem Size Used Avail Use% Mounted on tmpfs 4.0G 4.0G 23M 100% /tmp
However broswing /tmp won't show any files, because the handles are immediately deleted:
$ lsof -- | grep '/tmp' gdal_tran 22116 juliens 5u REG 0,35 241120800 20911278 /tmp/tmp.XXXXZon9cH (deleted) gdal_tran 22116 juliens 7u REG 0,35 241120800 20911279 /tmp/tmp.XXXXRmvPCC (deleted) gdal_tran 22116 juliens 9u REG 0,35 241120800 20911280 /tmp/tmp.XXXXT9wOY0 (deleted) gdal_tran 22116 juliens 11u REG 0,35 241120800 20911281 /tmp/tmp.XXXX5AzwVS (deleted) gdal_tran 22116 juliens 13u REG 0,35 241120800 20911282 /tmp/tmp.XXXXHWRJaU (deleted) gdal_tran 22116 juliens 15u REG 0,35 241120800 20911283 /tmp/tmp.XXXXzExgKq (deleted)
The wiki (https://trac.osgeo.org/gdal/wiki/ConfigOptions#CPL_TMPDIR) states that "By default, temporary files are written into current working directory.", but this seems not to be the case. I tried setting $CFL_TMPDIR but none of the following prevented the 4G /tmp to fill up.
$ TMPDIR=~/tmp gdal_translate [...] $ CPL_TMPDIR=~/tmp gdal_translate [...] $ export CPL_TMPDIR=~/tmp ; gdal_translate [...] $ gdal_translate [...] --config CPL_TMPDIR /tmp
Does gdal_translate ignore CPL_TMPDIR? Perhaps related: https://gis.stackexchange.com/questions/151153/is-gdal-translate-ignoring-the-cpl-tmpdir-switch
Change History (4)
comment:1 by , 7 years ago
comment:2 by , 7 years ago
I think it uses Jasper 1.900.1 (Debian). I will try to rebuild GDAL against a more recent version.
comment:3 by , 7 years ago
Resolution: | → invalid |
---|---|
Status: | new → closed |
Indeed I've grepped through the sources of Jasper, and it appears that if the uncompressed image size is above 16 MB, it creates a temporary file using tmpnam() (see jasper-1.900.1.uuid/src/libjasper/base/jas_stream.c). It doesn't appear that an environment variable can control the location of the temporary file. So nothing that can't be done on GDAL side. You'd better try recompiling GDAL against the openjpeg library (JP2OpenJPEG driver).
comment:4 by , 7 years ago
I could not compile against latest Jasper (2.0.12). Compiling against OpenJPEG (2.1.2) as you suggest solved the problem, and also made my code run much faster. Thank you very much!
Neither gdal_translate, the VRT or the GTiff driver generate temporary files, so I guess the issue comes from the underneath library of the JPEG2000 driver used. Which one is it ?