Opened 2 years ago

Closed 2 years ago

Last modified 2 years ago

#6980 closed defect (invalid)

gdal_translate ignore CFL_TMPDIR and fills up /tmp

Reported by: jsegu Owned by: warmerdam
Priority: normal Milestone:
Component: GDAL_Raster Version: 2.2.1
Severity: normal Keywords: gdal_translate, tmp, CFL_TMPDIR
Cc:

Description

When trying to patch and compose large RGB images from multiple single-band tiles (via vrt files), gdal_translate systematically fails with this enigmatic error:

$ gdal_translate -co PHOTOMETRIC=rgb in.vrt out.tif
write component failed
Input file size is 4000, 3000
0...10...20...30write component failed
error: cannot decode code stream
ERROR 1: in.jp2, band 1: IReadBlock failed at X offset 0, Y offset 0

I later figured out this was related to the /tmp filesystem filling up:

$ df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           4.0G  4.0G   23M 100% /tmp

However broswing /tmp won't show any files, because the handles are immediately deleted:

$ lsof -- | grep '/tmp'
gdal_tran 22116          juliens    5u      REG               0,35 241120800   20911278 /tmp/tmp.XXXXZon9cH (deleted)
gdal_tran 22116          juliens    7u      REG               0,35 241120800   20911279 /tmp/tmp.XXXXRmvPCC (deleted)
gdal_tran 22116          juliens    9u      REG               0,35 241120800   20911280 /tmp/tmp.XXXXT9wOY0 (deleted)
gdal_tran 22116          juliens   11u      REG               0,35 241120800   20911281 /tmp/tmp.XXXX5AzwVS (deleted)
gdal_tran 22116          juliens   13u      REG               0,35 241120800   20911282 /tmp/tmp.XXXXHWRJaU (deleted)
gdal_tran 22116          juliens   15u      REG               0,35 241120800   20911283 /tmp/tmp.XXXXzExgKq (deleted)

The wiki (https://trac.osgeo.org/gdal/wiki/ConfigOptions#CPL_TMPDIR) states that "By default, temporary files are written into current working directory.", but this seems not to be the case. I tried setting $CFL_TMPDIR but none of the following prevented the 4G /tmp to fill up.

$ TMPDIR=~/tmp gdal_translate [...]
$ CPL_TMPDIR=~/tmp gdal_translate [...]
$ export CPL_TMPDIR=~/tmp ; gdal_translate [...]
$ gdal_translate [...] --config CPL_TMPDIR /tmp

Does gdal_translate ignore CPL_TMPDIR? Perhaps related: https://gis.stackexchange.com/questions/151153/is-gdal-translate-ignoring-the-cpl-tmpdir-switch

Change History (4)

comment:1 Changed 2 years ago by Even Rouault

Neither gdal_translate, the VRT or the GTiff driver generate temporary files, so I guess the issue comes from the underneath library of the JPEG2000 driver used. Which one is it ?

comment:2 Changed 2 years ago by jsegu

I think it uses Jasper 1.900.1 (Debian). I will try to rebuild GDAL against a more recent version.

comment:3 Changed 2 years ago by Even Rouault

Resolution: invalid
Status: newclosed

Indeed I've grepped through the sources of Jasper, and it appears that if the uncompressed image size is above 16 MB, it creates a temporary file using tmpnam() (see jasper-1.900.1.uuid/src/libjasper/base/jas_stream.c). It doesn't appear that an environment variable can control the location of the temporary file. So nothing that can't be done on GDAL side. You'd better try recompiling GDAL against the openjpeg library (JP2OpenJPEG driver).

comment:4 Changed 2 years ago by jsegu

I could not compile against latest Jasper (2.0.12). Compiling against OpenJPEG (2.1.2) as you suggest solved the problem, and also made my code run much faster. Thank you very much!

Note: See TracTickets for help on using tickets.