jpeg driver and temporary files

[Even Rouault]

I'm creating this ticket to trace the discussion "gdal fails on vrt (virtual raster) file" on gdal dev list :

Hi,

Gdal fails on my virtual raster file that links to JPEG graphics. Map 
size is 17121x12100 pixel and 13.8 MB of filesize (JPEG images).

Here the command (but also gdal_translate fails with the same error):

D:\DOP>gdalinfo referenz.vrt
ERROR 1: libjpeg: Failed to create temporary file
gdalinfo failed - unable to open 'referenz.vrt'.
Open GDAL Datasets:
   1 N DriverIsNULL 512x512x0

Any help is welcome.
Wolfgang

Frank's answer :

Wolfgang,

This is an issue in the JPEG driver.  It seems that libjpeg tries to
create a temporary file when an operation (like processing a progressive
jpeg) would require quite a bit of memory.  If it fails to create the file
you get the message you see.

I recently changed the amount of memory at which this happens to be 500MB
instead of 1MB in the builtin libjpeg for GDAL (SVN trunk)

   http://trac.osgeo.org/gdal/changeset/11706

If you are building GDAL from source, then you might want to apply the
same change.

It might also be sufficient to copy the jpeg file to a directory where
you have read-write access.

My analysis of the situation :

Answering to my self and after a few more investigations, we can read in 
jpeglib.h :

  /* Limit on memory allocation for this JPEG object.  (Note that this is
   * merely advisory, not a guaranteed maximum; it only affects the space
   * used for virtual-array buffers.)  May be changed by outer application
   * after creating the JPEG object.
   */
  long max_memory_to_use;


So, an alternative to define JPEGMEM (which seems to do what I thought it 
should do in my previous message) or  could be to add the following lines :
    if (getenv("JPEGMEM") == NULL)
       poDS->sDInfo.mem->max_memory_to_use = 500 * 1024 * 1024;
just after creating the decompression object done by :
    jpeg_create_decompress( &(poDS->sDInfo) ) ;

This way, we don't need to patch libjpeg and if the user wants to define it's 
own value, he's still able to do so.

FYI, on my Ubuntu system, I found out that libjpeg is build by default with 
1GB for DEFAULT_MAX_MEM.
I've tried to define JPEGMEM to a small value but couldn't reproduce 
Wolfgang's crash. I think it's because, in jmemansi.c, we can see that the 
temporary file is created by the C library "tmpfile()" call. On Linux, I 
think this file is created in /tmp : doing a "lsof | grep gdalinfo" with 
small values of JPEGMEM on a 10000x10000 progressive jpeg seems to confirm 
this intuition. I don't know where the MS C library tries to create the 
temporary file. The answer to this question could show a need for a specific 
patch in libjpeg for FWTool to create the temporary file in an appropriate 
directory.

To answer one of Wolfgang's question, it may be surprising that gdalinfo is 
affected by this issue. But in JPGDataset::Open, we call 
jpeg_start_decompress that is really slow when I set a small value for 
JPEGMEM. Then we read image size, color space and other information. The 
question is : do we really need to do jpeg_start_decompress to get this 
values ? Probably, but I we don't, it could speed up gdalinfo.