jpeg driver and temporary files

[Even Rouault]

I'm creating this ticket to trace the discussion "gdal fails on vrt (virtual raster) file" on gdal dev list :

Hi,

Gdal fails on my virtual raster file that links to JPEG graphics. Map 
size is 17121x12100 pixel and 13.8 MB of filesize (JPEG images).

Here the command (but also gdal_translate fails with the same error):

D:\DOP>gdalinfo referenz.vrt
ERROR 1: libjpeg: Failed to create temporary file
gdalinfo failed - unable to open 'referenz.vrt'.
Open GDAL Datasets:
   1 N DriverIsNULL 512x512x0

Any help is welcome.
Wolfgang

Frank's answer :

Wolfgang,

This is an issue in the JPEG driver.  It seems that libjpeg tries to
create a temporary file when an operation (like processing a progressive
jpeg) would require quite a bit of memory.  If it fails to create the file
you get the message you see.

I recently changed the amount of memory at which this happens to be 500MB
instead of 1MB in the builtin libjpeg for GDAL (SVN trunk)

   http://trac.osgeo.org/gdal/changeset/11706

If you are building GDAL from source, then you might want to apply the
same change.

It might also be sufficient to copy the jpeg file to a directory where
you have read-write access.

My analysis of the situation :

Answering to my self and after a few more investigations, we can read in 
jpeglib.h :

  /* Limit on memory allocation for this JPEG object.  (Note that this is
   * merely advisory, not a guaranteed maximum; it only affects the space
   * used for virtual-array buffers.)  May be changed by outer application
   * after creating the JPEG object.
   */
  long max_memory_to_use;


So, an alternative to define JPEGMEM (which seems to do what I thought it 
should do in my previous message) or  could be to add the following lines :
    if (getenv("JPEGMEM") == NULL)
       poDS->sDInfo.mem->max_memory_to_use = 500 * 1024 * 1024;
just after creating the decompression object done by :
    jpeg_create_decompress( &(poDS->sDInfo) ) ;

This way, we don't need to patch libjpeg and if the user wants to define it's 
own value, he's still able to do so.

FYI, on my Ubuntu system, I found out that libjpeg is build by default with 
1GB for DEFAULT_MAX_MEM.
I've tried to define JPEGMEM to a small value but couldn't reproduce 
Wolfgang's crash. I think it's because, in jmemansi.c, we can see that the 
temporary file is created by the C library "tmpfile()" call. On Linux, I 
think this file is created in /tmp : doing a "lsof | grep gdalinfo" with 
small values of JPEGMEM on a 10000x10000 progressive jpeg seems to confirm 
this intuition. I don't know where the MS C library tries to create the 
temporary file. The answer to this question could show a need for a specific 
patch in libjpeg for FWTool to create the temporary file in an appropriate 
directory.

To answer one of Wolfgang's question, it may be surprising that gdalinfo is 
affected by this issue. But in JPGDataset::Open, we call 
jpeg_start_decompress that is really slow when I set a small value for 
JPEGMEM. Then we read image size, color space and other information. The 
question is : do we really need to do jpeg_start_decompress to get this 
values ? Probably, but I we don't, it could speed up gdalinfo.

[Even Rouault]

Patch commited in trunk in r12079.

Let's hope that it doesn't cause problems in very memory constrained environments though...

[warmerdam]

I have ported the simple change of default to 500MB into 1.4 branch. This bug was also tripping up Flavio generating large printable jpegs from MapServer.

[warmerdam]

Last change was r12378.

[timaranz]

patch for jmemansi.c to use windows temporary file api on windows

[timaranz]

The underlying bug in the jpeg library still remains. The windows version of tmpfile() creates a temporary file in the root folder, to which users often do not have write privilege.

Attached jmemansi.patch that rectifies the problems on win32.

[Even Rouault]

I see this comment in the patch "COPIED FROM cairo-misc.c in gtklibs", but from ​https://github.com/mozilla-servo/cairo/blob/master/src/cairo-misc.c , cairo-misc.c is LPGL licenced, which is more restrictive than the GDAL X/MIT licence

[timaranz]

Sorry - hadn't noticed that licensing issue. It's pretty standard stuff, you could ask them nicely for permission to use it.

[Jukka Rahkonen]

For getting this ticket solved in some way it would be very nice if timaranz could ask nicely the permission and attach the hopefully positive, signed answer to this ticket.

[timmitchell]

jratike80 - thanks for the reminder. Forgotten password so created new user id. Here is the permission from the cairo contributor.

Yes you can use the _cairo_win32_tmpfile function under a X/MIT license.

Note you may want to change the three character prefix in GetTempFileNameW to something different. I used "ps_" since cairo uses tmpfile when creating PostScript output.

Adrian Johnson

On 05/01/15 05:53, Tim Mitchell wrote:

Dear Adrian,

As a novice open source contributor I submitted a copy of the _cairo_win32_tmpfile function from cairo-misc.c as a patch to a temporary file issue in the GDAL open source project (http://trac.osgeo.org/gdal/ticket/1795). The maintainers noticed the LGPL licence and as GDAL uses an X/MIT licence we need your permission to use the code.

Can we have permission to use the _cairo_win32_tmpfile function in GDAL please.

[Even Rouault]

In 40841:

Internal libjpeg: provide implementation of tmpfile() that works better on Windows (patch by timaranz, fixes #1795)