jpeg driver hanging in CreateCopy

[lpinner]

Using GDAL 1.9.0 on Ubuntu 10.04 (64bit) and Win 7 Enterprise (64bit OS, 32 bit python & gdal).

I have some code that generates 24bit RGB or 8bit greyscale reduced resolution and stretched (using a VRT) JPEGS from our satellite imagery. I’m having an issue where CreateCopy? (and gdal_translate) is hanging.

The data I’ve had trouble with is a single band (Pan) GeoEye?-1 image:  http://geoeyemediaportal.s3.amazonaws.com/assets/images/ChannelPartnerPortal/Australia/Geo_11bit/Example1.zip (500+mb, sorry). If that link doesn't work, try:  http://www.geoeye.com/CorpSite/resource/sample_imagery_response.aspx

The VRT (test.vrt, also attached) is:

<VRTDataset rasterXSize="800" rasterYSize="1870">
  <VRTRasterBand dataType="Byte" band="1">
    <NoDataValue>0.0</NoDataValue>
    <ComplexSource>
      <SourceFilename relativeToVRT="0">/vsizip/Example1.zip/po_311944_pan_0000000.tif</SourceFilename>
      <SourceBand>1</SourceBand>
      <SrcRect xOff="0" yOff="0" xSize="13532" ySize="31624"/>
      <DstRect xOff="0" yOff="0" xSize="800" ySize="1870"/>
      <ScaleOffset>-53.125</ScaleOffset>
      <ScaleRatio>1.51785714286</ScaleRatio>
    </ComplexSource>
  </VRTRasterBand>
</VRTDataset>

The following gdal_translate command succeeds, i.e the output jpg is created and the progress meter says it’s done, but just never returns.

$ gdal_translate -of jpeg test.vrt test.jpg
Input file size is 800, 1870
0...10...20...30...40...50...60...70...80...90...100 - done.

If a tif is output then the command completes and returns as expected:

$ gdal_translate  test.vrt test.tif
0...10...20...30...40...50...60...70...80...90...100 - done.
$

I've attached the VRT, a python script that demonstrates the same issue with CreateCopy?() and a a gdb trace run on gdal_translate (but I don't have a version of gdal with debug symbols so don't know if it's of any use).

[lpinner]

VRT

[lpinner]

Test python script

[lpinner]

GDB trace output

[lpinner]

Ahah! If I remove <NoDataValue?>0.0</NoDataValue> from the VRT it works.

[rouault]

From the backtrace, I see that the JPEG driver was in the middle of writing a nodata mask. I believe that if you had waited more (probably a time equivalent to the duration needed to go from 0% to 100%), the command would have finally completed. Would be cool if the progress bar was more significant, but I haven't studied how feasable it would be

[lpinner]

The CreateCopy?/gdal_translate took approximately 30-45 seconds, the nodata mask was still running about 3 hours later before I killed it (not in the GBD backtrace, but when I was first trying to diagnose the problem).

[rouault]

Weird. Could you attach a full example (including the raster referenced by the VRT, possibly size-reduced so it is as small as possible) so it can be reproduced ?

[lpinner]

Ok, I'll see if I can reproduce it with a smaller raster in the next day or so.

[rouault]

ok, I somehow reproduced it with something that should be similar given the VRT you indicated. It took a long time overall (perhaps 2 hours), but at the end, it eventually completed. There are several phenomon that can explain the huge slowness, mainly the use of a TIFF in a zip file (very inefficient seeking) and VRT that does big subsampling. The default cache size isn't sufficient to handle this. I managed to reduce the run time to a couple of minutes by adding "--config GDAL_CACHEMAX 1000" to my gdal_translate command line.

And the following should give more realistic progress bar (when it will reach 100%, this will be really finished) :

r23893 /trunk/gdal/frmts/jpeg/jpgdataset.cpp: JPEG: add more realistic progress bar support in CreateCopy?() when no data mask is also written (#4488)

[lpinner]

I can reproduce it only with large rasters, and only if the BLOCKYSIZE is 128 (as per the large GeoEye? image linked to above). I've attached a test script that generates some random test data that demonstrates the hang. If you change the blockysize from 128 to 1 the issue doesn't occur.

[lpinner]

Replying to rouault:

There are several phenomon that can explain the huge slowness, mainly the use of a TIFF in a zip file (very inefficient seeking)

Forgot to mention, the test2.py doesn't use a zipped raster and still is very slow (I haven't let it run to completion, only for a number of minutes. If I run the script with a BLOCKYSIZE of 1, it completes quickly.

[lpinner]

Fix syntax error after minor change without testing...

[rouault]

r24299 /trunk/gdal/gcore/ (gdal_priv.h gdalnodatamaskband.cpp): GDALNoDataMaskBand: implement IRasterIO() for an optimization in common use case (#4488)

r24300 /trunk/gdal/frmts/jpeg/jpgdataset.cpp: JPEG: add a INTERNAL_MASK creation option to be able to disable appending the ZLib mask if not needed