Opened 11 years ago

Closed 11 years ago

#4488 closed defect (fixed)

jpeg driver hanging in CreateCopy

Reported by: lpinner Owned by: warmerdam
Priority: normal Milestone: 1.10.0
Component: GDAL_Raster Version: unspecified
Severity: normal Keywords:
Cc:

Description

Using GDAL 1.9.0 on Ubuntu 10.04 (64bit) and Win 7 Enterprise (64bit OS, 32 bit python & gdal).

I have some code that generates 24bit RGB or 8bit greyscale reduced resolution and stretched (using a VRT) JPEGS from our satellite imagery. I’m having an issue where CreateCopy (and gdal_translate) is hanging.

The data I’ve had trouble with is a single band (Pan) GeoEye-1 image: http://geoeyemediaportal.s3.amazonaws.com/assets/images/ChannelPartnerPortal/Australia/Geo_11bit/Example1.zip (500+mb, sorry). If that link doesn't work, try: http://www.geoeye.com/CorpSite/resource/sample_imagery_response.aspx

The VRT (test.vrt, also attached) is:

<VRTDataset rasterXSize="800" rasterYSize="1870">
  <VRTRasterBand dataType="Byte" band="1">
    <NoDataValue>0.0</NoDataValue>
    <ComplexSource>
      <SourceFilename relativeToVRT="0">/vsizip/Example1.zip/po_311944_pan_0000000.tif</SourceFilename>
      <SourceBand>1</SourceBand>
      <SrcRect xOff="0" yOff="0" xSize="13532" ySize="31624"/>
      <DstRect xOff="0" yOff="0" xSize="800" ySize="1870"/>
      <ScaleOffset>-53.125</ScaleOffset>
      <ScaleRatio>1.51785714286</ScaleRatio>
    </ComplexSource>
  </VRTRasterBand>
</VRTDataset>

The following gdal_translate command succeeds, i.e the output jpg is created and the progress meter says it’s done, but just never returns.

$ gdal_translate -of jpeg test.vrt test.jpg
Input file size is 800, 1870
0...10...20...30...40...50...60...70...80...90...100 - done.

If a tif is output then the command completes and returns as expected:

$ gdal_translate  test.vrt test.tif
0...10...20...30...40...50...60...70...80...90...100 - done.
$

I've attached the VRT, a python script that demonstrates the same issue with CreateCopy() and a a gdb trace run on gdal_translate (but I don't have a version of gdal with debug symbols so don't know if it's of any use).

Attachments (5)

test.vrt (561 bytes ) - added by lpinner 11 years ago.
VRT
test.py (321 bytes ) - added by lpinner 11 years ago.
Test python script
gdb_bt.txt (3.9 KB ) - added by lpinner 11 years ago.
GDB trace output
test2.py (2.0 KB ) - added by lpinner 11 years ago.
Fix syntax error after minor change without testing…
test2.2.py (2.0 KB ) - added by lpinner 11 years ago.

Download all attachments as: .zip

Change History (14)

by lpinner, 11 years ago

Attachment: test.vrt added

VRT

by lpinner, 11 years ago

Attachment: test.py added

Test python script

by lpinner, 11 years ago

Attachment: gdb_bt.txt added

GDB trace output

comment:1 by lpinner, 11 years ago

Ahah! If I remove <NoDataValue>0.0</NoDataValue> from the VRT it works.

comment:2 by Even Rouault, 11 years ago

From the backtrace, I see that the JPEG driver was in the middle of writing a nodata mask. I believe that if you had waited more (probably a time equivalent to the duration needed to go from 0% to 100%), the command would have finally completed. Would be cool if the progress bar was more significant, but I haven't studied how feasable it would be

comment:3 by lpinner, 11 years ago

The CreateCopy/gdal_translate took approximately 30-45 seconds, the nodata mask was still running about 3 hours later before I killed it (not in the GBD backtrace, but when I was first trying to diagnose the problem).

comment:4 by Even Rouault, 11 years ago

Weird. Could you attach a full example (including the raster referenced by the VRT, possibly size-reduced so it is as small as possible) so it can be reproduced ?

in reply to:  4 comment:5 by lpinner, 11 years ago

Ok, I'll see if I can reproduce it with a smaller raster in the next day or so.

comment:6 by Even Rouault, 11 years ago

ok, I somehow reproduced it with something that should be similar given the VRT you indicated. It took a long time overall (perhaps 2 hours), but at the end, it eventually completed. There are several phenomon that can explain the huge slowness, mainly the use of a TIFF in a zip file (very inefficient seeking) and VRT that does big subsampling. The default cache size isn't sufficient to handle this. I managed to reduce the run time to a couple of minutes by adding "--config GDAL_CACHEMAX 1000" to my gdal_translate command line.

And the following should give more realistic progress bar (when it will reach 100%, this will be really finished) :

r23893 /trunk/gdal/frmts/jpeg/jpgdataset.cpp: JPEG: add more realistic progress bar support in CreateCopy() when no data mask is also written (#4488)

in reply to:  6 comment:7 by lpinner, 11 years ago

I can reproduce it only with large rasters, and only if the BLOCKYSIZE is 128 (as per the large GeoEye image linked to above). I've attached a test script that generates some random test data that demonstrates the hang. If you change the blockysize from 128 to 1 the issue doesn't occur.

in reply to:  6 comment:8 by lpinner, 11 years ago

Replying to rouault:

There are several phenomon that can explain the huge slowness, mainly the use of a TIFF in a zip file (very inefficient seeking)

Forgot to mention, the test2.py doesn't use a zipped raster and still is very slow (I haven't let it run to completion, only for a number of minutes. If I run the script with a BLOCKYSIZE of 1, it completes quickly.

by lpinner, 11 years ago

Attachment: test2.py added

Fix syntax error after minor change without testing...

by lpinner, 11 years ago

Attachment: test2.2.py added

comment:9 by Even Rouault, 11 years ago

Milestone: 2.0.0
Resolution: fixed
Status: newclosed

r24299 /trunk/gdal/gcore/ (gdal_priv.h gdalnodatamaskband.cpp): GDALNoDataMaskBand: implement IRasterIO() for an optimization in common use case (#4488)

r24300 /trunk/gdal/frmts/jpeg/jpgdataset.cpp: JPEG: add a INTERNAL_MASK creation option to be able to disable appending the ZLib mask if not needed

Note: See TracTickets for help on using tickets.