gdal_translate crashes/hangs converting DEM TIF into JPEG

[jwderos]

I have some large GeoTIFF DEM files that I need to convert into JPEG. They are larger than the maximum JPEG size so I have to resize them. Originally, they would complete and then seg fault. Tonight, they seem to just be hanging forever after completion.

I am issuing commands like this: gdal_translate -of JPEG -ot Byte -scale -outsize 86% 86% --config GDAL_CACHEMAX 64 SandyPre_NC_Riegl_2011_1M_Part3.TIF out.jpg

An example file can be downloaded from: ​https://hdds.usgs.gov/hdds2/pub/data/disaster/200908_Hurricane_Test/data/SATELLITE/NON_INGESTED/GDAL

You will need to register for an account in order to download.

[Even Rouault]

Could you paste here the output of "gdalinfo SandyPre_NC_Riegl_2011_1M_Part3.TIF" ? Which GDAL version are you using ? Which OS ? Is it a Custom build or a pre-existing build downloaded somewhere ?

[jwderos]

Input file size is 75637, 67320 0...10...20...30...40...50...60...70...80...90...100 - done.

and then nothing. Originally, it was issuing a seg fault after completion but that isn't happening now for some reason. It is hanging at 100% CPU.

GDAL 1.9.2 on some Linux version. uname -a says: Linux edclxs89 2.6.9-100.0.0.0.1.ELsmp #1 SMP Thu Feb 17 17:58:07 EST 2011 x86_64 x86_64 x86_64 GNU/Linux

I also tried it on OS X Mountain Lion with the same results. The generated JPEG file appears to be valid. I can open it without problems on the Mac.

It is a custom build of GDAL but it has run fine (more or less) for many other data sets. In this particular data set, there are two smaller images that convert to JPEG with no problem.

[Even Rouault]

You didn't answer my question. I wanted to see the full output given by gdalinfo, in particular if there's nodata or a mask band associated with the input dataset, in which case the mask will be written at the end of the JPEG dataset which can take some time. In which case you can try adding -co INTERNAL_MASK=NO to the gdal_translate command line (this exist in the trunk version of GDAL, 1.10dev, but wasn't documented yet. Done now).

[jwderos]

Sorry. I thought you wanted gdal_translate output. Here is the gdalinfo output:

Driver: GTiff/GeoTIFF Files: SandyPre_NC_Riegl_2011_1M_Part3.TIF

SandyPre_NC_Riegl_2011_1M_Part3.TIF.ovr SandyPre_NC_Riegl_2011_1M_Part3.TFw SandyPre_NC_Riegl_2011_1M_Part3.TIF.aux.xml

Size is 75637, 67320 Coordinate System is: PROJCS["NAD_1983_UTM_Zone_18N",

GEOGCS["NAD83",

DATUM["North_American_Datum_1983",

SPHEROID["GRS 1980",6378137,298.2572221010002,

AUTHORITY["EPSG","7019"]],

AUTHORITY["EPSG","6269"]],

PRIMEM["Greenwich",0], UNIT["degree",0.0174532925199433], AUTHORITY["EPSG","4269"]],

PROJECTIONTransverse_Mercator, PARAMETER["latitude_of_origin",0], PARAMETER["central_meridian",-75], PARAMETER["scale_factor",0.9996], PARAMETER["false_easting",500000], PARAMETER["false_northing",0], UNIT["metre",1,

AUTHORITY["EPSG","9001"]],

AUTHORITY["EPSG","26918"]]

Origin = (382631.470000028610229,3962605.439999999478459) Pixel Size = (1.000000000000000,-1.000000000000000) Metadata:

AREA_OR_POINT=Area

Image Structure Metadata:

COMPRESSION=LZW INTERLEAVE=BAND

Corner Coordinates: Upper Left ( 382631.470, 3962605.440) ( 76d17'56.22"W, 35d48' 1.99"N) Lower Left ( 382631.470, 3895285.440) ( 76d17'21.18"W, 35d11'37.33"N) Upper Right ( 458268.470, 3962605.440) ( 75d27'42.84"W, 35d48'24.06"N) Lower Right ( 458268.470, 3895285.440) ( 75d27'30.38"W, 35d11'58.91"N) Center ( 420449.970, 3928945.440) ( 75d52'37.68"W, 35d30' 3.19"N) Band 1 Block=128x128 Type=Float32, ColorInterp=Gray

Min=-0.927 Max=18.993 Minimum=-0.927, Maximum=18.993, Mean=1.634, StdDev=1.187 NoData Value=-3.4028234663852886e+38 Overviews: 37819x33660, 18910x16830, 9455x8415, 4728x4208, 2364x2104, 1182x1052, 591x526, 296x263, 148x132 Metadata:

STATISTICS_MAXIMUM=18.993469238281 STATISTICS_MEAN=1.6338017833421 STATISTICS_MINIMUM=-0.92689234018326 STATISTICS_STDDEV=1.1866528094619

[jwderos]

Try this version instead...

Driver: GTiff/GeoTIFF
Files: SandyPre_NC_Riegl_2011_1M_Part3.TIF
       SandyPre_NC_Riegl_2011_1M_Part3.TIF.ovr
       SandyPre_NC_Riegl_2011_1M_Part3.TFw
       SandyPre_NC_Riegl_2011_1M_Part3.TIF.aux.xml
Size is 75637, 67320
Coordinate System is:
PROJCS["NAD_1983_UTM_Zone_18N",
    GEOGCS["NAD83",
        DATUM["North_American_Datum_1983",
            SPHEROID["GRS 1980",6378137,298.2572221010002,
                AUTHORITY["EPSG","7019"]],
            AUTHORITY["EPSG","6269"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433],
        AUTHORITY["EPSG","4269"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",-75],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AUTHORITY["EPSG","26918"]]
Origin = (382631.470000028610229,3962605.439999999478459)
Pixel Size = (1.000000000000000,-1.000000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (  382631.470, 3962605.440) ( 76d17'56.22"W, 35d48' 1.99"N)
Lower Left  (  382631.470, 3895285.440) ( 76d17'21.18"W, 35d11'37.33"N)
Upper Right (  458268.470, 3962605.440) ( 75d27'42.84"W, 35d48'24.06"N)
Lower Right (  458268.470, 3895285.440) ( 75d27'30.38"W, 35d11'58.91"N)
Center      (  420449.970, 3928945.440) ( 75d52'37.68"W, 35d30' 3.19"N)
Band 1 Block=128x128 Type=Float32, ColorInterp=Gray
  Min=-0.927 Max=18.993 
  Minimum=-0.927, Maximum=18.993, Mean=1.634, StdDev=1.187
  NoData Value=-3.4028234663852886e+38
  Overviews: 37819x33660, 18910x16830, 9455x8415, 4728x4208, 2364x2104, 1182x1052, 591x526, 296x263, 148x132
  Metadata:
    STATISTICS_MAXIMUM=18.993469238281
    STATISTICS_MEAN=1.6338017833421
    STATISTICS_MINIMUM=-0.92689234018326
    STATISTICS_STDDEV=1.1866528094619

[Even Rouault]

ok, that confirms my previous guess. There's a nodata value associated with the band, so the mask is appended after the JPEG stream. This should complete ultimately after some time... If you use trunk and don't need the mask band, then try adding -co INTERNAL_MASK=NO as suggested

[jwderos]

There is still the issue with the segmentation fault. I dug through my shell history and found the exact command that triggers it:

gdal_translate --config GDAL_CACHEMAX 128 -of JPEG -co QUALITY=40 -ot Byte -scale -outsize 82% 82% SandyPre_NC_Riegl_2011_1M_Part3.TIF out2.jpg

I had -co WORLDFILE=YES in there too and it still seg faults even without it. I am trying it without -co QUALITY=40 right now.

I even get a seg fault on my Mac build too. Here is the backtrace:

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libgdal.1.dylib               	0x00000001013162ea JPGDataset::CreateCopy(char const*, GDALDataset*, int, char**, int (*)(double, char const*, void*), void*) + 1338 (jpgdataset.cpp:2247)
1   libgdal.1.dylib               	0x0000000101450310 GDALDriver::CreateCopy(char const*, GDALDataset*, int, char**, int (*)(double, char const*, void*), void*) + 208 (gdaldriver.cpp:614)
2   libgdal.1.dylib               	0x0000000101450409 GDALCreateCopy + 89 (gdaldriver.cpp:657)
3   gdal_translate                	0x000000010116e952 ProxyMain(int, char**) + 11698 (gdal_translate.cpp:1221)
4   libdyld.dylib                 	0x00007fff94c387e1 start + 1

[jwderos]

I also get a segfault without -co QUALITY=40. I am running without --config GDAL_CACHEMAX 128 and have not gotten a segfault yet. However, it is hung up and has been like this for 30 minutes now. Even with the no-data value, that is a long time. I will try that on the Mac too.

[Even Rouault]

I've reproduced the segfault and fixed it in trunk (r25486) and branches/1.9 (r25487)