Conversion of MrSID datasets from NASA Zulu to JP2ECW hangs

[mloskot]

During tests of MrSID to JP2ECW translator, some issue has been discovered when trying to convert big MrSID images downloaded from NASA Zulu server.
The conversion process hangs for 2-3 hours with 100% CPU time but without any progress of the translation process.

Here is the full command line using GDAL utilities from FWTools 1.1.0, on Windows XP:

gdal_translate -of JP2ECW -co "LARGE_OK=YES" N-35-50_2000.sid out.jp2

Mark Sheridan noticed the same issue when trying to convert images from the NASA Zulu server. Mark and I tested on different images from the Zulu server, so it seems does not depend on a particular image but it applies to all images from Zulu server. Thus, It should be sufficient to download any image from the Zulu:

https://zulu.ssc.nasa.gov/mrsid/

I'm going to attach two screenshots taken at the beginning and at the end of the conversion process.

[mloskot]

Beginning of the conversion process

[mloskot]

End of the conversion process

[mloskot]

Here I paste last comments from Mark Sheridan:

--------
However with this version, on the NASA Zulu data (mentioned in previous
email), it seems to hang at about 10%.

I tested the tile "S-50-30_2000" (Perth in Western Australia), 190Mb
compressed, which is 5.2Gb uncompressed, which would normally compress in no
more than an hour, but the process was still going with no visible progress
and 100% CPU after 3 hours, so I killed it.
--------

[warmerdam]

I believe the problem is classic cache thrashing.  The default cache is 
too small to hold one full row of 128 pixel tall tiles for the input mrsid,
so the first row of tiles is re-read for each output scanline being generated.

I would suggest calling 

   GDALSetCacheMax( 100000000 );

See if that helps.

[mloskot]

Frank,

Thanks for the hint. Indeed, the problem was with too tiny cache size.
I think the bug can be closed, so I'm doing it now.

BTW, on my P4 3GHz with 512MB RAM, converting ~190 MB MrSID file to JP2ECW with target ratio 20:1 takes ~ 1hr 20min. The output is ~400MB
Do you think it's because I have not enough RAM or this timing is normal?