GDAL/ECW Fails for high resolution views of large images

[steve.lime@…]

Frank: Running into trouble when generating high resolution views of very large ECW images. Using MapServer 4.5, latest ECW toolkit, CVS-based GDAL (downloaded on 3/10.05) on SuSE 9.0. Here's the gdalinfo from the image in question (~33Gb):

stlime@noah:/r01/data/drs/data/gen/state/mn> gdalinfo img_fsa01bim4.ecw
Driver: ECW/ERMapper Compressed Wavelets
Size is 555647, 662765
Coordinate System is:
PROJCS["UTM Zone 15, Northern Hemisphere",
    GEOGCS["NAD83",
        DATUM["North_American_Datum_1983",
            SPHEROID["GRS 1980",6378137,298.257222101,
                AUTHORITY["EPSG","7019"]],
            TOWGS84[0,0,0,0,0,0,0],
            AUTHORITY["EPSG","6269"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9108"]],
        AXIS["Lat",NORTH],
        AXIS["Long",EAST],
        AUTHORITY["EPSG","4269"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",-93],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",0],
    UNIT["Meter",1]]
Origin = (187478.000000,5471805.000000)
Pixel Size = (1.00000000,-1.00000000)
Corner Coordinates:
Upper Left  (  187478.000, 5471805.000) ( 97d18'3.25"W, 49d19'7.87"N)
Lower Left  (  187478.000, 4809040.000) ( 96d51'26.05"W, 43d22'8.97"N)
Upper Right (  743125.000, 5471805.000) ( 89d39'7.83"W, 49d21'1.56"N)
Lower Right (  743125.000, 4809040.000) ( 89d59'52.66"W, 43d23'41.41"N)
Center      (  465301.500, 5140422.500) ( 93d27'5.45"W, 46d24'59.34"N)
Band 1 Block=555647x1 Type=Byte, ColorInterp=Red
  Overviews: arbitrary
Band 2 Block=555647x1 Type=Byte, ColorInterp=Green
  Overviews: arbitrary
Band 3 Block=555647x1 Type=Byte, ColorInterp=Blue
  Overviews: arbitrary

Everything's great until zooming in when I get an error:

msDrawMap(): Image handling error. Failed to draw layer named 'fsa'. <br>
An error has occurred: Error 48 "Could not perform Read/Write o: Unable to access file. GDALRasterIO() failed: drawGDAL() 

I've tested a number of extents and with this image the threshold for the error is approximately an extent with 6000 meters on a side (output image size 500x500).

Steve

[steve.lime@…]

Just to add a bit. It's definitely resolution related. If I take that same 6000 
meter extent and apply it to a 200x200 image I don't get the error...

Steve

[warmerdam]

I plan to try and reproduce and fix this.  However, my first efforts to
create a very large ECW file have failed (even with "lossless" compression
it was still only 1.7GB!). 

[steve.lime@…]

Can you think of a way I could get you a large image? Do you have a tape 
drive? Alternatively I could get you access to the computer here via ssh or 
even telnet for a short time so you could debug on a box with a large image...

Steve

[warmerdam]

Problem confirmed with the following test program.  It seems that any
request that forces reading past the 4GB limit is causing a failure. 
Specifically the call to lseek64() for Linux is resulting in a negative
value being returned from NCSFileSeekNative().  I suspect we are not 
not doing the 64bit build properly and either the correct lseek64 macro 
is not in place or something similar. 


#include <stdio.h>
#include "gdal.h"

int main()

{
    GDALDatasetH hSrcDS, hDstDS;
    int  nSrcWin[4] = { 200000, 200000, 3000, 3000 };
    int  nDstSize[2] = { 500, 500 };
    GByte abyData[500*500*3];
    CPLErr eErr;
    GDALDriverH hDrv;

    GDALAllRegister();
    hSrcDS = GDALOpen( "img_fsa01bim4.ecw", GA_ReadOnly );
    
    if( hSrcDS == NULL )
        exit(1);

    eErr = 
        GDALDatasetRasterIO( hSrcDS, GF_Read, 
                             nSrcWin[0], nSrcWin[1], nSrcWin[2], nSrcWin[3],
                             abyData, nDstSize[0], nDstSize[1], GDT_Byte, 
                             3, NULL, 0, 0, 0 );

    if( eErr == CE_Failure )
    {
        printf( "GDALDatasetRasterIO() read failed.\n" );
        exit( 1 );
    }

    GDALClose( hSrcDS );

    /* Create output file */

    hDrv = GDALGetDriverByName( "GTiff" );
    hDstDS = GDALCreate( hDrv, "out.tif", nDstSize[0], nDstSize[1], 3, 
                         GDT_Byte, NULL );

    eErr = 
        GDALDatasetRasterIO( hDstDS, GF_Write, 0, 0, nDstSize[0], nDstSize[1], 
                             abyData, nDstSize[0], nDstSize[1], GDT_Byte, 
                             3, NULL, 0, 0, 0 );
    GDALClose( hDstDS );
}

[warmerdam]

OK, it would *appear* that the lseek64 prototype from unistd.h is
not getting exposed properly.   I have tried a variety of things, but
I don't know how it is supposed to be exposed.  However, if I just manually
include it then things seem to work. 

So,I have added:

extern __off64_t lseek64 (int __fd, __off64_t __offset, int __whence);

near the top of the ecwsdk #ifdef'ed for linux.  Rebuilding now to verify
this fixes things. 

Ugg, big file access still sucks. 

[warmerdam]

OK, I have confirmed adding the lseek64 prototype fixes the problem. 

Steve, I think you can just copy the shared libraries from 
/home/frwarmer/local/lib to get going.  Please try that and let me know.

I will try and follow up with ERMapper about this. 

[Even Rouault]

Assuming this is fixed now