LANDSAT 7 FAST read problems

[Markus Neteler]

Frank,

LANDSAT 7 FAST files which are read by GDAL 1.3.2 are failing in SVN HEAD
due to parser errors of the header file. 

The FILENAMEs in the .hdr file are seeks at byte position -2 and
susequently everything fails. I attach a header for illustration
which worked in the past.

Example:
ftp://ftp.glcf.umiacs.umd.edu/glcf/Landsat/WRS2/p016/r035/p016r035_7x20000331.ETM-EarthSat/

Note: p016r035_7x20000331 header files are messed up and contain WRONG FILENAMEs which is a problem of GLCF, not of GDAL. Renaming the p016r035_7k20000331.b01.gz (other channels likewise) to L71016035_03520000331_B61.FST etc. helps. 

In this example also GDAL 1.3.2 fails to detect the second thermal channel.

Best,
Markus

[neteler@…]

sample LANDSAT7 header

[warmerdam]

Note, Rostic (rsheykhet at sanz.com) reports a likely related problem:

Frank,

 

It seems to me that FAST L7 code is broken and has been broken for a while (since revision 9952, August 2006).  We have tested the current build of GDAL with the FAST L7 data we have here and it does not seem to read it, whereas the previous version of GDAL has no problems with the same file.  We searched for the problem and what we found is that file frmts/raw/fastdataset.cpp has a problem introduced by refactoring.

 

Before refactor:

-----------------

#define FILENAME                "FILENAME ="

-----------------

 

After refactor:

-----------------

 

#define FILENAME                "FILENAME"

-----------------

 

 

However, this part is the same in both versions:

-----------------

 

    for ( i = 0; i < 6; i++ )

    {

        char *pszFilename = NULL ;

 

        if ( pszTemp )

            pszTemp = strstr( pszTemp, FILENAME );

        if ( pszTemp )

        {

            pszTemp += strlen(FILENAME);

            pszFilename = CPLScanString( pszTemp, FILENAME_SIZE, TRUE, FALSE );

        }

        else

            pszTemp = NULL;

        if ( poDS->FOpenChannel( pszFilename, poDS->nBands ) )

            poDS->nBands++;

        if ( pszFilename )

            CPLFree( pszFilename );

    }

-----------------

The parsing of the header file fails and therefore the open call fails.  There is another problem with the file that I am not quite sure what it is: when we changed FILENAME define to work, the file still could not read the projection information.  I see that a lot more commits have been done to the file after this change has been introduced, so I assume that it must work for some datasets.  If you think that this is an issue, I can provide a patch, and Martin Chapman will check it in.

 

Thanks a lot.
 

Rostic Sheykhet

----

We need to fix this in trunk and 1.4 branch, and Rostic should be notified
of any fix applied. 

I would also like a test case for this format added to the autotest, but with
an image file truncated to only a couple scanlines, and an appropriately
restricted test window. 

[warmerdam]

Adding Andrey to cc: list since I believe he was the original author of this
driver.  

[Mateusz Łoskot]

Indeed, it seems that main issue here is with the change of "FILENAME =" tag to "FILENAME". AFter removing 2 bytes " =", code reading filenames from header has not been updated to calculate bytes offset correctly.

The tag "FILENAME =" has constant length of 10 bytes and after " =" has been removed, they are required to be manually added to offset.
In other words, in fastdataset.cpp, the line 505

pszTemp += strlen(FILENAME);

should read:

pszTemp += strlen(FILENAME) + 2;

After this change, the sample LANDSAT7 header from the attachement gives following output:

mloskot:~/dev/gdal/bugs/1516/lsat7_header$ ~/bin/gdalinfo test.hdr 
Driver: FAST/EOSAT FAST Format
Size is 15781, 13961
Coordinate System is `'
Origin = (0.000000000000000,0.000000000000000)
Pixel Size = (1.000000000000000,1.000000000000000)
Metadata:
  ACQUISITION_DATE=20001009
  SATELLITE=LANDSAT7
  SENSOR=ETM+
  GAIN1=-4.699999809265137
  BIAS1=0.971764729069728
Corner Coordinates:
Upper Left  (   0.0000000,   0.0000000) 
Lower Left  (       0.000,   13961.000) 
Upper Right (   15781.000,       0.000) 
Lower Right (   15781.000,   13961.000) 
Center      (    7890.500,    6980.500) 
Band 1 Block=15781x1 Type=Byte, ColorInterp=Undefined
mloskot:~/dev/gdal/bugs/1516/lsat7_header$ 


Andrey, could you confirm if my analysis is correct?
I'd appreciate, because the FAST format is completely new to me, so I may be wrong in some/many places. Thanks!

[neteler@…]

Mateusz,

the parsing is still incomplete. If the header is read completely,
then the full geocoding comes out. I don't have access to GDAL 1.3.2
right now, but there it works (worked).

Best,
Markus

[neteler@…]

Mateusz,

I found a machine with
gdal-config --version
1.3.1

The output of the header attached to this bug report is:

gdalinfo test.hdr
Driver: FAST/EOSAT FAST Format
Size is 15781, 13961
Coordinate System is:
PROJCS["UTM Zone 36, Southern Hemisphere",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            TOWGS84[0,0,0,0,0,0,0],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9108"]],
        AXIS["Lat",NORTH],
        AXIS["Long",EAST],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",33],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",10000000],
    UNIT["Meter",1]]
Origin = (284092.500000,8186107.500000)
Pixel Size = (15.00000000,-15.00000000)
Metadata:
  ACQUISITION_DATE=20001009
  SATELLITE=LANDSAT7
  SENSOR=ETM+
  BIAS1=-4.699999809265137
  GAIN1=0.971764729069728
Corner Coordinates:
Upper Left  (  284092.500, 8186107.500) ( 30d58'41.99"E, 16d23'48.12"S)
Lower Left  (  284092.500, 7976692.500) ( 30d57'26.99"E, 18d17'18.44"S)
Upper Right (  520807.500, 8186107.500) ( 33d11'41.55"E, 16d24'22.80"S)
Lower Right (  520807.500, 7976692.500) ( 33d11'48.79"E, 18d17'57.38"S)
Center      (  402450.000, 8081400.000) ( 32d 4'54.68"E, 17d21'3.01"S)
Band 1 Block=15781x1 Type=Byte, ColorInterp=Undefined

Like this it should be in SVN HEAD, too.

Best,
Markus

[neteler@…]

As a reference: In the bugtracker are a few related (closed)
reports, they contain other headers for testing:

http://bugzilla.remotesensing.org/show_bug.cgi?id=182

http://bugzilla.remotesensing.org/show_bug.cgi?id=988  <- my friend :)

http://bugzilla.remotesensing.org/show_bug.cgi?id=489

thanks,
Markus

[Mateusz Łoskot]

The patch seems to fix the problem

[Mateusz Łoskot]

I'm assigning this bug to Andrey.
We talked on the IRC and Andrey suggested to leave any FAST issues to him.

[dron]

That should be now fixed both in HEAD and 1.4 branches.

Best regards,
Andrey

[Mateusz Łoskot]

I added test of FAST driver:
http://trac.osgeo.org/gdal/changeset/10983

The bug has been fixed and new tests pass, so I'm going to close this bug.