Attempt to read past EOF on AIG Windows VAT table

[grovduck]

Using svn-trunk on Windows XP with ArcInfo 9.3 grids. When I try to open grids, I receive this message:

ERROR 3: Attempt to read past EOF in D:/matt/vat_test/grid1/../info/../grid1/vat.adf

when it calls ReadRAT() in aigdataset.cpp. I've tested using two different grids, one with 1280 unique values with VALUE and COUNT fields and then a clip of the first with 398 unique values with VALUE and COUNT. Test 1 gives the error message, test 2 is fine.

As in ticket #2447, the attribute data actually reads in correctly, so this is more of an annoyance than a major issue.

Test grids are at: ​ftp://ftp.fsl.orst.edu/pub/gregorym/vat_test.zip (53.4 MB)

[Even Rouault]

Assigning to Daniel. The error is issued by the AVC driver

[Markus Neteler]

See also trac #2447

[Even Rouault]

Daniel,

I've pushed in trunk (r17430) and branches/1.6 (r17431) a workaround to enable users to open those datasets that trigger the error with binding languages. Open() in swig/include/gdal.i checks that there is no failure returned by CPLGetLastErrorType() even if the dataset is not NULL. I've tried to look a bit at why the error was issued in the AVC driver but it didn't seem obvious how to fix that without knowledge of the format.

[Even Rouault]

r17432 (trunk only) : differ opening of the RAT only when GetDefaultRat() is called. Will improve performances and make less likely the error message of #3031

[grovduck]

Changed location of test grids to: ​ftp://ftp.fsl.orst.edu/pub/gregorym/gdal/vat_test.zip

[maphew]

I get this same error on 2 AIG datasets from a set of 60. They were generated by Arcinfo Workstation (GRID) but at different times (2006 and 2007) and perhaps different versions, in the 9.x range though.

Dataset at ​ftp://ftp.geomaticsyukon.ca/users/matt/AIG_read_past_EOF.7z 25mb.

>gdalinfo --version
GDAL 1.7.0dev, released 2008/11/26

>for /d %a in (dem_105e dem_105d dem_105o) do gdal_translate %a gdalout\%a.tif

>gdal_translate dem_105e gdalout\dem_105e.tif
Input file size is 3701, 3857
0...10...20...30...40...50...60...70...80...90...100 - done.

>gdal_translate dem_105d gdalout\dem_105d.tif
Input file size is 3817, 3858
0ERROR 3: Attempt to read past EOF in dem_105d/../info/arc.dir.
ERROR 4: Failed to open table dem_105d.VAT
...10...20...30...40...50...60...70...80...90...100 - done.

>gdal_translate dem_105o gdalout\dem_105o.tif
Input file size is 3410, 3804
0ERROR 3: Attempt to read past EOF in dem_105o/../info/../dem_105o/vat.adf.
...10...20...30...40...50...60...70...80...90...100 - done.

[Markus Neteler]

The problem is still there in GDAL 1.7.svn (using a 64bit Linux box):

gdalinfo $INPUT/dsm000038/                                                                   
Driver: AIG/Arc/Info Binary Grid
Files: /geodata/public/pat_DTM_DSM_LIDAR_2009_UTM_WGS84_dati_grezzi/DSM_BUFF_GRD/dsm_orto_grd1/dsm000038/
       /geodata/public/pat_DTM_DSM_LIDAR_2009_UTM_WGS84_dati_grezzi/DSM_BUFF_GRD/dsm_orto_grd1/dsm000038/w001001x.adf
       /geodata/public/pat_DTM_DSM_LIDAR_2009_UTM_WGS84_dati_grezzi/DSM_BUFF_GRD/dsm_orto_grd1/dsm000038/sta.adf
       /geodata/public/pat_DTM_DSM_LIDAR_2009_UTM_WGS84_dati_grezzi/DSM_BUFF_GRD/dsm_orto_grd1/dsm000038/w001001.adf
       /geodata/public/pat_DTM_DSM_LIDAR_2009_UTM_WGS84_dati_grezzi/DSM_BUFF_GRD/dsm_orto_grd1/dsm000038/log
       /geodata/public/pat_DTM_DSM_LIDAR_2009_UTM_WGS84_dati_grezzi/DSM_BUFF_GRD/dsm_orto_grd1/dsm000038/dblbnd.adf
       /geodata/public/pat_DTM_DSM_LIDAR_2009_UTM_WGS84_dati_grezzi/DSM_BUFF_GRD/dsm_orto_grd1/dsm000038/hdr.adf
Size is 2020, 2020
Coordinate System is `'
Origin = (713990.000000000000000,5152010.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Corner Coordinates:
Upper Left  (  713990.000, 5152010.000)
Lower Left  (  713990.000, 5149990.000)
Upper Right (  716010.000, 5152010.000)
Lower Right (  716010.000, 5149990.000)
Center      (  715000.000, 5151000.000)
Band 1 Block=505x4 Type=Float32, ColorInterp=Undefined
  Min=1730.546 Max=2535.939
  NoData Value=-3.4028234663852886e+38
ERROR 3: Attempt to read past EOF in /geodata/public/pat_DTM_DSM_LIDAR_2009_UTM_WGS84_dati_grezzi/DSM_BUFF_GRD/dsm_orto_grd1/dsm000038//../info/arc.dir.
ERROR 4: Failed to open table .VAT

I wonder if it is harmless or harmful.

[chaitanya]

This message is shown only when the file length of vat.adf is a multiple of 1024. The error message is displayed because the the end-of-file test failed at the end of the buffer.

I also noticed that the driver does not report/recover if the file ends abruptly in the middle of a record.

Fixed in trunk (r24578) and 1.9 branch (r24579).

[warmerdam]

Good work on the fix! This code comes from an upstream source, so I'm reopening the ticket and assigning it to Daniel to upstream the changes.

[peifer]

Replying to neteler:

The problem is still there in GDAL 1.7.svn (using a 64bit Linux box):

(...)

> ERROR 3: Attempt to read past EOF in /geodata/public/pat_DTM_DSM_LIDAR_2009_UTM_WGS84_dati_grezzi/DSM_BUFF_GRD/dsm_orto_grd1/dsm000038//../info/arc.dir.
> ERROR 4: Failed to open table .VAT

I am getting this error (now with a somewhat different wording) when using GDAL from trunk.

ERROR 3: EOF encountered in path/to/soil01/../info/arc.dir after reading 0 bytes while trying to read 32 bytes. File may be corrupt.
ERROR 4: Failed to open table soil01.VAT

As far as I can see: the error only occurs for Float32 rasters. None of them has a vat.adf file. The error does not occur when using gdalinfo on my Int16 rasters (which all have a vat.adf file). Using the -norat switch avoids the error.

[Even Rouault]

Replying to peifer:

I am getting this error (now with a somewhat different wording) when using GDAL from trunk.

ERROR 3: EOF encountered in path/to/soil01/../info/arc.dir after reading 0 bytes while trying to read 32 bytes. File may be corrupt.
ERROR 4: Failed to open table soil01.VAT

For the sake of completness, it would be good if you could attach a dataset that causes issues to the ticket. (concerning the different wording of the error message, it is in fact the same error. The wording was changed in r24578)

[peifer]

Replying to rouault:

For the sake of completness, it would be good if you could attach a dataset that causes issues to the ticket. (concerning the different wording of the error message, it is in fact the same error. The wording was changed in r24578)

An attachment would be too big for trac, so I uploaded some test data here: ​http://tinyurl.com/qyvnbvt

soil00 is an Int16 raster where gdalinfo works fine. soil01 is a Float32 raster where the mentioned ERROR 3 and ERROR 4 messages are thrown. I am using GDAL from svn on a 64-bit Linux box.

[EliL]

small attachment that produces error for #3031

[Even Rouault]

r26515 "AIGRID: Turn off errors that can be triggered if the info has no VAT table related with this coverage (#3031)"

[Markus Neteler]

Does it also fix trac #2447 ?

[Even Rouault]

Replying to neteler:

Does it also fix trac #2447 ?

Perhaps... The best is that you try with your dataset. It just silents an error when there's no .vat to read.