Opened 7 years ago

Closed 7 years ago

Last modified 7 years ago

#5530 closed defect (fixed)

vsizip access fails on grib file

Reported by: Kyle Shannon Owned by: warmerdam
Priority: normal Milestone: 1.11.1
Component: default Version: svn-trunk
Severity: normal Keywords: vsizip grib
Cc:

Description

I have a small grib file downloaded from http://nomads.ncep.noaa.gov/. I can read it fine with gdalinfo, but if it's zipped, it isn't accessible via /vsizip/:

kyle@kyle-workstation:~$ ls
gfs.t00z.mastergrb2f03
kyle@kyle-workstation:~$ gdalinfo  -nomd gfs.t00z.mastergrb2f03 
Driver: GRIB/GRIdded Binary (.grb)
Files: gfs.t00z.mastergrb2f03
Size is 9, 9
Coordinate System is:
GEOGCS["Coordinate System imported from GRIB file",
    DATUM["unknown",
        SPHEROID["Sphere",6371229,0]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433]]
Origin = (244.250000000000000,46.250000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Corner Coordinates:
Upper Left  (     244.250,      46.250) (244d15' 0.00"E, 46d15' 0.00"N)
Lower Left  (     244.250,      41.750) (244d15' 0.00"E, 41d45' 0.00"N)
Upper Right (     248.750,      46.250) (248d45' 0.00"E, 46d15' 0.00"N)
Lower Right (     248.750,      41.750) (248d45' 0.00"E, 41d45' 0.00"N)
Center      (     246.500,      44.000) (246d30' 0.00"E, 44d 0' 0.00"N)
Band 1 Block=9x1 Type=Float64, ColorInterp=Undefined
  Description = 2[m] HTGL="Specified height level above ground"
Band 2 Block=9x1 Type=Float64, ColorInterp=Undefined
  Description = 10[m] HTGL="Specified height level above ground"
Band 3 Block=9x1 Type=Float64, ColorInterp=Undefined
  Description = 10[m] HTGL="Specified height level above ground"
Band 4 Block=9x1 Type=Float64, ColorInterp=Undefined
  Description = 0[-] CCY="Convective cloud level"
kyle@kyle-workstation:~$ zip gfs.zip gfs.t00z.mastergrb2f03 
  adding: gfs.t00z.mastergrb2f03 (deflated 53%)
kyle@kyle-workstation:~$ gdalinfo  -nomd /vsizip/gfs.zip/gfs.t00z.mastergrb2f03 
ERROR 4: /vsizip/gfs.zip/gfs.t00z.mastergrb2f03 is a grib file, but no raster dataset was successfully identified.
gdalinfo failed - unable to open '/vsizip/gfs.zip/gfs.t00z.mastergrb2f03'.

Not sure where to start with the issue, I don't think it's the grib driver, but I have know idea. Grib file attached.

Attachments (2)

gfs.t00z.mastergrb2f03 (1001 bytes) - added by Kyle Shannon 7 years ago.
one_one.grib2 (358 bytes) - added by Kyle Shannon 7 years ago.

Download all attachments as: .zip

Change History (9)

Changed 7 years ago by Kyle Shannon

Attachment: gfs.t00z.mastergrb2f03 added

comment:1 Changed 7 years ago by Even Rouault

The Grib libraries use the POSIX file API and not the CPL ones

comment:2 Changed 7 years ago by Kyle Shannon

It works with other grib files, and it looks like it does use the VSI*L api:

kyle@kyle-hp-laptop:~/src/gdal/trunk/gdal$ grep -nH VSI frmts/grib/gribdataset.cpp 
frmts/grib/gribdataset.cpp:44:CPL_CVSID("$Id: gribdataset.cpp 27464 2014-06-20 20:40:51Z kyle $");
frmts/grib/gribdataset.cpp:76:    VSILFILE	*fp;
frmts/grib/gribdataset.cpp:195:    GIntBig nOffset = VSIFTellL( poGDS->fp );
frmts/grib/gribdataset.cpp:199:    VSIFSeekL( poGDS->fp, start+16, SEEK_SET );
frmts/grib/gribdataset.cpp:200:    VSIFReadL( abyHead, 5, 1, poGDS->fp );
frmts/grib/gribdataset.cpp:207:        if( VSIFSeekL( poGDS->fp, nSectSize-5, SEEK_CUR ) != 0
frmts/grib/gribdataset.cpp:208:            || VSIFReadL( abyHead, 5, 1, poGDS->fp ) != 1 )
frmts/grib/gribdataset.cpp:224:        VSIFReadL( pabyBody, 1, nSectSize-5, poGDS->fp );
frmts/grib/gribdataset.cpp:251:    VSIFSeekL( poGDS->fp, nOffset, SEEK_SET );
frmts/grib/gribdataset.cpp:518:        VSIFCloseL( fp );
frmts/grib/gribdataset.cpp:617:    poDS->fp = VSIFOpenL( poOpenInfo->pszFilename, "r" );
frmts/grib/gribdataset.cpp:647:    VSIFSeekL( poDS->fp, 0, SEEK_SET );

file that works attached.

Changed 7 years ago by Kyle Shannon

Attachment: one_one.grib2 added

comment:3 Changed 7 years ago by Even Rouault

Hum OK but perhaps not in all parts of the degrib lib. Or an issue with our conformance with POSIX

comment:4 Changed 7 years ago by Kyle Shannon

Ah, okay. I see Even, thanks.

comment:5 Changed 7 years ago by Kyle Shannon

Even, I can do two things that get it to work, but I'm not certain why. It feels like the VSIBufferedReaderHandle class used with the VSIGZipHandle may be causing the issue. If I return a straight VSIGZipHandle from VSIZipFilesystemHandler::Open(), then it works fine. Also, by accident, if I force the VSIGZipHandle to seek, it works as well:

cpl_vsil_buffered_reader.cpp:152

+ bNeedBaseHandleSeek = TRUE;
  if (bNeedBaseHandleSeek)
      poBaseHandle->Seek(nBufferOffset + nBufferSize, SEEK_SET);

I couldn't track it down in the actual gzip code, but it seems like eof is getting set prematurely in the GZipHandler (or z_eof), but I am not sure why. Maybe a Seek() is getting missed, but I am unfamiliar with that code. The grib VSI stuff is fairly straight forward, as it just replaces the posix code internally. I'm going to keep messing with it, hopefully I'll find something more.

comment:6 Changed 7 years ago by Even Rouault

Milestone: 1.11.1
Resolution: fixed
Status: newclosed

trunk r27509, branches/1.11 r27510 "/vsizip/ : fix bug that caused premature end of file condition with some read patterns (#5530)"

comment:7 Changed 7 years ago by Kyle Shannon

Thanks Even.

Note: See TracTickets for help on using tickets.