vsizip access fails on grib file

[Kyle Shannon]

I have a small grib file downloaded from ​http://nomads.ncep.noaa.gov/. I can read it fine with gdalinfo, but if it's zipped, it isn't accessible via /vsizip/:

kyle@kyle-workstation:~$ ls
gfs.t00z.mastergrb2f03
kyle@kyle-workstation:~$ gdalinfo  -nomd gfs.t00z.mastergrb2f03 
Driver: GRIB/GRIdded Binary (.grb)
Files: gfs.t00z.mastergrb2f03
Size is 9, 9
Coordinate System is:
GEOGCS["Coordinate System imported from GRIB file",
    DATUM["unknown",
        SPHEROID["Sphere",6371229,0]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433]]
Origin = (244.250000000000000,46.250000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Corner Coordinates:
Upper Left  (     244.250,      46.250) (244d15' 0.00"E, 46d15' 0.00"N)
Lower Left  (     244.250,      41.750) (244d15' 0.00"E, 41d45' 0.00"N)
Upper Right (     248.750,      46.250) (248d45' 0.00"E, 46d15' 0.00"N)
Lower Right (     248.750,      41.750) (248d45' 0.00"E, 41d45' 0.00"N)
Center      (     246.500,      44.000) (246d30' 0.00"E, 44d 0' 0.00"N)
Band 1 Block=9x1 Type=Float64, ColorInterp=Undefined
  Description = 2[m] HTGL="Specified height level above ground"
Band 2 Block=9x1 Type=Float64, ColorInterp=Undefined
  Description = 10[m] HTGL="Specified height level above ground"
Band 3 Block=9x1 Type=Float64, ColorInterp=Undefined
  Description = 10[m] HTGL="Specified height level above ground"
Band 4 Block=9x1 Type=Float64, ColorInterp=Undefined
  Description = 0[-] CCY="Convective cloud level"
kyle@kyle-workstation:~$ zip gfs.zip gfs.t00z.mastergrb2f03 
  adding: gfs.t00z.mastergrb2f03 (deflated 53%)
kyle@kyle-workstation:~$ gdalinfo  -nomd /vsizip/gfs.zip/gfs.t00z.mastergrb2f03 
ERROR 4: /vsizip/gfs.zip/gfs.t00z.mastergrb2f03 is a grib file, but no raster dataset was successfully identified.
gdalinfo failed - unable to open '/vsizip/gfs.zip/gfs.t00z.mastergrb2f03'.

Not sure where to start with the issue, I don't think it's the grib driver, but I have know idea. Grib file attached.

[Even Rouault]

The Grib libraries use the POSIX file API and not the CPL ones

[Kyle Shannon]

It works with other grib files, and it looks like it does use the VSI*L api:

kyle@kyle-hp-laptop:~/src/gdal/trunk/gdal$ grep -nH VSI frmts/grib/gribdataset.cpp 
frmts/grib/gribdataset.cpp:44:CPL_CVSID("$Id: gribdataset.cpp 27464 2014-06-20 20:40:51Z kyle $");
frmts/grib/gribdataset.cpp:76:    VSILFILE	*fp;
frmts/grib/gribdataset.cpp:195:    GIntBig nOffset = VSIFTellL( poGDS->fp );
frmts/grib/gribdataset.cpp:199:    VSIFSeekL( poGDS->fp, start+16, SEEK_SET );
frmts/grib/gribdataset.cpp:200:    VSIFReadL( abyHead, 5, 1, poGDS->fp );
frmts/grib/gribdataset.cpp:207:        if( VSIFSeekL( poGDS->fp, nSectSize-5, SEEK_CUR ) != 0
frmts/grib/gribdataset.cpp:208:            || VSIFReadL( abyHead, 5, 1, poGDS->fp ) != 1 )
frmts/grib/gribdataset.cpp:224:        VSIFReadL( pabyBody, 1, nSectSize-5, poGDS->fp );
frmts/grib/gribdataset.cpp:251:    VSIFSeekL( poGDS->fp, nOffset, SEEK_SET );
frmts/grib/gribdataset.cpp:518:        VSIFCloseL( fp );
frmts/grib/gribdataset.cpp:617:    poDS->fp = VSIFOpenL( poOpenInfo->pszFilename, "r" );
frmts/grib/gribdataset.cpp:647:    VSIFSeekL( poDS->fp, 0, SEEK_SET );

file that works attached.

[Even Rouault]

Hum OK but perhaps not in all parts of the degrib lib. Or an issue with our conformance with POSIX

[Kyle Shannon]

Ah, okay. I see Even, thanks.

[Kyle Shannon]

Even, I can do two things that get it to work, but I'm not certain why. It feels like the VSIBufferedReaderHandle class used with the VSIGZipHandle may be causing the issue. If I return a straight VSIGZipHandle from VSIZipFilesystemHandler::Open(), then it works fine. Also, by accident, if I force the VSIGZipHandle to seek, it works as well:

cpl_vsil_buffered_reader.cpp:152

+ bNeedBaseHandleSeek = TRUE;
  if (bNeedBaseHandleSeek)
      poBaseHandle->Seek(nBufferOffset + nBufferSize, SEEK_SET);

I couldn't track it down in the actual gzip code, but it seems like eof is getting set prematurely in the GZipHandler (or z_eof), but I am not sure why. Maybe a Seek() is getting missed, but I am unfamiliar with that code. The grib VSI stuff is fairly straight forward, as it just replaces the posix code internally. I'm going to keep messing with it, hopefully I'll find something more.

[Even Rouault]

trunk r27509, branches/1.11 r27510 "/vsizip/ : fix bug that caused premature end of file condition with some read patterns (#5530)"

[Kyle Shannon]

Thanks Even.