read gzipped files in situ (e.g. raster.tif.gz)

A special file handler something like the one used for in-memory files could
be implemented to achieve this.  However, it is not partiuclarly high on my
personal priority list.  Due to challenges seeking in compressed data
streams it might be necessary to decompress the whole file into RAM which 
would not scale well to large datasets. 

Here's a patch that adds a new class, VSIGZipHandle, which implements the large file API and enable the transparent use of gzip'd file by drivers using the large file API. It only supports reading.

In VSIFOpenL, we look for the first 2 bytes. If they match the magic header of GZip file, we wrap the poVirtualHandle with a VSIGZipHandle object.

It required a change in GDALOpenInfo to first try to open the file with the large file API, instead of first trying with the old file API. This way the drivers will see the uncompressed stream as pabyHeader instead of the compressed one.

Tested successfully with GTiff and NITF drivers. A bit slower than access to uncompressed file on a MODIS and Ikonos TIFF image, but usable. A gdalinfo on a .tif.gz is slow since for some reason the GTiff driver seeks nearly to the end of the TIF file.

I didn't do any effort to make seeking fast, so depending on the way the file is accessed, it may be really slow. I've tested it on onearth images, and... it's actually slow.

Add capability of reading .gz files transparently from GDAL drivers

Even,

I think we should avoid the changes in GDALOpen and CPLReadDir. Instead I think gzip files should be accessed with the filename prefix /vsizip/ with the remainder of the path being the real path. So the abc.png file in /usr/data/def.zip would have a virtual filename "/vsizip/usr/data/def.zip/abc.png".

This wouldn't address how to navigate into a zip file directly or how to GDALOpen() it directly, but I think that aspect needs to be carefully considered.

Access to TIFF files in .zip files will generally suck because the TIFF format uses a lot of seeking, and typically the directory (the first thing read) is at the end of the file.

I'm attaching an improved version of the patch that adds quite fast random seek. It is done thanks the regular creation of 'snapshots' by using inflateCopy that can dump the gzip state. Performance when using .tif.gz in openev is thus much improved (after the initial seek at the end of the file that can take some time, but I don't see how this could be avoided) and makes them usable on the test cases I mentionned in my first post.

Frank, as far as your remarks are concerned, this version doesn't take them into account yet as I've a few remarks/questions too :

• I think we should use rather /vsigzip for GZIP files and keep /vsizip for ZIP (PkZIP) files
• The syntax /vsizip/usr/data/def.zip/abc.png seems a good idaa of ZIP files. But for gzip ones, IMHO, it doesn't make much sense, except if the .gz is in fact a .tar.gz
• The syntax /vsigzip/ should be kept intern to GDAL or for advanced users. I think we must find a way such that a "openev foo.xxx.gz" works
• For a .tar.gz, the user should be able to do "openev /usr/data/def.tar.gz/abc.png"
• For a .zip, the user should be able to do "openev /usr/data/def.zip/abc.png". But if the .zip is made of a single file (SRTM HGT files can be downloaded as single file ZIP archive for example), "openev /usr/data/def.zip" should work too.

All in all, the main question is : where to put the magic to translate from the "user friendly filename" to the GDAL virtual filename.

Another improved version of the patch that adds support for .zip files. Reading of zip files is done through unzip.c from contrib/minizip in zlib-1.2.3. With the zip file handler, I can read GTiff, BT and CADRG datasets included in .zip files.

I've also added /vsigzip and /vsizip FilesystemHandler?.

However I've still kept/added in VSIFOpenL, VSIFStatL and VSIReadDir the logic that automagically translates from the natural filename (/usr/data/def.zip/abc.png, foo.xxx.gz) to the virtual filename (/vsizip/usr/data/def.zip/abc.png, /vsigzip/foo.xxx.gz). I don't really see how to do it differently.

To sum up the current possibilities, one can do :

• gdalinfo foo.gz
• gdalinfo /vsigzip/foo.gz
• gdalinfo foo.zip (if the zip file contains only 1 file)
• gdalinfo /vsizip/foo.zip (if the zip file contains only 1 file)
• gdalinfo foo.zip/foo.XXX
• gdalinfo /vsizip/foo.zip/foo.XXX

Patch updated with changes required in srtmhgtdataset.cpp such as .hgt.zip downloaded from ftp://e0srp01u.ecs.nasa.gov/srtm/version2 can be read as such.

Patch updated to support stored (uncompressed) file in .zip