New Landsat-8 metadata format support

[dtesta]

I'm attaching a patch to add support for the new Landsat-8 metadata format.

The patch was made against the trunk.

I'd like to know if there is any interest in this driver and if it would be possible to add it to the official GDAL drivers set.

[etourigny]

a few remarks - disclaimer: I just skimmed through the code and didn't test it

• you should add some metadata like GDAL_DMD_EXTENSION, GDAL_DMD_SUBDATASETS and GDAL_DCAP_RASTER (new in gdal 2.0)
• a help topic would also be nice (and advertised as a metadata item)
• is there a particular compelling reason to use a hash container instead of CPLStringList? It's a good idea and should be used elsewhere, but I'm wondering if it's really necessary. Does it offer a significant speed increase?

-it would be nice to have a small dataset to test this with (instead of downloading an entire landsat .tar.gz file), but I'm not sure if it's possible

[dtesta]

• you should add some metadata like GDAL_DMD_EXTENSION, GDAL_DMD_SUBDATASETS

and GDAL_DCAP_RASTER (new in gdal 2.0)

Done. Added to fastv6dataset.cpp (I took as an example HDF5)

• a help topic would also be nice (and advertised as a metadata item)

Done. Added to both fastv6dataset.cpp and fastv6imagedataset.cpp

• is there a particular compelling reason to use a hash container instead

of CPLStringList? It's a good idea and should be used elsewhere, but I'm wondering if it's really necessary. Does it offer a significant speed increase?

I used that because I needed to hold a hashtable of hashtables. In this particular case to associate a (variable?) set of groups (names) with a set of key-value pairs. I've just implemented a basic hash container that doesn't have some things like a load factor to measure how full the hashtable is allowed to get before its capacity is automatically increased for instance.

• it would be nice to have a small dataset to test this with (instead of

downloading an entire landsat .tar.gz file), but I'm not sure if it's possible

I'm not sure I can share the exact same set of images I'm using right now but could try to create a set of images I could share.

[dtesta]

I've just updated the patch.

Thanks! :)

[Even Rouault]

Etienne, still interesting in handling this ? (personnaly I'm not sure putting the hash table in the public interface is really needed. I know I authored cpl_hash_set, but wondering if it was a good idea. Probably std::set is good enough)

[etourigny]

sorry I am on a long-term vacation, better not count on me...

[warmerdam]

A few notes:

• I'm not clear on why the metadata wouldn't just be flattened into a normal CPLStringList key/value set.
• Are thermal and reflective width/height ever different?
• why not just treat the reflective, thermal and qa bands all as bands of the primary dataset and just let the PAN band be a subdataset since it is a different size?
• somehow I had it my head there was already an _MTL.TXT reader somewhere in GDAL - ah, I think I am thinking this because of similarity to the PDS label file reader (frmts/pds/nasakeywordhandler.cpp) which might be repurposable.
• For test data, a real _MTL.TXT file and all-zero compressed TIFFs would be fine to keep things small.
• it would be nice to conform that this driver works against the new landsat-pds archive at Amazon I help maintain (ie. ​https://s3-us-west-2.amazonaws.com/landsat-pds/L8/009/010/LC80090102015043LGN00/index.html)

Generally a very impressive set of work!

[dtesta]

I'm not clear on why the metadata wouldn't just be flattened into a normal CPLStringList key/value set.

Just thought that loading everything at once (the whole _MTL.txt file) and then working with that would be better and easier. There's metadata that I'm not loading yet.

Are thermal and reflective width/height ever different?

Different sensor and different pixel size so they might.

why not just treat the reflective, thermal and qa bands all as bands of the primary dataset and just let the PAN band be a subdataset since it is a different size?

That's because in this way I can treat each set of bands as a whole. They are grouped by type. Instead of specifying each band from a primary dataset I know that the thermal subdataset contains bands related to thermal data, the reflective subdataset contains bands related to reflective data and so on.

somehow I had it my head there was already an _MTL.TXT reader somewhere in GDAL - ah, I think I am thinking this because of similarity to the PDS label file reader (frmts/pds/nasakeywordhandler.cpp) which might be repurposable.

When I tried with version 1.11.0 I did not find support for this.

For test data, a real _MTL.TXT file and all-zero compressed TIFFs would be fine to keep things small.

Attaching to the TCK along with an updated patch.

it would be nice to conform that this driver works against the new landsat-pds archive at Amazon I help maintain (ie. ​https://s3-us-west-2.amazonaws.com/landsat-pds/L8/009/010/LC80090102015043LGN00/index.html)

I'll try with it and let you know.

[Jukka Rahkonen]

GDAL since version 2.0 can read metadata from _MTL.TXT with ​https://svn.osgeo.org/gdal/trunk/gdal/gcore/mdreader/reader_landsat.cpp

I suppose that this patch won't be utilized as it and this ticket can be closed. Create a new ticket about improving the existing implementation if this one has some advantages.

[dtesta]

Hi, please take a look at the documentation in the patch. With the patch you don't just read the metadata but you can also open the raster files. You can read the files as if you were opening a regular raster file by just reading the MTL file.

By reading the documentation you'll find examples using gdalinfo and even gdal_translate.

Regards, Daniel.

[Jukka Rahkonen]

I understand better now after reading the comments and mail thread ​http://osgeo-org.1560.x6.nabble.com/gdal-dev-new-landsat-8-metadata-format-td5140773.html.

[Even Rouault]

This ticket has been automatically closed because Trac is no longer used for GDAL bug tracking, since the project has migrated to GitHub. If you believe this ticket is still valid, you may file it to ​https://github.com/OSGeo/gdal/issues if it is not already reported there.