Changes between Version 5 and Version 6 of CloudOptimizedGeoTIFF


Ignore:
Timestamp:
Jan 31, 2017, 7:14:50 AM (7 years ago)
Author:
Even Rouault
Comment:

More performance testing

Legend:

Unmodified
Added
Removed
Modified

  • CloudOptimizedGeoTIFF

    v5 v6  
    7474The source image is the True Color Image of a Sentinel 2A L1C product (10980x10980 pixels, RGB bands of type Byte)
    7575
    76 Creation of a "regular" GeoTIFF with overviews:
     76Creation of a regular GeoTIFF without overviews:
     77
     78{{{
     79gdal_translate SENTINEL2_L1C:S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441.SAFE/MTD_MSIL1C.xml:TCI:EPSG_32630 S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif -co TILED=YES -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR
     80}}}
     81
     82Creation of a regular GeoTIFF with overviews:
    7783
    7884{{{
    … …  
    9197=== Reading a single pixel ===
    9298
    93 * Regular GeoTIFF:
     99* Regular GeoTIFF without overviews:
     100
     101{{{
     102$ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdallocationinfo --debug on \
     103  /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif  5000 5000
     104
     105VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)=2123472  response_code=200
     106VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)...
     107VSICURL: Got response_code=206
     108VSICURL: Downloading 16384-49151 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)...
     109VSICURL: Got response_code=206
     110GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif, this=0x108a4c0) succeeds as GTiff.
     111Report:
     112  Location: (5000P,5000L)
     113  Band 1:
     114GDAL: GDAL_CACHEMAX = 791 MB
     115VSICURL: Downloading 1556480-1572863 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)...
     116VSICURL: Got response_code=206
     117    Value: 255
     118  Band 2:
     119    Value: 255
     120  Band 3:
     121    Value: 255
     122}}}
     123
     124* Regular GeoTIFF with overviews:
    94125
    95126{{{
    … …  
    149180}}}
    150181
    151 No significant time difference (individual runs may differ by a few tens of milliseconds). Same amount of I/O (64 KB read). Note the use of CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif to avoid reading any side car files (.aux.xml, etc...) and GDAL_DISABLE_READDIR_ON_OPEN=YES to avoid any attempt of listing the files in the same directory.
     182No significant time difference (individual runs may differ by a few tens of milliseconds). Same amount of I/O (64 KB read).
     183The fact that we even read 16 + 32 KB in the case without overviews is due to the fact that the first IFD is slightly larger than 16 KB, so we need to read a bit more (and the heuristics double the chunk size as this is contiguous to the previous region read).
     184
     185Note the use of CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif to avoid reading any side car files (.aux.xml, etc...) and GDAL_DISABLE_READDIR_ON_OPEN=YES to avoid any attempt of listing the files in the same directory.
    152186
    153187Same conclusions if using a AWS S3 hosting, with both GDAL_DISABLE_READDIR_ON_OPEN=YES and CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif defined as well.
    … …  
    156190=== Reading a block of pixels at full resolution ===
    157191
    158 * Regular GeoTIFF:
     192* Regular GeoTIFF without overviews
     193
     194{{{
     195$ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdal_translate --debug on \
     196  /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif \
     197  -srcwin 1024 1024 256 256 out.tif
     198VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)=2123472  response_code=200
     199VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)...
     200VSICURL: Got response_code=206
     201VSICURL: Downloading 16384-49151 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)...
     202VSICURL: Got response_code=206
     203GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif, this=0xce8660) succeeds as GTiff.
     204Input file size is 10980, 10980
     205GDAL: GDALDefaultOverviews::OverviewScan()
     206...10...20...30...40...50...60...70...80...90...100 - done.
     207
     208real    0m0.527s
     209user    0m0.084s
     210sys     0m0.020s
     211}}}
     212
     213* Regular GeoTIFF with overviews
    159214
    160215{{{
    … …  
    256311=== Getting a subsampled version of the image ===
    257312
    258 * Regular GeoTIFF:
     313* Regular GeoTIFF without overviews:
     314{{{
     315$ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdal_translate --debug on  /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif out.tif -outsize 1% 1%
     316VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)=2123472  response_code=200
     317VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)...
     318VSICURL: Got response_code=206
     319VSICURL: Downloading 16384-49151 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)...
     320VSICURL: Got response_code=206
     321GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif, this=0x7c3650) succeeds as GTiff.
     322Input file size is 10980, 10980
     323GDAL: GDALDefaultOverviews::OverviewScan()
     324GDAL: GDAL_CACHEMAX = 791 MB
     325GDAL: GDALDatasetCopyWholeRaster(): 109*109 swaths, bInterleave=1
     326GDAL: GDALDefaultOverviews::OverviewScan()
     327GDAL: GDALDefaultOverviews::OverviewScan()
     328GDAL: Potential thrashing on band 1 of /vsimem/sparse_0x7d0e10.
     329VSICURL: Downloading 49152-114687 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)...
     330VSICURL: Got response_code=206
     331VSICURL: Downloading 114688-245759 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)...
     332VSICURL: Got response_code=206
     333VSICURL: Downloading 245760-507903 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)...
     334VSICURL: Got response_code=206
     335VSICURL: Downloading 507904-1032191 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)...
     336VSICURL: Got response_code=206
     337VSICURL: Downloading 1032192-2080767 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)...
     338VSICURL: Got response_code=206
     339VSICURL: Downloading 2080768-2123471 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)...
     340VSICURL: Got response_code=206
     341...10...20...30...40...50...60...70...80...90...100 - done.
     342GDAL: GDALClose(out.tif, this=0x83bdc0)
     343GDAL: 59168 block reads on 32 block band 1 of /vsimem/sparse_0x7d0e10.
     344
     345real    0m2.960s
     346user    0m0.740s
     347sys     0m0.096s
     348}}}
     349
     350As one could anticipate, the whole file needs to be read.
     351
     352* Regular GeoTIFF with overviews:
    259353
    260354{{{
    … …  
    325419
    326420As the IFD are at the beginning of the files, as well as the pixel data for the smallest overview, the request can be completed with the 2 first HTTP GET requests (this is a bit of an extreme case of course)
     421
     422=== Conclusions ===
     423
     424* GDAL can make an efficient use of overviews, even on files hosted on a HTTP storage
     425* The cloud compatible file organization can reduce a bit the amount of GET requests required.