Changes between Version 7 and Version 8 of CloudOptimizedGeoTIFF


Ignore:
Timestamp:
Jan 31, 2017, 11:40:00 AM (7 years ago)
Author:
Even Rouault
Comment:

More benchmarking

Legend:

Unmodified
Added
Removed
Modified

  • CloudOptimizedGeoTIFF

    v7 v8  
    7474The source image is the True Color Image of a Sentinel 2A L1C product (10980x10980 pixels, RGB bands of type Byte)
    7575
    76 Creation of a regular GeoTIFF without overviews:
     76Creation of a strip oriented GeoTIFF without overviews ("traditional"):
     77
     78{{{
     79gdal_translate SENTINEL2_L1C:S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441.SAFE/MTD_MSIL1C.xml:TCI:EPSG_32630 \
     80               S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif \
     81               -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR
     82}}}
     83
     84Creation of a tiled GeoTIFF without overviews:
    7785
    7886{{{
    … …  
    8290}}}
    8391
    84 Creation of a regular GeoTIFF with overviews:
     92Creation of a tiled GeoTIFF with overviews:
    8593
    8694{{{
    … …  
    103111}}}
    104112
     113Creation of a cloud optimized GeoTIFF with tiles of dimension 512x512
     114
     115{{{
     116gdal_translate SENTINEL2_L1C:S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441.SAFE/MTD_MSIL1C.xml:TCI:EPSG_32630 \
     117               S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI.tif \
     118               -co TILED=YES -co COMPRESS=DEFLATE
     119gdaladdo -r average  S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI.tif 2 4 8 16 32
     120gdal_translate S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI.tif \
     121               S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif \
     122               -co TILED=YES -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -co COPY_SRC_OVERVIEWS=YES \
     123               -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 --config GDAL_TIFF_OVR_BLOCKSIZE 512
     124}}}
     125
    105126=== Reading a single pixel ===
    106127
    107 * Regular GeoTIFF without overviews:
     128* Traditional GeoTIFF
     129
     130{{{
     131$ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdallocationinfo --debug on \
     132    /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif 5000 5000
     133VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)=2009538  response_code=200
     134VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     135VSICURL: Got response_code=206
     136GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif, this=0x1910590) succeeds as GTiff.
     137Report:
     138  Location: (5000P,5000L)
     139  Band 1:
     140GDAL: GDAL_CACHEMAX = 791 MB
     141VSICURL: Downloading 1523712-1540095 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     142VSICURL: Got response_code=206
     143    Value: 255
     144  Band 2:
     145    Value: 255
     146  Band 3:
     147    Value: 255
     148
     149real    0m0.397s
     150user    0m0.060s
     151sys     0m0.024s
     152}}}
     153
     154* Tiled GeoTIFF without overviews:
    108155
    109156{{{
    … …  
    188235}}}
    189236
    190 No significant time difference (individual runs may differ by a few tens of milliseconds). Same amount of I/O (64 KB read).
     237* Cloud optimized GeoTIFF with 512x512 tiles
     238
     239{{{
     240$ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdallocationinfo --debug on \
     241   /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif 5000 5000
     242VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif)=2918044  response_code=200
     243VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif)...
     244VSICURL: Got response_code=206
     245GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif, this=0x15cd450) succeeds as GTiff.
     246Report:
     247  Location: (5000P,5000L)
     248  Band 1:
     249GDAL: GDAL_CACHEMAX = 791 MB
     250VSICURL: Downloading 2392064-2408447 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif)...
     251VSICURL: Got response_code=206
     252    Value: 255
     253  Band 2:
     254    Value: 255
     255  Band 3:
     256    Value: 255
     257GDAL: GDALClose(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif, this=0x15cd450)
     258
     259real    0m0.407s
     260user    0m0.068s
     261sys     0m0.016s
     262}}}
     263
     264The winners are the traditional GeoTIFF organization and the cloud optimized 512x512 tiled GeoTIFF since the directory fits in the first 16 KB.
     265
     266In other cases, no significant time difference (individual runs may differ by a few tens of milliseconds). Same amount of I/O (64 KB read).
    191267The fact that we even read 16 + 32 KB in the case without overviews is due to the fact that the first IFD is slightly larger than 16 KB, so we need to read a bit more (and the heuristics double the chunk size as this is contiguous to the previous region read).
    192268
    … …  
    198274=== Reading a block of pixels at full resolution ===
    199275
    200 * Regular GeoTIFF without overviews
     276* Traditional GeoTIFF
     277
     278{{{
     279$ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdal_translate --debug on \
     280   /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif \
     281   -srcwin 1024 1024 256 256 out.tif
     282VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)=2009538  response_code=200
     283VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     284VSICURL: Got response_code=206
     285GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif, this=0xf1e5a0) succeeds as GTiff.
     286Input file size is 10980, 10980
     287GDAL: GDALDefaultOverviews::OverviewScan()
     288VSICURL: Downloading 163840-196607 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     289VSICURL: Got response_code=206
     290VSICURL: Downloading 196608-262143 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     291VSICURL: Got response_code=206
     292VSICURL: Downloading 262144-393215 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     293VSICURL: Got response_code=206
     294...10...20...30...40...50...60...70...80...90...100 - done.
     295
     296real    0m0.923s
     297user    0m0.256s
     298sys     0m0.020s
     299}}}
     300
     301The access to the pixel values requires 3 GET requests since whole lines need to be downloaded. A smarter implementation in the GeoTIFF reader with a better interaction with the GDAL HTTP code could probably collapse the 3 GET into a single larger one.
     302
     303* Tiled GeoTIFF without overviews:
    201304
    202305{{{
    … …  
    219322}}}
    220323
    221 * Regular GeoTIFF with overviews
     324* Regular GeoTIFF with overviews:
    222325
    223326{{{
    … …  
    317420No need to specify -mask none to get the maximum performance: as the IFD are at the beginning of the files, they have been fetched with the 2 first HTTP GET requests.
    318421
     422
     423* Cloud optimized GeoTIFF with tiles 512x512:
     424
     425{{{
     426$ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdal_translate --debug on \
     427  /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif \
     428  -srcwin 1024 1024 256 256 out.tif
     429VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif)=2918044  response_code=200
     430VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif)...
     431VSICURL: Got response_code=206
     432GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif, this=0x1c815c0) succeeds as GTiff.
     433Input file size is 10980, 10980
     434GTiff: ScanDirectories()
     435GTiff: Opened 5490x5490 overview.
     436GTiff: Opened 2745x2745 overview.
     437GTiff: Opened 1373x1373 overview.
     438GTiff: Opened 687x687 overview.
     439GTiff: Opened 344x344 overview.
     440GDAL: GDALDefaultOverviews::OverviewScan()
     441VSICURL: Downloading 1114112-1146879 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif)...
     442VSICURL: Got response_code=206
     443...10...20...30...40...50...60...70...80...90...100 - done.
     444
     445real    0m0.441s
     446user    0m0.096s
     447sys     0m0.016s
     448}}}
     449
     450Compared to the default cloud optimized GeoTIFF, we save a few bytes for the reading of the IFD since all IFDs fit into the first 16 KB
     451
     452
    319453=== Getting a subsampled version of the image ===
    320454
    321 * Regular GeoTIFF without overviews:
    322 {{{
    323 $ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdal_translate --debug on  /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif out.tif -outsize 1% 1%
     455* Traditional GeoTIFF
     456
     457{{{
     458$ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdal_translate --debug on \
     459     /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif \
     460     out.tif -outsize 1% 1%
     461VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)=2009538  response_code=200
     462VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     463VSICURL: Got response_code=206
     464GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif, this=0x2327590) succeeds as GTiff.
     465Input file size is 10980, 10980
     466GDAL: GDALDefaultOverviews::OverviewScan()
     467GDAL: GDALDefaultOverviews::OverviewScan()
     468VSICURL: Downloading 16384-49151 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     469VSICURL: Got response_code=206
     470VSICURL: Downloading 49152-114687 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     471VSICURL: Got response_code=206
     472VSICURL: Downloading 114688-245759 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     473VSICURL: Got response_code=206
     474VSICURL: Downloading 245760-507903 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     475VSICURL: Got response_code=206
     476VSICURL: Downloading 507904-1032191 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     477VSICURL: Got response_code=206
     478VSICURL: Downloading 1064960-1081343 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     479VSICURL: Got response_code=206
     480VSICURL: Downloading 1097728-1114111 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     481VSICURL: Got response_code=206
     482VSICURL: Downloading 1130496-1146879 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     483VSICURL: Got response_code=206
     484VSICURL: Downloading 1163264-1179647 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     485VSICURL: Got response_code=206
     486VSICURL: Downloading 1196032-1212415 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     487VSICURL: Got response_code=206
     488VSICURL: Downloading 1212416-1245183 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     489VSICURL: Got response_code=206
     490VSICURL: Downloading 1245184-1310719 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     491VSICURL: Got response_code=206
     492VSICURL: Downloading 1310720-1441791 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     493VSICURL: Got response_code=206
     494VSICURL: Downloading 1441792-1703935 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     495VSICURL: Got response_code=206
     496VSICURL: Downloading 1703936-2009537 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)...
     497VSICURL: Got response_code=206
     498...10...20...30...40...50...60...70...80...90...100 - done.
     499
     500real    0m3.197s
     501user    0m0.304s
     502sys     0m0.056s
     503}}}
     504
     505As one could anticipate, the whole file needs to be read.
     506
     507* Tiled GeoTIFF without overviews:
     508{{{
     509$ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdal_translate --debug on \
     510    /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif out.tif -outsize 1% 1%
    324511VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)=2123472  response_code=200
    325512VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)...
    … …  
    428615As the IFD are at the beginning of the files, as well as the pixel data for the smallest overview, the request can be completed with the 2 first HTTP GET requests (this is a bit of an extreme case of course)
    429616
     617* Cloud optimized GeoTIFF with tiles of 512x512:
     618
     619{{{
     620$ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdal_translate --debug on \
     621    /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif \
     622    out.tif -outsize 1% 1%
     623VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif)=2918044  response_code=200
     624VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif)...
     625VSICURL: Got response_code=206
     626GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif, this=0x152b5b0) succeeds as GTiff.
     627Input file size is 10980, 10980
     628GTiff: ScanDirectories()
     629GTiff: Opened 5490x5490 overview.
     630GTiff: Opened 2745x2745 overview.
     631GTiff: Opened 1373x1373 overview.
     632GTiff: Opened 687x687 overview.
     633GTiff: Opened 344x344 overview.
     634VSICURL: Downloading 16384-49151 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif)...
     635VSICURL: Got response_code=206
     636...10...20...30...40...50...60...70...80...90...100 - done.
     637real    0m0.439s
     638user    0m0.088s
     639sys     0m0.024s
     640}}}
     641
     642Same performance as the default cloud optimized case (GDAL could potentially read in a less gready way for the pixel data since the tile size is only 8KB here)
     643
    430644=== Conclusions ===
    431645
    432646* GDAL can make an efficient use of overviews, even on files hosted on a HTTP storage
    433647* The cloud compatible file organization can reduce a bit the amount of GET requests required.
     648* Increasing the block size can reduce the size of the IFD. But larger blocks can cause more bytes to be pulled for random access if the compression rate is not high.