[[TOC]] = Cloud optimized GeoTIFF = == Definition == A cloud optimized GeoTIFF is a regular GeoTIFF file, aimed at being hosted on a HTTP file server, whose internal organization is friendly for consumption by clients issuing [https://tools.ietf.org/html/rfc7233 HTTP GET range request] ("bytes: start_offset-end_offset" HTTP header). It contains at its beginning the metadata of the full resolution imagery, followed by the optional presence of overview metadata, and finally the imagery itself. To make it friendly with streaming and progressive rendering, we recommand starting with the imagery of the smallest overview and finishing with the imagery of the full resolution level. More formally, the structure of such a file is: * TIFF / BigTIFF signature * IFD ([http://www.awaresystems.be/imaging/tiff/faq.html#q3 Image File Directory]) of full resolution image * Values of TIFF tags that don't fit inline in the IFD directory, such as TileOffsets, TileByteCounts and GeoTIFF keys * Optional: IFD (Image File Directory) of first overview (typically subsampled by a factor of 2), followed by the values of its tags that don't fit inline * Optional: IFD (Image File Directory) of second overview (typically subsampled by a factor of 4), followed by the values of its tags that don't fit inline * ... * Optional: IFD (Image File Directory) of last overview (typically subsampled by a factor of 2^N^), followed by the values of its tags that don't fit inline * Optional: tile content of last overview level * ... * Optional: tile content of first overview level * Tile content of full resolution image. == Unspecified points == * size of tile: 256 or 512 pixels are typical however * compression methods allowed. typically, no compression, DEFLATE or LZW can be used for lossless, or JPEG for lossy. (note that DEFLATE while more efficient can LZW can cause compatibility issues with some software packages) == How to generate it with GDAL == Given an input dataset in.tif with already generated internal or external overviews, a cloud optimized GeoTIFF can be generated with: gdal_translate in.tif out.tif -co TILED=YES -co COPY_SRC_OVERVIEWS=YES -co COMPRESS=LZW This will result in a images with tiles of dimension 256x256 pixel for main resolution, and 128x128 tiles for overviews. For an image of 4096x4096 with 4 overview levels, the 5 IFDs and their TileOffsets and TileByteCounts tag data fit into the first 6KB of the file. Note: for JPEG compression, the above method produce cloud optimized files only if using GDAL 2.2 (or a dev version >= r36879). For older versions, the IFD of the overviews will be written towards the end of the file. A recent version of GDAL (2.2 or dev version >= r37257) built against internal libtiff (or libtiff >= 4.0.8) will also help reducing the amount of bytes read for JPEG compressed files with YCbCr subsampling. == How to read it with GDAL == GDAL includes special filesystems that can read a file hosted on a HTTP/FTP server by chunks. The base filesystem is [http://gdal.org/cpl__vsi_8h.html#a4f791960f2d86713d16e99e9c0c36258 /vsicurl/] (Virtual System Interface for Curl) and the filename it accepts are of the form "/vsicurl/http://example.com/path/to/some.tif". They can be used whereever GDAL expects a dataset / filename to be passed: gdalinfo, gdal_translate, GDALOpen() API, etc... Currently /vsicurl/ uses 16 KB as the minimum unit for downloading with HTTP range requests, and a in-memory cache of up to 1000 16KB blocks, with a least recently used eviction strategy. To minimize the total number of HTTP requests outside of the target GeoTIFF file, setting the GDAL_DISABLE_READDIR_ON_OPEN=YES and CPL_VSIL_CURL_ALLOWED_EXTENSIONS=tif environment variables/configuration options is recommended so as to avoid any side-car files (such a .ovr, .aux.xml, .aux, etc.) to be probed. Running gdalinfo or GDALOpen() on such a cloud optimized GeoTIFF will retrieve all the metadata with a single HTTP request of 16 KB. When reading pixels in a tile, only the blocks of 16 KB intersecting the range of the tile will be downloaded. For files hosted on Amazon S3 storage, with non-public sharing rights, [http://www.gdal.org/cpl__vsi_8h.html#a5b4754999acd06444bfda172ff2aaa16 /vsis3/] can be used. == How to check if a GeoTIFF has a cloud optimization internal organization ? == The [https://svn.osgeo.org/gdal/trunk/gdal/swig/python/samples/validate_cloud_optimized_geotiff.py validate_cloud_optimized_geotiff.py] script can be used to check that a (GeoTIFF) file follows the above described file structure {{{ $ python validate_cloud_optimized_geotiff.py test.tif }}} or {{{ $ python import validate_cloud_optimized_geotiff.py validate_cloud_optimized_geotiff.validate('test.tif') }}} == Performance testing == Done with GDAL trunk r37259 with internal libtiff. === Preparation === The source image is the True Color Image of a Sentinel 2A L1C product (10980x10980 pixels, RGB bands of type Byte) Creation of a strip oriented GeoTIFF without overviews ("traditional"): {{{ gdal_translate SENTINEL2_L1C:S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441.SAFE/MTD_MSIL1C.xml:TCI:EPSG_32630 \ S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif \ -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR }}} Creation of a tiled GeoTIFF without overviews: {{{ gdal_translate SENTINEL2_L1C:S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441.SAFE/MTD_MSIL1C.xml:TCI:EPSG_32630 \ S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif \ -co TILED=YES -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR }}} Creation of a tiled GeoTIFF with overviews: {{{ gdal_translate SENTINEL2_L1C:S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441.SAFE/MTD_MSIL1C.xml:TCI:EPSG_32630 \ S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif \ -co TILED=YES -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR gdaladdo -r average S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif 2 4 8 16 32 }}} Creation of a cloud optimized GeoTIFF: {{{ gdal_translate SENTINEL2_L1C:S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441.SAFE/MTD_MSIL1C.xml:TCI:EPSG_32630 \ S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI.tif \ -co TILED=YES -co COMPRESS=DEFLATE gdaladdo -r average S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI.tif 2 4 8 16 32 gdal_translate S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI.tif \ S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_2.tif \ -co TILED=YES -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -co COPY_SRC_OVERVIEWS=YES }}} Creation of a cloud optimized GeoTIFF with tiles of dimension 512x512 {{{ gdal_translate SENTINEL2_L1C:S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441.SAFE/MTD_MSIL1C.xml:TCI:EPSG_32630 \ S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI.tif \ -co TILED=YES -co COMPRESS=DEFLATE gdaladdo -r average S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI.tif 2 4 8 16 32 gdal_translate S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI.tif \ S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif \ -co TILED=YES -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -co COPY_SRC_OVERVIEWS=YES \ -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 --config GDAL_TIFF_OVR_BLOCKSIZE 512 }}} === Reading a single pixel === * Traditional GeoTIFF {{{ $ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdallocationinfo --debug on \ /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif 5000 5000 VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)=2009538 response_code=200 VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif, this=0x1910590) succeeds as GTiff. Report: Location: (5000P,5000L) Band 1: GDAL: GDAL_CACHEMAX = 791 MB VSICURL: Downloading 1523712-1540095 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 Value: 255 Band 2: Value: 255 Band 3: Value: 255 real 0m0.397s user 0m0.060s sys 0m0.024s }}} * Tiled GeoTIFF without overviews: {{{ $ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdallocationinfo --debug on \ /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif 5000 5000 VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)=2123472 response_code=200 VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 16384-49151 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)... VSICURL: Got response_code=206 GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif, this=0x108a4c0) succeeds as GTiff. Report: Location: (5000P,5000L) Band 1: GDAL: GDAL_CACHEMAX = 791 MB VSICURL: Downloading 1556480-1572863 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)... VSICURL: Got response_code=206 Value: 255 Band 2: Value: 255 Band 3: Value: 255 }}} * Regular GeoTIFF with overviews: {{{ $ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdallocationinfo --debug on \ /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif 5000 5000 VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif)=3363607 response_code=200 VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 16384-49151 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif)... VSICURL: Got response_code=206 GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif, this=0x7a84c0) succeeds as GTiff. Report: Location: (5000P,5000L) Band 1: GDAL: GDAL_CACHEMAX = 791 MB VSICURL: Downloading 1556480-1572863 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif)... VSICURL: Got response_code=206 Value: 255 Band 2: Value: 255 Band 3: Value: 255 GDAL: GDALClose(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif, this=0x7a84c0) real 0m0.520s user 0m0.080s sys 0m0.012s }}} * Cloud optimized GeoTIFF: {{{ $ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdallocationinfo--debug on \ /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_2.tif 5000 5000 VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_2.tif)=3355470 response_code=200 VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_2.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 16384-49151 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_2.tif)... VSICURL: Got response_code=206 GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_2.tif, this=0x1c544c0) succeeds as GTiff. Report: Location: (5000P,5000L) Band 1: GDAL: GDAL_CACHEMAX = 791 MB VSICURL: Downloading 2785280-2801663 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_2.tif)... VSICURL: Got response_code=206 Value: 255 Band 2: Value: 255 Band 3: Value: 255 GDAL: GDALClose(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_2.tif, this=0x1c544c0) real 0m0.527s user 0m0.088s sys 0m0.024s }}} * Cloud optimized GeoTIFF with 512x512 tiles {{{ $ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdallocationinfo --debug on \ /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif 5000 5000 VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif)=2918044 response_code=200 VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif)... VSICURL: Got response_code=206 GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif, this=0x15cd450) succeeds as GTiff. Report: Location: (5000P,5000L) Band 1: GDAL: GDAL_CACHEMAX = 791 MB VSICURL: Downloading 2392064-2408447 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif)... VSICURL: Got response_code=206 Value: 255 Band 2: Value: 255 Band 3: Value: 255 GDAL: GDALClose(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif, this=0x15cd450) real 0m0.407s user 0m0.068s sys 0m0.016s }}} The winners are the traditional GeoTIFF organization and the cloud optimized 512x512 tiled GeoTIFF since the directory fits in the first 16 KB. In other cases, no significant time difference (individual runs may differ by a few tens of milliseconds). Same amount of I/O (64 KB read). The fact that we even read 16 + 32 KB in the case without overviews is due to the fact that the first IFD is slightly larger than 16 KB, so we need to read a bit more (and the heuristics double the chunk size as this is contiguous to the previous region read). Note the use of CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif to avoid reading any side car files (.aux.xml, etc...) and GDAL_DISABLE_READDIR_ON_OPEN=YES to avoid any attempt of listing the files in the same directory. Same conclusions if using a AWS S3 hosting, with both GDAL_DISABLE_READDIR_ON_OPEN=YES and CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif defined as well. === Reading a block of pixels at full resolution === * Traditional GeoTIFF {{{ $ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdal_translate --debug on \ /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif \ -srcwin 1024 1024 256 256 out.tif VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)=2009538 response_code=200 VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif, this=0xf1e5a0) succeeds as GTiff. Input file size is 10980, 10980 GDAL: GDALDefaultOverviews::OverviewScan() VSICURL: Downloading 163840-196607 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 196608-262143 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 262144-393215 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 ...10...20...30...40...50...60...70...80...90...100 - done. real 0m0.923s user 0m0.256s sys 0m0.020s }}} The access to the pixel values requires 3 GET requests since whole lines need to be downloaded. A smarter implementation in the GeoTIFF reader with a better interaction with the GDAL HTTP code could probably collapse the 3 GET into a single larger one. * Tiled GeoTIFF without overviews: {{{ $ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdal_translate --debug on \ /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif \ -srcwin 1024 1024 256 256 out.tif VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)=2123472 response_code=200 VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 16384-49151 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)... VSICURL: Got response_code=206 GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif, this=0xce8660) succeeds as GTiff. Input file size is 10980, 10980 GDAL: GDALDefaultOverviews::OverviewScan() ...10...20...30...40...50...60...70...80...90...100 - done. real 0m0.527s user 0m0.084s sys 0m0.020s }}} * Regular GeoTIFF with overviews: {{{ $ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdal_translate --debug on \ /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif \ -srcwin 1024 1024 256 256 out.tif VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif)=3363607 response_code=200 VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 16384-49151 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif)... VSICURL: Got response_code=206 GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif, this=0xc6d620) succeeds as GTiff. Input file size is 10980, 10980 GTiff: ScanDirectories() VSICURL: Downloading 2113536-2129919 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 2129920-2162687 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif)... VSICURL: Got response_code=206 GTiff: Opened 5490x5490 overview. GTiff: Opened 2745x2745 overview. GTiff: Opened 1373x1373 overview. GTiff: Opened 687x687 overview. GTiff: Opened 344x344 overview. GDAL: GDALDefaultOverviews::OverviewScan() VSICURL: Downloading 196608-212991 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif)... VSICURL: Got response_code=206 ...10...20...30...40...50...60...70...80...90...100 - done. real 0m0.757s user 0m0.100s sys 0m0.032s }}} One can see that a directory scan is done (GTiff: ScanDirectories() trace), despite a few optimizations done in r37258 and r37259. This is due to gdal_translate trying to copy mask bands, which requires scaning directories to find a potential internal mask band. This scan can be avoided by adding the -mask none switch. {{{ $ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdal_translate --debug on \ /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif -srcwin 1024 1024 256 256 -mask none out.tif VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif)=3363607 response_code=200 VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 16384-49151 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif)... VSICURL: Got response_code=206 GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif, this=0x1078620) succeeds as GTiff. Input file size is 10980, 10980 GDAL: GDALDefaultOverviews::OverviewScan() GDAL: GDALDatasetCopyWholeRaster(): 256*256 swaths, bInterleave=1 VSICURL: Downloading 196608-212991 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif)... VSICURL: Got response_code=206 ...10...20...30...40...50...60...70...80...90...100 - done. real 0m0.518s user 0m0.092s sys 0m0.012s }}} Best timing on a AWS S3 bucket (us-east-1 region, accessed from France): ~ 1.9s This is also the performance one gets with : {{{ $ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif \ python -c 'from osgeo import gdal; ds = gdal.Open("/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif"); ds.ReadAsArray(1024,1024,256,256)' }}} * Cloud optimized GeoTIFF: {{{ $ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdal_translate --debug on \ /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_2.tif \ -srcwin 1024 1024 256 256 out.tif VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_2.tif)=3355470 response_code=200 VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_2.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 16384-49151 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_2.tif)... VSICURL: Got response_code=206 GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_2.tif, this=0xdeb620) succeeds as GTiff. Input file size is 10980, 10980 GTiff: ScanDirectories() GTiff: Opened 5490x5490 overview. GTiff: Opened 2745x2745 overview. GTiff: Opened 1373x1373 overview. GTiff: Opened 687x687 overview. GTiff: Opened 344x344 overview. VSICURL: Downloading 1425408-1441791 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_2.tif)... VSICURL: Got response_code=206 ...10...20...30...40...50...60...70...80...90...100 - done. real 0m0.519s user 0m0.096s sys 0m0.008s }}} Best timing on a AWS S3 bucket: ~ 1.9s No need to specify -mask none to get the maximum performance: as the IFD are at the beginning of the files, they have been fetched with the 2 first HTTP GET requests. * Cloud optimized GeoTIFF with tiles 512x512: {{{ $ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdal_translate --debug on \ /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif \ -srcwin 1024 1024 256 256 out.tif VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif)=2918044 response_code=200 VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif)... VSICURL: Got response_code=206 GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif, this=0x1c815c0) succeeds as GTiff. Input file size is 10980, 10980 GTiff: ScanDirectories() GTiff: Opened 5490x5490 overview. GTiff: Opened 2745x2745 overview. GTiff: Opened 1373x1373 overview. GTiff: Opened 687x687 overview. GTiff: Opened 344x344 overview. GDAL: GDALDefaultOverviews::OverviewScan() VSICURL: Downloading 1114112-1146879 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif)... VSICURL: Got response_code=206 ...10...20...30...40...50...60...70...80...90...100 - done. real 0m0.441s user 0m0.096s sys 0m0.016s }}} Compared to the default cloud optimized GeoTIFF, we save a few bytes for the reading of the IFD since all IFDs fit into the first 16 KB === Getting a subsampled version of the image === * Traditional GeoTIFF {{{ $ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdal_translate --debug on \ /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif \ out.tif -outsize 1% 1% VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)=2009538 response_code=200 VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif, this=0x2327590) succeeds as GTiff. Input file size is 10980, 10980 GDAL: GDALDefaultOverviews::OverviewScan() GDAL: GDALDefaultOverviews::OverviewScan() VSICURL: Downloading 16384-49151 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 49152-114687 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 114688-245759 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 245760-507903 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 507904-1032191 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 1064960-1081343 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 1097728-1114111 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 1130496-1146879 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 1163264-1179647 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 1196032-1212415 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 1212416-1245183 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 1245184-1310719 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 1310720-1441791 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 1441792-1703935 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 1703936-2009537 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr_strip.tif)... VSICURL: Got response_code=206 ...10...20...30...40...50...60...70...80...90...100 - done. real 0m3.197s user 0m0.304s sys 0m0.056s }}} As one could anticipate, the whole file needs to be read. * Tiled GeoTIFF without overviews: {{{ $ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdal_translate --debug on \ /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif out.tif -outsize 1% 1% VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)=2123472 response_code=200 VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 16384-49151 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)... VSICURL: Got response_code=206 GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif, this=0x7c3650) succeeds as GTiff. Input file size is 10980, 10980 GDAL: GDALDefaultOverviews::OverviewScan() GDAL: GDAL_CACHEMAX = 791 MB GDAL: GDALDatasetCopyWholeRaster(): 109*109 swaths, bInterleave=1 GDAL: GDALDefaultOverviews::OverviewScan() GDAL: GDALDefaultOverviews::OverviewScan() GDAL: Potential thrashing on band 1 of /vsimem/sparse_0x7d0e10. VSICURL: Downloading 49152-114687 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 114688-245759 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 245760-507903 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 507904-1032191 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 1032192-2080767 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 2080768-2123471 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_no_ovr.tif)... VSICURL: Got response_code=206 ...10...20...30...40...50...60...70...80...90...100 - done. GDAL: GDALClose(out.tif, this=0x83bdc0) GDAL: 59168 block reads on 32 block band 1 of /vsimem/sparse_0x7d0e10. real 0m2.960s user 0m0.740s sys 0m0.096s }}} As one could anticipate, the whole file needs to be read. * Regular GeoTIFF with overviews: {{{ $ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdal_translate --debug on \ /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif \ out.tif -outsize 1% 1% VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif)=3363607 response_code=200 VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 16384-49151 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif)... VSICURL: Got response_code=206 GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif, this=0x23d7610) succeeds as GTiff. Input file size is 10980, 10980 GTiff: ScanDirectories() VSICURL: Downloading 2113536-2129919 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 2129920-2162687 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif)... VSICURL: Got response_code=206 GTiff: Opened 5490x5490 overview. GTiff: Opened 2745x2745 overview. GTiff: Opened 1373x1373 overview. GTiff: Opened 687x687 overview. GTiff: Opened 344x344 overview. VSICURL: Downloading 3342336-3358719 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 3358720-3363606 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_regular_with_ovr_2.tif)... VSICURL: Got response_code=206 ...10...20...30...40...50...60...70...80...90...100 - done. real 0m0.810s user 0m0.108s sys 0m0.020s }}} Best timing on a AWS S3 bucket: ~ 2.5s A full scan of the IFD is necessary to find the appropriate overview level. * Cloud optimized GeoTIFF: {{{ $ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdal_translate --debug on \ /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_2.tif \ out.tif -outsize 1% 1% VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_2.tif)=3355470 response_code=200 VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_2.tif)... VSICURL: Got response_code=206 VSICURL: Downloading 16384-49151 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_2.tif)... VSICURL: Got response_code=206 GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_2.tif, this=0x1847610) succeeds as GTiff. Input file size is 10980, 10980 GTiff: ScanDirectories() GTiff: Opened 5490x5490 overview. GTiff: Opened 2745x2745 overview. GTiff: Opened 1373x1373 overview. GTiff: Opened 687x687 overview. GTiff: Opened 344x344 overview. GDAL: GDALDatasetCopyWholeRaster(): 109*109 swaths, bInterleave=1 ...10...20...30...40...50...60...70...80...90...100 - done. real 0m0.435s user 0m0.088s sys 0m0.028s }}} Best timing on a AWS S3 bucket: ~ 1.5s As the IFD are at the beginning of the files, as well as the pixel data for the smallest overview, the request can be completed with the 2 first HTTP GET requests (this is a bit of an extreme case of course) * Cloud optimized GeoTIFF with tiles of 512x512: {{{ $ time GDAL_DISABLE_READDIR_ON_OPEN=YES CPL_VSIL_CURL_ALLOWED_EXTENSIONS=.tif gdal_translate --debug on \ /vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif \ out.tif -outsize 1% 1% VSICURL: GetFileSize(http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif)=2918044 response_code=200 VSICURL: Downloading 0-16383 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif)... VSICURL: Got response_code=206 GDAL: GDALOpen(/vsicurl/http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif, this=0x152b5b0) succeeds as GTiff. Input file size is 10980, 10980 GTiff: ScanDirectories() GTiff: Opened 5490x5490 overview. GTiff: Opened 2745x2745 overview. GTiff: Opened 1373x1373 overview. GTiff: Opened 687x687 overview. GTiff: Opened 344x344 overview. VSICURL: Downloading 16384-49151 (http://even.rouault.free.fr/gtiff_test/S2A_MSIL1C_20170102T111442_N0204_R137_T30TXT_20170102T111441_TCI_cloudoptimized_512.tif)... VSICURL: Got response_code=206 ...10...20...30...40...50...60...70...80...90...100 - done. real 0m0.439s user 0m0.088s sys 0m0.024s }}} Same performance as the default cloud optimized case (GDAL could potentially read in a less gready way for the pixel data since the tile size is only 8KB here) === Conclusions === * GDAL can make an efficient use of overviews, even on files hosted on a HTTP storage * The cloud compatible file organization can reduce a bit the amount of GET requests required. * Increasing the block size can reduce the size of the IFD. But larger blocks can cause more bytes to be pulled for random access if the compression rate is not high.