Opened 14 years ago
Closed 9 years ago
#3165 closed enhancement (wontfix)
Performance issues with raster .vrt and .sid
Reported by: | craigleat | Owned by: | warmerdam |
---|---|---|---|
Priority: | normal | Milestone: | 1.8.1 |
Component: | default | Version: | 1.6.0 |
Severity: | normal | Keywords: | .sid .vrt |
Cc: |
Description
I am experiencing severe performance issues when accessing .sid files via a .vrt file. In QGIS on Ubuntu accessing .sid files directly is fast, but accessing them via a .vrt is too slow. I can reproduce the behaviour using gdalinfo on Windows. My test involves 9 .sid files and 24 .tif files with identical base resolution.
gdalinfo --version
GDAL 1.6.0dev, FWTools 2.2.8, released 2008/10/29
timeit.exe gdalinfo -mm -nomd 2730BC.vrt
Driver: VRT/Virtual Raster Files: 2730BC.vrt Size is 28541, 39741 Coordinate System is: LOCAL_CS["WG31",
UNIT["meters",1]]
Origin = (-49487.200000000004000,-3029248.250000000000000) Pixel Size = (0.350000000000000,-0.350000000000000) Corner Coordinates: Upper Left ( -49487.200,-3029248.250) Lower Left ( -49487.200,-3043157.600) Upper Right ( -39497.850,-3029248.250) Lower Right ( -39497.850,-3043157.600) Center ( -44492.525,-3036202.925) Band 1 Block=128x128 Type=Byte, ColorInterp=Red
Computed Min/Max=0.000,255.000
Band 2 Block=128x128 Type=Byte, ColorInterp=Green
Computed Min/Max=0.000,255.000
Band 3 Block=128x128 Type=Byte, ColorInterp=Blue
Computed Min/Max=0.000,255.000
Version Number: Windows NT 5.1 (Build 2600) Exit Time: 6:47 pm, Friday, October 2 2009 Elapsed Time: 2:47:10.140 Process Time: 2:45:30.171 System Calls: 258594074 Context Switches: 16703731 Page Faults: 103810694 Bytes Read: 600989608 Bytes Written: 75379170 Bytes Other: 133707797
Next I process the same .sid files directly using a script (test.bat) containing: @ECHO OFF FOR %%f IN (*.sid) DO gdalinfo -mm -nomd %%f
timeit test.bat
Driver: MrSID/Multi-resolution Seamless Image Database (MrSID) Files: 2730BC_21A.sid
2730BC_21A.sid.aux.xml
Size is 7240, 8080 Coordinate System is: LOCAL_CS["WG31",
UNIT["meters",1]]
Origin = (-49454.650000000001000,-3037559.349999999600000) Pixel Size = (0.350000000000000,-0.350000000000000) Corner Coordinates: Upper Left ( -49454.650,-3037559.350) Lower Left ( -49454.650,-3040387.350) Upper Right ( -46920.650,-3037559.350) Lower Right ( -46920.650,-3040387.350) Center ( -48187.650,-3038973.350) Band 1 Block=1024x128 Type=Byte, ColorInterp=Red
Computed Min/Max=0.000,212.000
Minimum=5.000, Maximum=185.000, Mean=104.478, StdDev=21.099 Overviews: 3620x4040, 1810x2020, 905x1010, 453x505, 227x253, 114x127, 57x64
Band 2 Block=1024x128 Type=Byte, ColorInterp=Green
Computed Min/Max=0.000,220.000
Minimum=24.000, Maximum=189.000, Mean=112.464, StdDev=18.596 Overviews: 3620x4040, 1810x2020, 905x1010, 453x505, 227x253, 114x127, 57x64
Band 3 Block=1024x128 Type=Byte, ColorInterp=Blue
Computed Min/Max=0.000,211.000
Minimum=29.000, Maximum=169.000, Mean=97.389, StdDev=13.177 Overviews: 3620x4040, 1810x2020, 905x1010, 453x505, 227x253, 114x127, 57x64
<snip> </snip>
Version Number: Windows NT 5.1 (Build 2600) Exit Time: 9:06 am, Monday, October 5 2009 Elapsed Time: 0:56:58.546 Process Time: 0:00:00.015 System Calls: 74072437 Context Switches: 5050402 Page Faults: 23826615 Bytes Read: 3310547095 Bytes Written: 112085122 Bytes Other: 62805368
Next I process a directory of 24 GeoTiff files with the same base resolution and use a .vrt file:
timeit gdalinfo -mm -nomd 2729BD_TIF.vrt
Driver: VRT/Virtual Raster Files: 2729BD_TIF.vrt Size is 29597, 47838 Coordinate System is: PROJCS["WG31",
GEOGCS["GCS_Hartebeesthoek_1994",
DATUM["Hartebeesthoek94",
SPHEROID["WGS_1984",6378137.0,298.257223563]],
PRIMEM["Greenwich",0.0], UNIT["Degree",0.0174532925199433]],
PROJECTIONTransverse_Mercator, PARAMETER["False_Easting",0.0], PARAMETER["False_Northing",0.0], PARAMETER["Central_Meridian",31.0], PARAMETER["Scale_Factor",1.0], PARAMETER["Latitude_Of_Origin",0.0], UNIT["Meter",1.0]]
Origin = (-109153.616167264410000,-3026790.550000000300000) Pixel Size = (0.350000000000000,-0.350000000000000) Corner Coordinates: Upper Left ( -109153.616,-3026790.550) ( 29d53'48.78"E, 27d20'56.60"S) Lower Left ( -109153.616,-3043533.850) ( 29d53'43.37"E, 27d30'0.45"S) Upper Right ( -98794.666,-3026790.550) ( 30d 0'5.61"E, 27d20'59.43"S) Lower Right ( -98794.666,-3043533.850) ( 30d 0'0.71"E, 27d30'3.31"S) Center ( -103974.141,-3035162.200) ( 29d56'54.62"E, 27d25'29.98"S) Band 1 Block=128x128 Type=Byte, ColorInterp=Red
Computed Min/Max=0.000,255.000
Band 2 Block=128x128 Type=Byte, ColorInterp=Green
Computed Min/Max=0.000,255.000
Band 3 Block=128x128 Type=Byte, ColorInterp=Blue
Computed Min/Max=0.000,255.000
Version Number: Windows NT 5.1 (Build 2600) Exit Time: 10:21 am, Monday, October 5 2009 Elapsed Time: 0:26:32.765 Process Time: 0:08:51.937 System Calls: 406714937 Context Switches: 8876715 Page Faults: 4870522 Bytes Read: 4091589454 Bytes Written: 15469725 Bytes Other: 21520441
Next I process the same .tif files directly using the script (test2.bat): @ECHO OFF FOR %%f IN (*.tif) DO gdalinfo -mm -nomd %%f
timeit test2.bat
<snip> </snip> Driver: GTiff/GeoTIFF Files: 2729BD_14C.TIF
2729BD_14C.TIF.ovr
Size is 8142, 8017 Coordinate System is: PROJCS["WG31",
GEOGCS["GCS_Hartebeesthoek_1994",
DATUM["Hartebeesthoek94",
SPHEROID["WGS_1984",6378137.0,298.257223563]],
PRIMEM["Greenwich",0.0], UNIT["Degree",0.0174532925199433]],
PROJECTIONTransverse_Mercator, PARAMETER["False_Easting",0.0], PARAMETER["False_Northing",0.0], PARAMETER["Central_Meridian",31.0], PARAMETER["Scale_Factor",1.0], PARAMETER["Latitude_Of_Origin",0.0], UNIT["Meter",1.0]]
Origin = (-109153.616167264410000,-3029645.394898317800000) Pixel Size = (0.350000000000000,-0.350000000000023) Corner Coordinates: Upper Left ( -109153.616,-3029645.395) ( 29d53'47.86"E, 27d22'29.33"S) Lower Left ( -109153.616,-3032451.345) ( 29d53'46.95"E, 27d24'0.47"S) Upper Right ( -106303.916,-3029645.395) ( 29d55'31.55"E, 27d22'30.14"S) Lower Right ( -106303.916,-3032451.345) ( 29d55'30.66"E, 27d24'1.28"S) Center ( -107728.766,-3031048.370) ( 29d54'39.26"E, 27d23'15.31"S) Band 1 Block=8142x1 Type=Byte, ColorInterp=Red
Computed Min/Max=0.000,255.000
Overviews: 4071x4009, 2036x2005, 1018x1003, 509x502, 255x251, 128x126, 64x63
Band 2 Block=8142x1 Type=Byte, ColorInterp=Green
Computed Min/Max=0.000,255.000
Overviews: 4071x4009, 2036x2005, 1018x1003, 509x502, 255x251, 128x126, 64x63
Band 3 Block=8142x1 Type=Byte, ColorInterp=Blue
Computed Min/Max=0.000,255.000
Overviews: 4071x4009, 2036x2005, 1018x1003, 509x502, 255x251, 128x126, 64x63
Version Number: Windows NT 5.1 (Build 2600) Exit Time: 9:49 am, Monday, October 5 2009 Elapsed Time: 0:09:25.750 Process Time: 0:00:00.171 System Calls: 108149830 Context Switches: 3118430 Page Faults: 3935461 Bytes Read: 4172543406 Bytes Written: 2386658 Bytes Other: 15278822
To summarize: Processing 9 .sid files via a .vrt takes 2h:47m Processing 9 .sid files directly takes 0h:57m Processing 24 .tif files via a .vrt takes 0h:27m Processing 24 .tif files directly takes 0h:9m Working via a .vrt is approximately three times slower than working with the tiles directly and working with .sid files is much slower than .tif files. I'm marking this as an enhancement, as gdal works with .vrt and .sid albeit VERY slowly.
Testing what is the behaviour with current GDAL and MrSID driver versions is hard because no test data were ever attached to the ticket. Closing as wontfix.