Opened 14 years ago

Closed 9 years ago

#3165 closed enhancement (wontfix)

Performance issues with raster .vrt and .sid

Reported by: craigleat Owned by: warmerdam
Priority: normal Milestone: 1.8.1
Component: default Version: 1.6.0
Severity: normal Keywords: .sid .vrt
Cc:

Description

I am experiencing severe performance issues when accessing .sid files via a .vrt file. In QGIS on Ubuntu accessing .sid files directly is fast, but accessing them via a .vrt is too slow. I can reproduce the behaviour using gdalinfo on Windows. My test involves 9 .sid files and 24 .tif files with identical base resolution.

gdalinfo --version

GDAL 1.6.0dev, FWTools 2.2.8, released 2008/10/29

timeit.exe gdalinfo -mm -nomd 2730BC.vrt

Driver: VRT/Virtual Raster Files: 2730BC.vrt Size is 28541, 39741 Coordinate System is: LOCAL_CS["WG31",

UNIT["meters",1]]

Origin = (-49487.200000000004000,-3029248.250000000000000) Pixel Size = (0.350000000000000,-0.350000000000000) Corner Coordinates: Upper Left ( -49487.200,-3029248.250) Lower Left ( -49487.200,-3043157.600) Upper Right ( -39497.850,-3029248.250) Lower Right ( -39497.850,-3043157.600) Center ( -44492.525,-3036202.925) Band 1 Block=128x128 Type=Byte, ColorInterp=Red

Computed Min/Max=0.000,255.000

Band 2 Block=128x128 Type=Byte, ColorInterp=Green

Computed Min/Max=0.000,255.000

Band 3 Block=128x128 Type=Byte, ColorInterp=Blue

Computed Min/Max=0.000,255.000

Version Number: Windows NT 5.1 (Build 2600) Exit Time: 6:47 pm, Friday, October 2 2009 Elapsed Time: 2:47:10.140 Process Time: 2:45:30.171 System Calls: 258594074 Context Switches: 16703731 Page Faults: 103810694 Bytes Read: 600989608 Bytes Written: 75379170 Bytes Other: 133707797

Next I process the same .sid files directly using a script (test.bat) containing: @ECHO OFF FOR %%f IN (*.sid) DO gdalinfo -mm -nomd %%f

timeit test.bat

Driver: MrSID/Multi-resolution Seamless Image Database (MrSID) Files: 2730BC_21A.sid

2730BC_21A.sid.aux.xml

Size is 7240, 8080 Coordinate System is: LOCAL_CS["WG31",

UNIT["meters",1]]

Origin = (-49454.650000000001000,-3037559.349999999600000) Pixel Size = (0.350000000000000,-0.350000000000000) Corner Coordinates: Upper Left ( -49454.650,-3037559.350) Lower Left ( -49454.650,-3040387.350) Upper Right ( -46920.650,-3037559.350) Lower Right ( -46920.650,-3040387.350) Center ( -48187.650,-3038973.350) Band 1 Block=1024x128 Type=Byte, ColorInterp=Red

Computed Min/Max=0.000,212.000

Minimum=5.000, Maximum=185.000, Mean=104.478, StdDev=21.099 Overviews: 3620x4040, 1810x2020, 905x1010, 453x505, 227x253, 114x127, 57x64

Band 2 Block=1024x128 Type=Byte, ColorInterp=Green

Computed Min/Max=0.000,220.000

Minimum=24.000, Maximum=189.000, Mean=112.464, StdDev=18.596 Overviews: 3620x4040, 1810x2020, 905x1010, 453x505, 227x253, 114x127, 57x64

Band 3 Block=1024x128 Type=Byte, ColorInterp=Blue

Computed Min/Max=0.000,211.000

Minimum=29.000, Maximum=169.000, Mean=97.389, StdDev=13.177 Overviews: 3620x4040, 1810x2020, 905x1010, 453x505, 227x253, 114x127, 57x64

<snip> </snip>

Version Number: Windows NT 5.1 (Build 2600) Exit Time: 9:06 am, Monday, October 5 2009 Elapsed Time: 0:56:58.546 Process Time: 0:00:00.015 System Calls: 74072437 Context Switches: 5050402 Page Faults: 23826615 Bytes Read: 3310547095 Bytes Written: 112085122 Bytes Other: 62805368

Next I process a directory of 24 GeoTiff files with the same base resolution and use a .vrt file:

timeit gdalinfo -mm -nomd 2729BD_TIF.vrt

Driver: VRT/Virtual Raster Files: 2729BD_TIF.vrt Size is 29597, 47838 Coordinate System is: PROJCS["WG31",

GEOGCS["GCS_Hartebeesthoek_1994",

DATUM["Hartebeesthoek94",

SPHEROID["WGS_1984",6378137.0,298.257223563]],

PRIMEM["Greenwich",0.0], UNIT["Degree",0.0174532925199433]],

PROJECTIONTransverse_Mercator, PARAMETER["False_Easting",0.0], PARAMETER["False_Northing",0.0], PARAMETER["Central_Meridian",31.0], PARAMETER["Scale_Factor",1.0], PARAMETER["Latitude_Of_Origin",0.0], UNIT["Meter",1.0]]

Origin = (-109153.616167264410000,-3026790.550000000300000) Pixel Size = (0.350000000000000,-0.350000000000000) Corner Coordinates: Upper Left ( -109153.616,-3026790.550) ( 29d53'48.78"E, 27d20'56.60"S) Lower Left ( -109153.616,-3043533.850) ( 29d53'43.37"E, 27d30'0.45"S) Upper Right ( -98794.666,-3026790.550) ( 30d 0'5.61"E, 27d20'59.43"S) Lower Right ( -98794.666,-3043533.850) ( 30d 0'0.71"E, 27d30'3.31"S) Center ( -103974.141,-3035162.200) ( 29d56'54.62"E, 27d25'29.98"S) Band 1 Block=128x128 Type=Byte, ColorInterp=Red

Computed Min/Max=0.000,255.000

Band 2 Block=128x128 Type=Byte, ColorInterp=Green

Computed Min/Max=0.000,255.000

Band 3 Block=128x128 Type=Byte, ColorInterp=Blue

Computed Min/Max=0.000,255.000

Version Number: Windows NT 5.1 (Build 2600) Exit Time: 10:21 am, Monday, October 5 2009 Elapsed Time: 0:26:32.765 Process Time: 0:08:51.937 System Calls: 406714937 Context Switches: 8876715 Page Faults: 4870522 Bytes Read: 4091589454 Bytes Written: 15469725 Bytes Other: 21520441

Next I process the same .tif files directly using the script (test2.bat): @ECHO OFF FOR %%f IN (*.tif) DO gdalinfo -mm -nomd %%f

timeit test2.bat

<snip> </snip> Driver: GTiff/GeoTIFF Files: 2729BD_14C.TIF

2729BD_14C.TIF.ovr

Size is 8142, 8017 Coordinate System is: PROJCS["WG31",

GEOGCS["GCS_Hartebeesthoek_1994",

DATUM["Hartebeesthoek94",

SPHEROID["WGS_1984",6378137.0,298.257223563]],

PRIMEM["Greenwich",0.0], UNIT["Degree",0.0174532925199433]],

PROJECTIONTransverse_Mercator, PARAMETER["False_Easting",0.0], PARAMETER["False_Northing",0.0], PARAMETER["Central_Meridian",31.0], PARAMETER["Scale_Factor",1.0], PARAMETER["Latitude_Of_Origin",0.0], UNIT["Meter",1.0]]

Origin = (-109153.616167264410000,-3029645.394898317800000) Pixel Size = (0.350000000000000,-0.350000000000023) Corner Coordinates: Upper Left ( -109153.616,-3029645.395) ( 29d53'47.86"E, 27d22'29.33"S) Lower Left ( -109153.616,-3032451.345) ( 29d53'46.95"E, 27d24'0.47"S) Upper Right ( -106303.916,-3029645.395) ( 29d55'31.55"E, 27d22'30.14"S) Lower Right ( -106303.916,-3032451.345) ( 29d55'30.66"E, 27d24'1.28"S) Center ( -107728.766,-3031048.370) ( 29d54'39.26"E, 27d23'15.31"S) Band 1 Block=8142x1 Type=Byte, ColorInterp=Red

Computed Min/Max=0.000,255.000

Overviews: 4071x4009, 2036x2005, 1018x1003, 509x502, 255x251, 128x126, 64x63

Band 2 Block=8142x1 Type=Byte, ColorInterp=Green

Computed Min/Max=0.000,255.000

Overviews: 4071x4009, 2036x2005, 1018x1003, 509x502, 255x251, 128x126, 64x63

Band 3 Block=8142x1 Type=Byte, ColorInterp=Blue

Computed Min/Max=0.000,255.000

Overviews: 4071x4009, 2036x2005, 1018x1003, 509x502, 255x251, 128x126, 64x63

Version Number: Windows NT 5.1 (Build 2600) Exit Time: 9:49 am, Monday, October 5 2009 Elapsed Time: 0:09:25.750 Process Time: 0:00:00.171 System Calls: 108149830 Context Switches: 3118430 Page Faults: 3935461 Bytes Read: 4172543406 Bytes Written: 2386658 Bytes Other: 15278822

To summarize: Processing 9 .sid files via a .vrt takes 2h:47m Processing 9 .sid files directly takes 0h:57m Processing 24 .tif files via a .vrt takes 0h:27m Processing 24 .tif files directly takes 0h:9m Working via a .vrt is approximately three times slower than working with the tiles directly and working with .sid files is much slower than .tif files. I'm marking this as an enhancement, as gdal works with .vrt and .sid albeit VERY slowly.

Change History (1)

comment:1 by Jukka Rahkonen, 9 years ago

Resolution: wontfix
Status: newclosed

Testing what is the behaviour with current GDAL and MrSID driver versions is hard because no test data were ever attached to the ticket. Closing as wontfix.

Note: See TracTickets for help on using tickets.