Opened 13 years ago
Closed 8 years ago
#1429 closed enhancement (fixed)
r.watershed large grids
Reported by: | dnewcomb | Owned by: | |
---|---|---|---|
Priority: | normal | Milestone: | 7.0.0 |
Component: | Raster | Version: | svn-trunk |
Keywords: | r.watershed, Large grids | Cc: | |
CPU: | x86-64 | Platform: | Linux |
Description (last modified by )
tried r.watershed on a large elevation grid (51000 rows, 133000 columns, 6783000000 cells) with the following options:
r.watershed -m elevation=nc_20ft_ncfpm@statewide threshold=500000 accumulation=nc_flow_accum_wsh@statewide drainage=nc_direct_wsh@statewide basin=nc_wshed_label_wsh stream=nc_stream_seg_wsh half_basin=nc_half_basin_wsh memory=8000
and got the following output:
SECTION 1 beginning: Initiating Variables. 5 sections total. SECTION 1a: Mark masked and NULL cells segment_format(fd,1,-803458658,1,4096,8): illegal value(s) seg_open(): illegal configuration parameter(s) segment_format(fd,1,-803458657,1,262144,16): illegal value(s) seg_open(): illegal configuration parameter(s) SECTION 1b: Determining Offmap Flow. Subprocess failed with exit code 11 category information for [nc_flow_accum_wsh@statewide] in [statewide] missing or invalid category information for [nc_direct_wsh@statewide] in [statewide] missing or invalid category information for [nc_wshed_label_wsh] in [statewide] missing or invalid category information for [nc_stream_seg_wsh] in [statewide] missing or invalid category information for [nc_half_basin_wsh] in [statewide] missing or invalid (Sat Aug 27 11:38:42 2011) Command finished (45054 sec) GRASS 7.0 svn pull from 8/17/2011
looking at raster/r.watershed/seg/cseg.h
Change History (4)
comment:1 by , 13 years ago
comment:2 by , 11 years ago
Component: | Default → Raster |
---|---|
Keywords: | r.watershed added |
I tried with the current GRASS 7.svn (r59279, approximately 7.0.0beta1) and r.watershed worked smoothly with 9 billion pixels (77 hours, way more pixels than in the original report):
GRASS 7.0.svn (nc_spm_08_grass7):~ > g.region rast=elevation -p res=0.15 projection: 99 (Lambert Conformal Conic) zone: 0 datum: nad83 ellipsoid: a=6378137 es=0.006694380022900787 north: 228500 south: 215000 west: 630000 east: 645000 nsres: 0.15 ewres: 0.15 rows: 90000 cols: 100000 cells: 9000000000 GRASS 7.0.svn (nc_spm_08_grass7):~ > time -p r.watershed elevation=elevation \ threshold=500000 accumulation=nc_flow_accum_wsh drainage=nc_direct_wsh \ basin=nc_wshed_label_wsh stream=nc_stream_seg_wsh \ half_basin=nc_half_basin_wsh memory=42000 -m SECTION 1 beginning: Initiating Variables. 5 sections total. SECTION 1a: Mark masked and NULL cells 100% SECTION 1b: Determining Offmap Flow. 100% SECTION 2: A* Search. 100% SECTION 3a: Accumulating Surface Flow with MFD. 100% SECTION 3b: Adjusting drainage directions. SECTION 4: Watershed determination. 100% SECTION 5: Closing Maps. Closing stream segments map 100% Closing basins map 100% Closing half basins map 100% Closing accumulation map 100% Closing flow direction map real 277828.64 user 59672.08 sys 4481.16 --> 77.2 hours for rows: 90000 cols: 100000 cells: 9000000000
Peak memory usage was 10GB (checked with munin).
Special kudos to Markus Metz for making this possible.
(I suppose the ticket can be closed at this point)
comment:3 by , 8 years ago
Milestone: | 7.0.0 → 7.0.5 |
---|
comment:4 by , 8 years ago
Description: | modified (diff) |
---|---|
Milestone: | 7.0.5 → 7.0.0 |
Resolution: | → fixed |
Status: | new → closed |
Closing as suggested, it was already working 2 years ago as per upper comment.
Replying to dnewcomb:
For larger grids, GRASS must be compiled with LFS: configure --enable-largefile, and quite a bit of free disk space must be available where the current location is living.
Apart from that, 6783000000 cells are currently too much, see recent similar post in the user ml [0]. The current limit is 231 - 1 = 2,147,483,647 cells. This can be overcome in trunk, but requires quite a few modifications both in the segment library and in r.watershed.
Markus M
[0] http://lists.osgeo.org/pipermail/grass-user/2011-August/061724.html