Opened 5 years ago

Last modified 7 months ago

#2356 new defect

r.stream.basisns uses more memory than given in memory option

Reported by: sbl Owned by: grass-dev@…
Priority: normal Milestone:
Component: Addons Version: svn-trunk
Keywords: r.stream.basins, memory Cc:
CPU: x86-64 Platform: Linux

Description

I tried to run r.stream.basins on using large flow direction / stream maps generated with r.watershed.

With m-flag and memory option set to 35GB (~2/3 of the RAM of my system) the process got killed (by out-of-mmemory-killer) after input maps were read.

So, I re-ran the module with a smaller amount of memory allocated (20 and 10 GB) in the memory option and monitored memory usage (UNIX top command). There I found that r.stream.basisns used 2x the amount of RAM allowed, on each input map. In other words, with memory=10000, it used

20GB for reading flow direction map, plus

20GB for reading streams map (40GB in total).

Finally, the process gets killed when it starts finding nodes.

I am using Ubuntu 12.04 LTS 64bit and GRASS 7.0 (revision 58620).

Change History (17)

comment:1 Changed 5 years ago by hamish

Component: RasterAddons
Keywords: r.stream.basins added; r.stream.basisns removed

were the input maps CELL, FCELL or DCELL?

fwiw here's the code,

 opt_swapsize->description = _("Maximum memory used in memory swap mode (MB)");
 opt_swapsize->answer = "300";
...
 number_of_segs = (int)atof(opt_swapsize->answer);
 number_of_segs = number_of_segs < 32 ? (int)(32 / 0.12) : number_of_segs / 0.12;

if the stream_rast= option is used then two ram_create_map()s happen. Likely the above number_of_segs isn't taking that into account.

did it work ok when you halved the memory= option?

Hamish

comment:2 Changed 5 years ago by hamish

I wrote:

if the stream_rast= option is used then two ram_create_map()s happen.

sorry, two seg_create_map() happen without stream_rast=, three if with it.

comment:3 Changed 5 years ago by sbl

Both input maps (dirs and streams) were generated by r.watershed and are CELL.

r.stream.basins worked just fine when I set the memory option to ~1/7 of the available RAM.

comment:4 Changed 5 years ago by hamish

how many rows and columns in the raster array?

comment:5 Changed 5 years ago by sbl

Both input maps had: Rows: 180752 Columns: 141312 The region followed exactly those maps (it was not changed after maps were generated with r.watershed).

comment:6 Changed 5 years ago by hamish

so for an unsegmented cell map, 180752 * 141312 * 4 bytes = ~97GB to hold one whole map in ram.

r.stream.basins/io.h defines segments as 256x256, so for memory=10000 (mb)

  seg_create_map(&map_dirs, SROWS, SCOLS, number_of_segs, CELL_TYPE);

we get

bytes = ~20.8GB, and needs that for each of the two or three open maps.

can you try r61149?

Hamish

comment:7 Changed 5 years ago by sbl

FYI, 20.8GB is exactly the amount of memory consumption for reading the input maps reported by the top command, when I ran r.stream.basins with memory=10000.

comment:8 in reply to:  7 Changed 5 years ago by hamish

Replying to sbl:

FYI, 20.8GB is exactly the amount of memory consumption for reading the input maps reported by the top command, when I ran r.stream.basins with memory=10000.

ok, then the math works. :-)

can someone familiar with this module give a 3rd party confirmation that the memory= option is now properly respected?

thanks, Hamish

comment:9 Changed 5 years ago by neteler

TODO for all r.stream.* modules (reported by mmetz in grass-dev):

Most importantly it is the NULL handling that needs to be fixed, and synced between the ram and disk modes. More generally, the ram and disk modes need to be synced and validated. For ease of maintenance, the ram mode could be removed.

comment:10 Changed 4 years ago by neteler

Milestone: 7.1.07.2.0

Milestone renamed

comment:11 Changed 3 years ago by neteler

Milestone: 7.2.07.2.1

Ticket retargeted after milestone closed

comment:12 Changed 3 years ago by martinl

Milestone: 7.2.17.2.2

comment:13 Changed 2 years ago by neteler

Milestone: 7.2.27.2.3

Ticket retargeted after milestone closed

comment:14 Changed 2 years ago by sbl

Should we close this and maybe open a new ticket for "Syncing and validating ram and disk modes in r.stream.*" ?

comment:15 Changed 20 months ago by martinl

Milestone: 7.2.3

Ticket retargeted after milestone closed

comment:16 Changed 20 months ago by martinl

Milestone: 7.2.4

comment:17 Changed 7 months ago by martinl

Milestone: 7.2.4

Remove Milestone from Addons bugreports.

Note: See TracTickets for help on using tickets.