r.stream.basisns uses more memory than given in memory option

[sbl]

I tried to run r.stream.basins on using large flow direction / stream maps generated with r.watershed.

With m-flag and memory option set to 35GB (~2/3 of the RAM of my system) the process got killed (by out-of-mmemory-killer) after input maps were read.

So, I re-ran the module with a smaller amount of memory allocated (20 and 10 GB) in the memory option and monitored memory usage (UNIX top command). There I found that r.stream.basisns used 2x the amount of RAM allowed, on each input map. In other words, with memory=10000, it used

20GB for reading flow direction map, plus

20GB for reading streams map (40GB in total).

Finally, the process gets killed when it starts finding nodes.

I am using Ubuntu 12.04 LTS 64bit and GRASS 7.0 (revision 58620).

[hamish]

were the input maps CELL, FCELL or DCELL?

fwiw here's the code,

 opt_swapsize->description = _("Maximum memory used in memory swap mode (MB)");
 opt_swapsize->answer = "300";
...
 number_of_segs = (int)atof(opt_swapsize->answer);
 number_of_segs = number_of_segs < 32 ? (int)(32 / 0.12) : number_of_segs / 0.12;

if the stream_rast= option is used then two ram_create_map()s happen. Likely the above number_of_segs isn't taking that into account.

did it work ok when you halved the memory= option?

Hamish

[hamish]

I wrote:

if the stream_rast= option is used then two ram_create_map()s happen.

sorry, two seg_create_map() happen without stream_rast=, three if with it.

[sbl]

Both input maps (dirs and streams) were generated by r.watershed and are CELL.

r.stream.basins worked just fine when I set the memory option to ~1/7 of the available RAM.

[hamish]

how many rows and columns in the raster array?

[sbl]

Both input maps had: Rows: 180752 Columns: 141312 The region followed exactly those maps (it was not changed after maps were generated with r.watershed).

[hamish]

so for an unsegmented cell map, 180752 * 141312 * 4 bytes = ~97GB to hold one whole map in ram.

r.stream.basins/io.h defines segments as 256x256, so for memory=10000 (mb)

  seg_create_map(&map_dirs, SROWS, SCOLS, number_of_segs, CELL_TYPE);

we get

bytes = ~20.8GB, and needs that for each of the two or three open maps.

can you try r61149?

Hamish

[sbl]

FYI, 20.8GB is exactly the amount of memory consumption for reading the input maps reported by the top command, when I ran r.stream.basins with memory=10000.

[hamish]

Replying to sbl:

FYI, 20.8GB is exactly the amount of memory consumption for reading the input maps reported by the top command, when I ran r.stream.basins with memory=10000.

ok, then the math works. :-)

can someone familiar with this module give a 3rd party confirmation that the memory= option is now properly respected?

thanks, Hamish

[neteler]

TODO for all r.stream.* modules (reported by mmetz in grass-dev):

Most importantly it is the NULL handling that needs to be fixed, and synced between the ram and disk modes. More generally, the ram and disk modes need to be synced and validated. For ease of maintenance, the ram mode could be removed.

[neteler]

Milestone renamed

[neteler]

Ticket retargeted after milestone closed

[neteler]

Ticket retargeted after milestone closed

[sbl]

Should we close this and maybe open a new ticket for "Syncing and validating ram and disk modes in r.stream.*" ?

[martinl]

Ticket retargeted after milestone closed

[martinl]

Remove Milestone from Addons bugreports.