Changes between Initial Version and Version 1 of Ticket #2764, comment 33


Ignore:
Timestamp:
Jan 9, 2018, 10:52:17 PM (6 years ago)
Author:
dylan

Legend:

Unmodified
Added
Removed
Modified

  • Ticket #2764, comment 33

    initial v1  
    33> Markus Neteler in particular spent a lot of time to fix various systems for parallel execution of GRASS commands. GRASS itself was never the problem, instead the main problem was that the multiple outputs to be written to a single storage device were too much for that storage device.
    44
    5 OK. Good to know. Are there any other diagnostics for these kind of problems, other than looking through the output from `dmesg` or kernel messages? I typically run `dstat` while developing parallel processing scripts, but I haven't noticed any disk-thrashing in this case. I see that the SSD is at about 5% of its "write bandwidth" as reported by `dstat`.
     5OK. Good to know. Are there any other diagnostics for these kind of problems, other than looking through the output from `dmesg` or kernel messages? I typically run `dstat` while developing parallel processing scripts, but I haven't noticed any disk-thrashing in this case.
    66
    7 Each instance of `r.sun` is writing out ~5000 rows of data over a period of about 8 minutes, so that is 5000 rows * 8 processes * 1/8 proc per minute * 1/60 minutes per second = ~80 write operations per second (assuming rows are written as processed). Are rows written 1-by-1, or in batches?
     7**Update**
     8Looking at disk and I/O stats at a 1 second granularity via:
     9{{{
     10dstat -m -c --top-cpu --top-bio --top-io -d -D sdd1 -r --disk-util  1
     11}}}
     12I see that the SSD (sdd1) is idle for most of the time and then spikes to 30-80% of its "disk utilization" as reported by `dstat --disk-util` when several `r.sun` jobs finish (?). As a comparison, making a .tgz on the SSD yields ~ 12% disk utilization.
     13
     14Each instance of `r.sun` is writing out ~5000 rows of data over a period of about 8 minutes, so that is 5000 rows * 8 processes * 1/8 proc per minute * 1/60 minutes per second = ~80 write operations per second (assuming rows are written as processed). It would appear (from the `dstat` output) that rows are written in batches?
    815
    916