r.terraflow: file=/home/mlennert/STREAM/STREAM_tQhXkQ:cannot read!: Bad address

[mlennert]

In all three version (6.4, 6devel and 7), I get the following error running r.terraflow in the North Carolina demo data set:

GRASS 6.5.svn (nc_spm_06):~ > g.region rast=elevation
GRASS 6.5.svn (nc_spm_06):~ > r.terraflow elevation=elevation@PERMANENT filled=fill direction=flow swatershed=waters accumulation=accum tci=tci STREAM_DIR=/home/mlennert/STREAM
STREAM temporary files in /home/mlennert/STREAM  (THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!)
file stats.out exists - renaming.
MFD flow direction
D8CUT=999999986991104.000000
Memory size: 300.00M (314572800) bytes
Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode.
total elements=2025000, nodata elements=0
largest temporary files:
FILL: 108.15M (113400000) [2025000 elements, 56B each]
FLOW: 154.50M (162000000) [2025000 elements, 80B each]
Will need at least 308.99M (324000000) space available in
/home/mlennert/STREAM
------------------------------
COMPUTING FLOW DIRECTIONS
classifying nodata (inner & boundary)
assigning preliminary directions
finding flat areas (plateaus and depressions)
file=/home/mlennert/STREAM/STREAM_lRe0NI:cannot read!: Bad address
r.terraflow: /home/mlennert/SRC/GRASS/grass6_devel/dist.i486-pc-linux-gnu/include/grass/iostream/ami_sort_impl.h:91: size_t makeRun_Block(AMI_STREAM<T>*, T*, unsigned int, Compare*) [with T = plateauType, Compare = ijCmpPlateauType]: Assertion `err == AMI_ERROR_NO_ERROR || err == AMI_ERROR_END_OF_STREAM' failed.
Aborted

I have over 60GB of disk space in /home (and I get the same error running it with the default /var/tmp). This is using Debian Lenny. The same error has been reported by a student running Ubuntu Intrepid.

Moritz

[glynn]

Replying to mlennert:

In all three version (6.4, 6devel and 7), I get the following error running r.terraflow in the North Carolina demo data set:

file=/home/mlennert/STREAM/STREAM_lRe0NI:cannot read!: Bad address

The error message corresponds to fread() failing with EFAULT (invalid buffer pointer) in either AMI_STREAM<T>::read_item() or AMI_STREAM<T>::read_array(). I would suspect the latter, as the former reads into the read_tmp member of the AMI_STREAM<T> object

The assert() failure:

r.terraflow: /home/mlennert/SRC/GRASS/grass6_devel/dist.i486-pc-linux-gnu/include/grass/iostream/ami_sort_impl.h:91: size_t makeRun_Block(AMI_STREAM<T>*, T*, unsigned int, Compare*) [with T = plateauType, Compare = ijCmpPlateauType]: Assertion `err == AMI_ERROR_NO_ERROR || err == AMI_ERROR_END_OF_STREAM' failed.

also suggests the read_array() method, and indicates that a bogus pointer is passed as the "data" parameter to makeRun_Block().

Beyond that, I can only suggest running it under gdb and printing a full backtrace ("bt full") when it aborts.

[mlennert]

Replying to glynn:

Beyond that, I can only suggest running it under gdb and printing a full backtrace ("bt full") when it aborts.

Done.

[mlennert]

Replying to glynn:

Replying to mlennert:

In all three version (6.4, 6devel and 7), I get the following error running r.terraflow in the North Carolina demo data set:

file=/home/mlennert/STREAM/STREAM_lRe0NI:cannot read!: Bad address

The error message corresponds to fread() failing with EFAULT (invalid buffer pointer) in either AMI_STREAM<T>::read_item() or AMI_STREAM<T>::read_array(). I would suspect the latter, as the former reads into the read_tmp member of the AMI_STREAM<T> object

The assert() failure:

r.terraflow: /home/mlennert/SRC/GRASS/grass6_devel/dist.i486-pc-linux-gnu/include/grass/iostream/ami_sort_impl.h:91: size_t makeRun_Block(AMI_STREAM<T>*, T*, unsigned int, Compare*) [with T = plateauType, Compare = ijCmpPlateauType]: Assertion `err == AMI_ERROR_NO_ERROR || err == AMI_ERROR_END_OF_STREAM' failed.

also suggests the read_array() method, and indicates that a bogus pointer is passed as the "data" parameter to makeRun_Block().

Beyond that, I can only suggest running it under gdb and printing a full backtrace ("bt full") when it aborts.

Any ideas based on the backtrace ?

Moritz

[glynn]

Replying to mlennert:

Any ideas based on the backtrace ?

The backtrace doesn't entirely make sense, e.g. frame #4 has:

        err = AMI_ERROR_NO_ERROR

so the assert() shouldn't be triggering. But that may just be an artifact of optimisation. Can you reproduce this if libiostream and r.terraflow are built without optimisation?

[mlennert]

Replying to glynn:

Replying to mlennert:

Any ideas based on the backtrace ?

The backtrace doesn't entirely make sense, e.g. frame #4 has:

        err = AMI_ERROR_NO_ERROR

so the assert() shouldn't be triggering. But that may just be an artifact of optimisation. Can you reproduce this if libiostream and r.terraflow are built without optimisation?

Now I get err = AMI_ERROR_IO_ERROR.

[glynn]

Replying to mlennert:

so the assert() shouldn't be triggering. But that may just be an artifact of optimisation. Can you reproduce this if libiostream and r.terraflow are built without optimisation?

Now I get err = AMI_ERROR_IO_ERROR.

Well, the immediate issue is that fread() is returning a short count but not setting the end-of-file indicator.

The short count is to be expected: it's trying to read run_size = 13027967 items although it only expects there to be last_run_size = 38343 items in the file (and has only allocated enough memory for that many).

Given the "Bad address" from perror(), my suspicion is that fread() is complaining that &data[len] isn't valid (i.e. the buffer isn't large enough to hold the requested number of items), rather than simply trying to read them and seeing if it hits EOF before it segfaults.

I suggest changing runFormation() to only try to read as many items as it has space for, i.e. replace the existing code with the commented-out version in:

  //for (size_t i=0; i< nb_runs; i++) {
  while(!instream->eof()) {
    //crt_run_size = (i == nb_runs-1) ? last_run_size: run_size;
    
    //SDEBUG cout << "i=" << i << ":  runsize=" << crt_run_size << ", ";  

    crt_run_size = makeRun_Block(instream, data, run_size, cmp);
/* #ifdef BLOCKED_RUN */
/*     makeRun(instream, data, crt_run_size, cmp); */
/* #else         */
/*     makeRun_Block(instream, data, crt_run_size, cmp); */
/* #endif */

[mlennert]

Replying to glynn:

I suggest changing runFormation() to only try to read as many items as it has space for, i.e. replace the existing code with the commented-out version in:

  //for (size_t i=0; i< nb_runs; i++) {
  while(!instream->eof()) {
    //crt_run_size = (i == nb_runs-1) ? last_run_size: run_size;
    
    //SDEBUG cout << "i=" << i << ":  runsize=" << crt_run_size << ", ";  

    crt_run_size = makeRun_Block(instream, data, run_size, cmp);
/* #ifdef BLOCKED_RUN */
/*     makeRun(instream, data, crt_run_size, cmp); */
/* #else         */
/*     makeRun_Block(instream, data, crt_run_size, cmp); */
/* #endif */

Is this what you meant ?:

--- SRC/GRASS/grass_trunk/include/iostream/ami_sort_impl.h	2008-08-13 21:36:15.000000000 +0200
+++ SRC/GRASS/grass_trunk/dist.i486-pc-linux-gnu/include/grass/iostream/ami_sort_impl.h	2009-10-13 17:00:30.000000000 +0200
@@ -221,13 +221,13 @@
   }
   SDEBUG MM_manager.print();
  
-  //for (size_t i=0; i< nb_runs; i++) {
-  while(!instream->eof()) {
-    //crt_run_size = (i == nb_runs-1) ? last_run_size: run_size;
+  for (size_t i=0; i< nb_runs; i++) {
+  //while(!instream->eof()) {
+    crt_run_size = (i == nb_runs-1) ? last_run_size: run_size;
     
-    //SDEBUG cout << "i=" << i << ":  runsize=" << crt_run_size << ", ";  
+    SDEBUG cout << "i=" << i << ":  runsize=" << crt_run_size << ", ";  
 
-    crt_run_size = makeRun_Block(instream, data, run_size, cmp);
+    //crt_run_size = makeRun_Block(instream, data, run_size, cmp);
 /* #ifdef BLOCKED_RUN */
 /*     makeRun(instream, data, crt_run_size, cmp); */
 /* #else         */

Now I get:

GRASS 7.0.svn (nc_spm_06):~ > r.terraflow elevation=elevation@PERMANENT filled=fill direction=flow swatershed=waters accumulation=accum tci=tci
STREAM temporary files in /var/tmp/  (THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!)
file stats.out exists - renaming.
MFD flow direction
D8CUT=999999986991104.000000
Memory size: 300.00M (314572800) bytes
Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode.
total elements=2025000, nodata elements=0
largest temporary files:
FILL: 108.15M (113400000) [2025000 elements, 56B each]
FLOW: 154.50M (162000000) [2025000 elements, 80B each]
Will need at least 308.99M (324000000) space available in /var/tmp/
------------------------------
COMPUTING FLOW DIRECTIONS
classifying nodata (inner & boundary)
assigning preliminary directions
finding flat areas (plateaus and depressions)
EMPQUEUEADAPTIVE: starting in-memory pqueue
EMPQUEUEADAPTIVE: available memory: 297.413MB
EMPQUEUEADAPTIVE: desired memory: 297.413MB
sz_stream: 270388 buf_arity: 200 mm_overhead: 8665728 mm_avail: 311859944.
EMPQUEUEADAPTIVE: memory overhead set to 8.26428MB
EMPQUEUEADAPTIVE: pqsize set to 37899277
r.terraflow: plateau.cc:364: void detectPlateaus::relabelPlateaus(): Assertion `root >= LABEL_START' failed.
Aborted

backtrace of that attached. I again see

ae = AMI_ERROR_NO_ERROR

even though everything is compiled with -g -O0.

[glynn]

Replying to mlennert:

Is this what you meant ?:

You also need to uncomment at least one of the makeRun/makeRun_Block calls, otherwise it isn't actually reading anything. Or just leave the existing makeRun_Block call uncommented.

[mlennert]

Replying to glynn:

Replying to mlennert:

Is this what you meant ?:

You also need to uncomment at least one of the makeRun/makeRun_Block calls, otherwise it isn't actually reading anything. Or just leave the existing makeRun_Block call uncommented.

The latter didn't work (same error message as initially, but the former seems to do the trick (diff attached).

[fram]

I tried to apply the patch of r.terraflow (r.terraflow.diff). I do not know if I did it fine, what I did was to change the original "include/iostream/ami_sort_impl.h" file with the modifications in the .diff file. If this was fine, below is the output of r.terraflow which to me seems to give a different error as before.

=============================================================== GRASS 6.4.0RC5 (WGS84_UTM33):~ > r.terraflow elevation=dem_tagliato filled=flood direction=flow swatershed=sink accumulation=accumulation tci=tci d8cut=infinity memory=1600 STREAM_DIR=/tmp stats=stats.out STREAM temporary files in /tmp (THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!) MFD flow direction D8CUT=999999986991104.000000 Memory size: 1.56G (1677721600) bytes Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode. total elements=67071935, nodata elements=61864898 largest temporary files: FILL: 3.50G (3756028360) [67071935 elements, 56B each] FLOW: 397.27M (416562960) [5207037 elements, 80B each] Will need at least 7.00G (7512056720) space available in /tmp

---

COMPUTING FLOW DIRECTIONS classifying nodata (inner & boundary) EMPQUEUEADAPTIVE: starting in-memory pqueue EMPQUEUEADAPTIVE: available memory: 1597.93MB EMPQUEUEADAPTIVE: desired memory: 1597.93MB sz_stream: 270388 buf_arity: 200 mm_overhead: 8665728 mm_avail: 1675549602. EMPQUEUEADAPTIVE: memory overhead set to 8.26428MB EMPQUEUEADAPTIVE: pqsize set to 208360484 assigning preliminary directions finding flat areas (plateaus and depressions) EMPQUEUEADAPTIVE: starting in-memory pqueue EMPQUEUEADAPTIVE: available memory: 1597.41MB EMPQUEUEADAPTIVE: desired memory: 1597.41MB sz_stream: 270388 buf_arity: 200 mm_overhead: 8665728 mm_avail: 1675008754. EMPQUEUEADAPTIVE: memory overhead set to 8.26428MB EMPQUEUEADAPTIVE: pqsize set to 208292878 assigning directions on plateaus generating watersheds and watershed graph AMI_STREAM::write_item failed. /tmp/STREAM_HHSoQw: File too large r.terraflow: /usr/local/svn/grass/grass640_rc5/dist.i686-pc-linux-gnu/include/grass/iostream/ami_stream.h:560: AMI_err AMI_STREAM<T>::write_item(const T&) [with T = compressedWaterWindowType]: Assertion `0' failed. Abortito GRASS 6.4.0RC5 (WGS84_UTM33):~ > ===============================================================

Francesco

[neteler]

Replying to fram:

I tried to apply the patch of r.terraflow (r.terraflow.diff). I do not know if I did it fine, what I did was to change the original "include/iostream/ami_sort_impl.h" file with the modifications in the .diff file. If this was fine, below is the output of r.terraflow which to me seems to give a different error as before.

For the record: the user didn't compile with LFS support. Done that, the patch works.

Fixed in 6.4 (r39700), 6.5 (r39701), 7 (r39702). Closing.