Opened 10 years ago

Closed 10 years ago

#775 closed defect (fixed)

r.terraflow: file=/home/mlennert/STREAM/STREAM_tQhXkQ:cannot read!: Bad address

Reported by: mlennert Owned by: grass-dev@…
Priority: normal Milestone: 6.4.0
Component: Raster Version: 6.4.0 RCs
Keywords: terraflow aborted Cc:
CPU: Unspecified Platform: Linux

Description

In all three version (6.4, 6devel and 7), I get the following error running r.terraflow in the North Carolina demo data set:

GRASS 6.5.svn (nc_spm_06):~ > g.region rast=elevation
GRASS 6.5.svn (nc_spm_06):~ > r.terraflow elevation=elevation@PERMANENT filled=fill direction=flow swatershed=waters accumulation=accum tci=tci STREAM_DIR=/home/mlennert/STREAM
STREAM temporary files in /home/mlennert/STREAM  (THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!)
file stats.out exists - renaming.
MFD flow direction
D8CUT=999999986991104.000000
Memory size: 300.00M (314572800) bytes
Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode.
total elements=2025000, nodata elements=0
largest temporary files:
FILL: 108.15M (113400000) [2025000 elements, 56B each]
FLOW: 154.50M (162000000) [2025000 elements, 80B each]
Will need at least 308.99M (324000000) space available in
/home/mlennert/STREAM
------------------------------
COMPUTING FLOW DIRECTIONS
classifying nodata (inner & boundary)
assigning preliminary directions
finding flat areas (plateaus and depressions)
file=/home/mlennert/STREAM/STREAM_lRe0NI:cannot read!: Bad address
r.terraflow: /home/mlennert/SRC/GRASS/grass6_devel/dist.i486-pc-linux-gnu/include/grass/iostream/ami_sort_impl.h:91: size_t makeRun_Block(AMI_STREAM<T>*, T*, unsigned int, Compare*) [with T = plateauType, Compare = ijCmpPlateauType]: Assertion `err == AMI_ERROR_NO_ERROR || err == AMI_ERROR_END_OF_STREAM' failed.
Aborted

I have over 60GB of disk space in /home (and I get the same error running it with the default /var/tmp). This is using Debian Lenny. The same error has been reported by a student running Ubuntu Intrepid.

Moritz

Attachments (3)

r.terraflow.backtrace (8.9 KB) - added by mlennert 10 years ago.
r.terraflow.backtrace2 (6.5 KB) - added by mlennert 10 years ago.
r.terraflow.diff (1.1 KB) - added by mlennert 10 years ago.

Download all attachments as: .zip

Change History (14)

comment:1 in reply to:  description ; Changed 10 years ago by glynn

Replying to mlennert:

In all three version (6.4, 6devel and 7), I get the following error running r.terraflow in the North Carolina demo data set:

file=/home/mlennert/STREAM/STREAM_lRe0NI:cannot read!: Bad address

The error message corresponds to fread() failing with EFAULT (invalid buffer pointer) in either AMI_STREAM<T>::read_item() or AMI_STREAM<T>::read_array(). I would suspect the latter, as the former reads into the read_tmp member of the AMI_STREAM<T> object

The assert() failure:

r.terraflow: /home/mlennert/SRC/GRASS/grass6_devel/dist.i486-pc-linux-gnu/include/grass/iostream/ami_sort_impl.h:91: size_t makeRun_Block(AMI_STREAM<T>*, T*, unsigned int, Compare*) [with T = plateauType, Compare = ijCmpPlateauType]: Assertion `err == AMI_ERROR_NO_ERROR || err == AMI_ERROR_END_OF_STREAM' failed.

also suggests the read_array() method, and indicates that a bogus pointer is passed as the "data" parameter to makeRun_Block().

Beyond that, I can only suggest running it under gdb and printing a full backtrace ("bt full") when it aborts.

comment:2 in reply to:  1 Changed 10 years ago by mlennert

Replying to glynn:

Beyond that, I can only suggest running it under gdb and printing a full backtrace ("bt full") when it aborts.

Done.

comment:3 in reply to:  1 ; Changed 10 years ago by mlennert

Replying to glynn:

Replying to mlennert:

In all three version (6.4, 6devel and 7), I get the following error running r.terraflow in the North Carolina demo data set:

file=/home/mlennert/STREAM/STREAM_lRe0NI:cannot read!: Bad address

The error message corresponds to fread() failing with EFAULT (invalid buffer pointer) in either AMI_STREAM<T>::read_item() or AMI_STREAM<T>::read_array(). I would suspect the latter, as the former reads into the read_tmp member of the AMI_STREAM<T> object

The assert() failure:

r.terraflow: /home/mlennert/SRC/GRASS/grass6_devel/dist.i486-pc-linux-gnu/include/grass/iostream/ami_sort_impl.h:91: size_t makeRun_Block(AMI_STREAM<T>*, T*, unsigned int, Compare*) [with T = plateauType, Compare = ijCmpPlateauType]: Assertion `err == AMI_ERROR_NO_ERROR || err == AMI_ERROR_END_OF_STREAM' failed.

also suggests the read_array() method, and indicates that a bogus pointer is passed as the "data" parameter to makeRun_Block().

Beyond that, I can only suggest running it under gdb and printing a full backtrace ("bt full") when it aborts.

Any ideas based on the backtrace ?

Moritz

comment:4 in reply to:  3 ; Changed 10 years ago by glynn

Replying to mlennert:

Any ideas based on the backtrace ?

The backtrace doesn't entirely make sense, e.g. frame #4 has:

        err = AMI_ERROR_NO_ERROR

so the assert() shouldn't be triggering. But that may just be an artifact of optimisation. Can you reproduce this if libiostream and r.terraflow are built without optimisation?

Changed 10 years ago by mlennert

Attachment: r.terraflow.backtrace added

comment:5 in reply to:  4 ; Changed 10 years ago by mlennert

Replying to glynn:

Replying to mlennert:

Any ideas based on the backtrace ?

The backtrace doesn't entirely make sense, e.g. frame #4 has:

        err = AMI_ERROR_NO_ERROR

so the assert() shouldn't be triggering. But that may just be an artifact of optimisation. Can you reproduce this if libiostream and r.terraflow are built without optimisation?

Now I get err = AMI_ERROR_IO_ERROR.

comment:6 in reply to:  5 ; Changed 10 years ago by glynn

Replying to mlennert:

so the assert() shouldn't be triggering. But that may just be an artifact of optimisation. Can you reproduce this if libiostream and r.terraflow are built without optimisation?

Now I get err = AMI_ERROR_IO_ERROR.

Well, the immediate issue is that fread() is returning a short count but not setting the end-of-file indicator.

The short count is to be expected: it's trying to read run_size = 13027967 items although it only expects there to be last_run_size = 38343 items in the file (and has only allocated enough memory for that many).

Given the "Bad address" from perror(), my suspicion is that fread() is complaining that &data[len] isn't valid (i.e. the buffer isn't large enough to hold the requested number of items), rather than simply trying to read them and seeing if it hits EOF before it segfaults.

I suggest changing runFormation() to only try to read as many items as it has space for, i.e. replace the existing code with the commented-out version in:

  //for (size_t i=0; i< nb_runs; i++) {
  while(!instream->eof()) {
    //crt_run_size = (i == nb_runs-1) ? last_run_size: run_size;
    
    //SDEBUG cout << "i=" << i << ":  runsize=" << crt_run_size << ", ";  

    crt_run_size = makeRun_Block(instream, data, run_size, cmp);
/* #ifdef BLOCKED_RUN */
/*     makeRun(instream, data, crt_run_size, cmp); */
/* #else         */
/*     makeRun_Block(instream, data, crt_run_size, cmp); */
/* #endif */

comment:7 in reply to:  6 ; Changed 10 years ago by mlennert

Replying to glynn:

I suggest changing runFormation() to only try to read as many items as it has space for, i.e. replace the existing code with the commented-out version in:

  //for (size_t i=0; i< nb_runs; i++) {
  while(!instream->eof()) {
    //crt_run_size = (i == nb_runs-1) ? last_run_size: run_size;
    
    //SDEBUG cout << "i=" << i << ":  runsize=" << crt_run_size << ", ";  

    crt_run_size = makeRun_Block(instream, data, run_size, cmp);
/* #ifdef BLOCKED_RUN */
/*     makeRun(instream, data, crt_run_size, cmp); */
/* #else         */
/*     makeRun_Block(instream, data, crt_run_size, cmp); */
/* #endif */

Is this what you meant ?:

--- SRC/GRASS/grass_trunk/include/iostream/ami_sort_impl.h	2008-08-13 21:36:15.000000000 +0200
+++ SRC/GRASS/grass_trunk/dist.i486-pc-linux-gnu/include/grass/iostream/ami_sort_impl.h	2009-10-13 17:00:30.000000000 +0200
@@ -221,13 +221,13 @@
   }
   SDEBUG MM_manager.print();
  
-  //for (size_t i=0; i< nb_runs; i++) {
-  while(!instream->eof()) {
-    //crt_run_size = (i == nb_runs-1) ? last_run_size: run_size;
+  for (size_t i=0; i< nb_runs; i++) {
+  //while(!instream->eof()) {
+    crt_run_size = (i == nb_runs-1) ? last_run_size: run_size;
     
-    //SDEBUG cout << "i=" << i << ":  runsize=" << crt_run_size << ", ";  
+    SDEBUG cout << "i=" << i << ":  runsize=" << crt_run_size << ", ";  
 
-    crt_run_size = makeRun_Block(instream, data, run_size, cmp);
+    //crt_run_size = makeRun_Block(instream, data, run_size, cmp);
 /* #ifdef BLOCKED_RUN */
 /*     makeRun(instream, data, crt_run_size, cmp); */
 /* #else         */

Now I get:

GRASS 7.0.svn (nc_spm_06):~ > r.terraflow elevation=elevation@PERMANENT filled=fill direction=flow swatershed=waters accumulation=accum tci=tci
STREAM temporary files in /var/tmp/  (THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!)
file stats.out exists - renaming.
MFD flow direction
D8CUT=999999986991104.000000
Memory size: 300.00M (314572800) bytes
Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode.
total elements=2025000, nodata elements=0
largest temporary files:
FILL: 108.15M (113400000) [2025000 elements, 56B each]
FLOW: 154.50M (162000000) [2025000 elements, 80B each]
Will need at least 308.99M (324000000) space available in /var/tmp/
------------------------------
COMPUTING FLOW DIRECTIONS
classifying nodata (inner & boundary)
assigning preliminary directions
finding flat areas (plateaus and depressions)
EMPQUEUEADAPTIVE: starting in-memory pqueue
EMPQUEUEADAPTIVE: available memory: 297.413MB
EMPQUEUEADAPTIVE: desired memory: 297.413MB
sz_stream: 270388 buf_arity: 200 mm_overhead: 8665728 mm_avail: 311859944.
EMPQUEUEADAPTIVE: memory overhead set to 8.26428MB
EMPQUEUEADAPTIVE: pqsize set to 37899277
r.terraflow: plateau.cc:364: void detectPlateaus::relabelPlateaus(): Assertion `root >= LABEL_START' failed.
Aborted

backtrace of that attached. I again see

ae = AMI_ERROR_NO_ERROR

even though everything is compiled with -g -O0.

Changed 10 years ago by mlennert

Attachment: r.terraflow.backtrace2 added

comment:8 in reply to:  7 ; Changed 10 years ago by glynn

Replying to mlennert:

Is this what you meant ?:

You also need to uncomment at least one of the makeRun/makeRun_Block calls, otherwise it isn't actually reading anything. Or just leave the existing makeRun_Block call uncommented.

comment:9 in reply to:  8 Changed 10 years ago by mlennert

Replying to glynn:

Replying to mlennert:

Is this what you meant ?:

You also need to uncomment at least one of the makeRun/makeRun_Block calls, otherwise it isn't actually reading anything. Or just leave the existing makeRun_Block call uncommented.

The latter didn't work (same error message as initially, but the former seems to do the trick (diff attached).

Changed 10 years ago by mlennert

Attachment: r.terraflow.diff added

comment:10 Changed 10 years ago by fram

Keywords: aborted added; bad address removed
Version: svn-develbranch66.4.0 RCs

I tried to apply the patch of r.terraflow (r.terraflow.diff). I do not know if I did it fine, what I did was to change the original "include/iostream/ami_sort_impl.h" file with the modifications in the .diff file. If this was fine, below is the output of r.terraflow which to me seems to give a different error as before.

=============================================================== GRASS 6.4.0RC5 (WGS84_UTM33):~ > r.terraflow elevation=dem_tagliato filled=flood direction=flow swatershed=sink accumulation=accumulation tci=tci d8cut=infinity memory=1600 STREAM_DIR=/tmp stats=stats.out STREAM temporary files in /tmp (THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!) MFD flow direction D8CUT=999999986991104.000000 Memory size: 1.56G (1677721600) bytes Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode. total elements=67071935, nodata elements=61864898 largest temporary files: FILL: 3.50G (3756028360) [67071935 elements, 56B each] FLOW: 397.27M (416562960) [5207037 elements, 80B each] Will need at least 7.00G (7512056720) space available in /tmp


COMPUTING FLOW DIRECTIONS classifying nodata (inner & boundary) EMPQUEUEADAPTIVE: starting in-memory pqueue EMPQUEUEADAPTIVE: available memory: 1597.93MB EMPQUEUEADAPTIVE: desired memory: 1597.93MB sz_stream: 270388 buf_arity: 200 mm_overhead: 8665728 mm_avail: 1675549602. EMPQUEUEADAPTIVE: memory overhead set to 8.26428MB EMPQUEUEADAPTIVE: pqsize set to 208360484 assigning preliminary directions finding flat areas (plateaus and depressions) EMPQUEUEADAPTIVE: starting in-memory pqueue EMPQUEUEADAPTIVE: available memory: 1597.41MB EMPQUEUEADAPTIVE: desired memory: 1597.41MB sz_stream: 270388 buf_arity: 200 mm_overhead: 8665728 mm_avail: 1675008754. EMPQUEUEADAPTIVE: memory overhead set to 8.26428MB EMPQUEUEADAPTIVE: pqsize set to 208292878 assigning directions on plateaus generating watersheds and watershed graph AMI_STREAM::write_item failed. /tmp/STREAM_HHSoQw: File too large r.terraflow: /usr/local/svn/grass/grass640_rc5/dist.i686-pc-linux-gnu/include/grass/iostream/ami_stream.h:560: AMI_err AMI_STREAM<T>::write_item(const T&) [with T = compressedWaterWindowType]: Assertion `0' failed. Abortito GRASS 6.4.0RC5 (WGS84_UTM33):~ > ===============================================================

Francesco

comment:11 in reply to:  10 Changed 10 years ago by neteler

Resolution: fixed
Status: newclosed

Replying to fram:

I tried to apply the patch of r.terraflow (r.terraflow.diff). I do not know if I did it fine, what I did was to change the original "include/iostream/ami_sort_impl.h" file with the modifications in the .diff file. If this was fine, below is the output of r.terraflow which to me seems to give a different error as before.

For the record: the user didn't compile with LFS support. Done that, the patch works.

Fixed in 6.4 (r39700), 6.5 (r39701), 7 (r39702). Closing.

Note: See TracTickets for help on using tickets.