Opened 10 years ago

Closed 9 years ago

#1006 closed defect (fixed)

r.terraflow fails to stat() stream file on Windows

Reported by: marisn Owned by: grass-dev@…
Priority: critical Milestone: 6.4.0
Component: Raster Version: svn-releasebranch64
Keywords: wingrass, r.terraflow Cc:
CPU: x86-32 Platform: MSWindows Vista

Description

Temporary file folder (specified as STREAM_DIR) exists and is writeable for current user.

Run r.terraflow with default STREAM_DIR option within wxgui:

Sun Mar 14 18:47:32 2010)                                                      
r.terraflow elevation=reljefs_rst_t100@zonde filled=floddeed direction=dirrectionss swatershed=sinkss accumulation=akumulacija2 tci=tcii
stats.out: Permission denied
STREAM temporary files in C:/Users/Maris/AppData/Local/Temp
(THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF
ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE
DELETE THESE FILES MANUALLY!)
(Sun Mar 14 18:47:32 2010) Command finished (0 sec)

Running the same from CLI:

GRASS 6.4.0svn (latvija3)> r.terraflow elevation=reljefs_rst_t100 filled=fffllll
l direction=diiiirrr swatershed=siiiknk accumulation=aacccum tci=tttopoc
STREAM temporary files in C:/Users/Maris/AppData/Local/Temp  (THESE INTERMEDIATE
 STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO
SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!)
MFD flow direction
D8CUT=999999986991104.000000
Memory size: 300.00M (314572800) bytes
Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode.
Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 144


This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

WinGRASS-6.4.SVN-r41428 on Windows Vista

Attachments (2)

ami_stream_size65.diff (1.4 KB) - added by hamish 10 years ago.
include/iostream/ami_stream.h vs. grass65 for WinGrass support
r.terraflow_patch_log2 (7.4 KB) - added by hellik 10 years ago.
Error Log

Download all attachments as: .zip

Change History (60)

comment:1 Changed 10 years ago by marisn

Aaah - just noticed - could this be related to #520?

comment:2 in reply to:  description Changed 10 years ago by hellik

Replying to marisn:

Temporary file folder (specified as STREAM_DIR) exists and is writeable for current user.

Run r.terraflow with default STREAM_DIR option within wxgui:

Sun Mar 14 18:47:32 2010)                                                      
r.terraflow elevation=reljefs_rst_t100@zonde filled=floddeed direction=dirrectionss swatershed=sinkss accumulation=akumulacija2 tci=tcii
stats.out: Permission denied
STREAM temporary files in C:/Users/Maris/AppData/Local/Temp
(THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF
ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE
DELETE THESE FILES MANUALLY!)
(Sun Mar 14 18:47:32 2010) Command finished (0 sec)

Running the same from CLI:

GRASS 6.4.0svn (latvija3)> r.terraflow elevation=reljefs_rst_t100 filled=fffllll
l direction=diiiirrr swatershed=siiiknk accumulation=aacccum tci=tttopoc
STREAM temporary files in C:/Users/Maris/AppData/Local/Temp  (THESE INTERMEDIATE
 STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO
SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!)
MFD flow direction
D8CUT=999999986991104.000000
Memory size: 300.00M (314572800) bytes
Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode.
Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 144


This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

WinGRASS-6.4.SVN-r41428 on Windows Vista

I've tried this on my WinVista32-box, from command line and wx-gui, the intermediate temp files are partly written in C:\Users\syringia\AppData?\Local\Temp, but r.terraflow fails with

Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 144

Helmut

comment:3 Changed 10 years ago by hamish

what does "r.terraflow --help" say ?

(specifically the STREAM_DIR section)

  /* temporary STREAM path */
  struct Option *streamdir;
  streamdir = G_define_option() ;
  streamdir->key        = "STREAM_DIR";
  streamdir->type       = TYPE_STRING;
  streamdir->required   = NO;
#ifdef __MINGW32__
  streamdir->answer     = G_convert_dirseps_from_host(G_store(getenv("TEMP")));
#else
  streamdir->answer     = G_store("/var/tmp/");
#endif
  streamdir->description=
     _("Directory to hold temporary files (they can be large)");

Maris:

Aaah - just noticed - could this be related to #520?

probably not as that should be already fixed. (just not ideal)

can you try on a small (file size) raster like elevation.dem @ Spearfish? (existing 300mb doesn't seem so bad though)

can you try with --verbose so percent done is shown for nrows and we get more details?

Hamish

comment:4 in reply to:  3 Changed 10 years ago by hellik

Replying to hamish:

what does "r.terraflow --help" say ?

GRASS 6.4> r.terraflow --help

Beschreibung:
 Flow computation for massive grids (Float version).

Schlüsselwörter:
 Raster

Benutzung:
 r.terraflow [-sq] elevation=name filled=name direction=name
   swatershed=name accumulation=name tci=name [d8cut=value]
   [memory=value] [STREAM_DIR=string] [stats=string] [--overwrite]
   [--verbose] [--quiet]

Flags:
  -s   SFD (D8) flow (default is MFD)
  -q   schweigsam
 --o   Ausgabedateien dürfen bereits existierende Dateien überschreiben.
 --v   Detaillierter Ausgabemodus
 --q   Schweigsame Modulausgabe

Parameter:
     elevation   Name der Höhenkarte.
        filled   Output filled (flooded) elevation raster map
     direction   Output flow direction raster map
    swatershed   Output sink-watershed raster map
  accumulation   Output flow accumulation raster map
           tci   Output topographic convergence index (tci) raster map
         d8cut   If flow accumulation is larger than this value it is routed using SFD (D8) direction 
                 (meaningfull only  for MFD flow)
                  Standard: infinity
        memory   Maximum runtime memory size (in MB)
                  Standard: 300
    STREAM_DIR   Directory to hold temporary files (they can be large)
                  Standard: C:/Users/syringia/AppData/Local/Temp
         stats   Name of file containing runtime statistics
                  Standard: stats.out

(specifically the STREAM_DIR section)

  /* temporary STREAM path */
  struct Option *streamdir;
  streamdir = G_define_option() ;
  streamdir->key        = "STREAM_DIR";
  streamdir->type       = TYPE_STRING;
  streamdir->required   = NO;
#ifdef __MINGW32__
  streamdir->answer     = G_convert_dirseps_from_host(G_store(getenv("TEMP")));
#else
  streamdir->answer     = G_store("/var/tmp/");
#endif
  streamdir->description=
     _("Directory to hold temporary files (they can be large)");

Maris:

Aaah - just noticed - could this be related to #520?

probably not as that should be already fixed. (just not ideal)

can you try on a small (file size) raster like elevation.dem @ Spearfish? (existing 300mb doesn't seem so bad though)

nc-data set:

g.region -p -a rast=el_D782_6m@PERMANENT                                        
projection: 99 (Lambert Conformal Conic)
zone:       0
datum:      nad83
ellipsoid:  a=6378137 es=0.006694380022900787
north:      222504.44509601
south:      219456.439
west:       633985.268
east:       637033.27409601
nsres:      6.09601219
ewres:      6.09601219
rows:       500
cols:       500
cells:      250000

can you try with --verbose so percent done is shown for nrows and we get more details?

Hamish

g.gisenv set=DEBUG=3 
r.terraflow --verbose elevation=el_D782_6m@PERMANENT filled=floodeda direction=dirr swatershed=swat accumulation=accr tci=topgr
D2/3: G__read_Cell_head
D2/3: G__read_Cell_head_array
D3/3: region item: proj:       99
D3/3: region item: zone:       0
D3/3: region item: north:      222504.44509601
D3/3: region item: south:      219456.439
D3/3: region item: east:       637033.27409601
D3/3: region item: west:       633985.268
D3/3: region item: cols:       500
D3/3: region item: rows:       500
D3/3: region item: e-w resol:  6.09601219
D3/3: region item: n-s resol:  6.09601219
D3/3: region item: top:        1
D3/3: region item: bottom:     0
D3/3: region item: cols3:      500
D3/3: region item: rows3:      500
D3/3: region item: depths:     1
D3/3: region item: e-w resol3: 6.09601219
D3/3: region item: n-s resol3: 6.09601219
D3/3: region item: t-b resol:  1
D2/3: G__read_Cell_head
D2/3: G__read_Cell_head_array
D3/3: region item: proj:       99
D3/3: region item: zone:       0
D3/3: region item: north:      222504.44509601
D3/3: region item: south:      219456.439
D3/3: region item: east:       637033.27409601
D3/3: region item: west:       633985.268
D3/3: region item: cols:       500
D3/3: region item: rows:       500
D3/3: region item: e-w resol:  6.09601219
D3/3: region item: n-s resol:  6.09601219
D3/3: region item: format:     -1
D3/3: region item: compressed: 1
cell el_D782_6m header compatible with region header
Elevation stored as FLOAT (4B)
Region size is 500 x 500
STREAM temporary files in
C:/Users/syringia/AppData/Local/Temp  (THESE INTERMEDIATE
STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION
OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES
MANUALLY!)
file stats.out exists - renaming.
reading data from el_D782_6m to stream
C:/Users/syringia/AppData/Local/Temp/STREAM_a05332: D2/3:
G__read_Cell_head
D2/3: G__read_Cell_head_array
D3/3: region item: proj:       99
D3/3: region item: zone:       0
D3/3: region item: north:      222504.44509601
D3/3: region item: south:      219456.439
D3/3: region item: east:       637033.27409601
D3/3: region item: west:       633985.268
D3/3: region item: cols:       500
D3/3: region item: rows:       500
D3/3: region item: e-w resol:  6.09601219
D3/3: region item: n-s resol:  6.09601219
D3/3: region item: format:     -1
D3/3: region item: compressed: 1
D3/3: create window mapping (500 columns)
MFD flow direction
D8CUT=999999986991104.000000
Memory size: 300.00M (314572800) bytes
Memory manager registering memory in MM_WARN_ON_MEMORY_EXCEEDED mode.
Assertion failed: nrows * ncols == str->stream_len(), file
grass2str.h, line 144
This application has requested the Runtime to terminate it
in an unusual way.
Please contact the application's support team for more
information.

the only maybe usefull information?

Assertion failed: nrows * ncols == str->stream_len(), file
grass2str.h, line 144

Helmut

comment:5 Changed 10 years ago by hamish

STREAM temporary files in
C:/Users/syringia/AppData/Local/Temp  (THESE INTERMEDIATE
STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION
OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES
MANUALLY!)
file stats.out exists - renaming.

could you clean out the temp dir? in the original report you had:

stats.out: Permission denied

at that stage. does your user account have write permission in that dir?

this isn't related to the bug AFAIK, but I notice the D8CUT variable can't seem to decide if it wants to be an integer, float, or double:

module output: D8CUT=999999986991104.000000
option.h:  float d8cut;           /* flow value where flow accu comp switches to D8 */
main.cc:  d8cut->type = TYPE_DOUBLE;
main.cc:    opt->d8cut = atoi(d8cut->answer);
main.cc:  sprintf(buf, "D8CUT=%f", opt->d8cut);
types.h:static const flowaccumulation_type MAX_ACCU = 1e+15;

Hamish

comment:6 in reply to:  5 Changed 10 years ago by marisn

Replying to hamish:

could you clean out the temp dir? in the original report you had:

stats.out: Permission denied

at that stage. does your user account have write permission in that dir?

This isn't Slashdot where You don't have to read original report before replying.

(Thu Mar 18 13:55:51 2010)                                                      
r.terraflow elevation=elevation.dem@PERMANENT filled=flo1 direction=dir1 swatershed=sink1 accumulation=accu1 tci=tci1
stats.out: Permission denied
raster elevation.dem is of type CELL_TYPE --you should use r.terraflow.short
STREAM temporary files in C:/Users/Maris/AppData/Local/Temp
(THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF
ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE
DELETE THESE FILES MANUALLY!)
(Thu Mar 18 13:55:51 2010) Command finished (0 sec)                             
(Thu Mar 18 13:55:58 2010)                                                      
r.terraflow --verbose elevation=elevation.dem@PERMANENT filled=flo1 direction=dir1 swatershed=sink1 accumulation=accu1 tci=tci1
cell elevation.dem header compatible with region header
stats.out: Permission denied
Elevation stored as FLOAT (4B)
raster elevation.dem is of type CELL_TYPE --you should use r.terraflow.short
Region size is 466 x 633
STREAM temporary files in C:/Users/Maris/AppData/Local/Temp
(THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF
ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE
DELETE THESE FILES MANUALLY!)
(Thu Mar 18 13:55:58 2010) Command finished (0 sec)
GRASS 6.4.0svn (Spearfish60)> cd C:/Users/Maris/AppData/Local/Temp

GRASS 6.4.0svn (Spearfish60)> pwd
/tmp

GRASS 6.4.0svn (Spearfish60)> echo "11111">stats.out

GRASS 6.4.0svn (Spearfish60)> ls
Comodo         plugtmp        tmp8dmank      tmphq36zm.ppm  tmpu1igd0.ppm
Low            stats.out      tmp8dmank.pgm  tmpjwcn5a      tmpulgzoq.ppm
Maris.bmp      tmp38qqbr.ppm  tmp8dmank.ppm  tmpmvlbga      tmpv1ri6t.ppm
STREAM_a03888  tmp6qna7p      tmp_qcrpo.ppm  tmpqnwipi      tmpx9g0rx.ppm
WPDNSE         tmp6qna7p.pgm  tmpg45dva.ppm  tmpqnwipi.pgm  tmpyxrqeo.ppm
jusched.log    tmp6qna7p.ppm  tmpghekic.ppm  tmpqnwipi.ppm  ~nsu.tmp

GRASS 6.4.0svn (Spearfish60)> cat stats.out
"11111"

comment:7 Changed 10 years ago by hamish

Priority: normalmajor
Assertion failed: nrows * ncols == str->stream_len(), file
grass2str.h, line 144

Hi, I can reproduce this in Windows XP Prof. with the 6.4.0rc6 native wingrass installer.

The temp dir option seems to be respected.

Interestingly the stats.out file ends up in C:\Program Files\GRASS-6.4\msys

it contains:

Free Memory=-1
[0.0] Fri Mar 26 00:57:42 2010

Command Line: 
r.terraflow --verbose elevation=elevation.dem@PERMANENT filled=terra.flooded \
   direction=terra.dir swatershed=terra.sink accumulation=terra.accum \
   tci=terra.tci memory=600 
input elevation grid: elevation.dem
output (flooded) elevations grid: terra.flooded
output directions grid: terra.dir
output sinkwatershed grid: terra.sink
output accumulation grid: terra.accum
output tci grid: terra.tci
[0.0] MFD flow direction
[0.0] D8CUT=999999986991104.000000
[0.0] Memory size: 600.00M (629145600) bytes
region size = 288.06K (294978) elts (466 rows x 633 cols)
internal nodata value: -9999
reading raster map elevation.demto stream C:/DOCUME~1/HAMISH/LOCALS~1/Temp/STREAM_a02052

Note the Free Memory=-1 at the top of the file ...

Hamish

comment:8 in reply to:  7 ; Changed 10 years ago by hamish

Replying to hamish:

Note the Free Memory=-1 at the top of the file ...

that's nothing. freeMem() is hardcoded to that on Windows.

in r41535 I've added a new debug message to devbr6.5 to show the values tested in that assert(). So we'll see what tomorrow's nightly build says about it.

On linux for elevation.dem@spearfish I get:

D3/3: nrows=466   ncols=633    stream_len()=294978

as expected.

Hamish

comment:9 in reply to:  8 ; Changed 10 years ago by hellik

Replying to hamish:

Replying to hamish:

Note the Free Memory=-1 at the top of the file ...

that's nothing. freeMem() is hardcoded to that on Windows.

in r41535 I've added a new debug message to devbr6.5 to show the values tested in that assert(). So we'll see what tomorrow's nightly build says about it.

On linux for elevation.dem@spearfish I get:

D3/3: nrows=466   ncols=633    stream_len()=294978

as expected.

Hamish

grass65, self compiled in osgeo4w-stack, WinVista32 r41535

g.gisenv set=DEBUG=3 
g.region -p -a rast=elevation@PERMANENT                                         
projection: 99 (Lambert Conformal Conic)
zone:       0
datum:      nad83
ellipsoid:  a=6378137 es=0.006694380022900787
north:      228500
south:      215000
west:       630000
east:       645000
nsres:      10
ewres:      10
rows:       1350
cols:       1500
cells:      2025000
r.terraflow --verbose elevation=elevation@PERMANENT filled=elevflood1 direction=elevdir1 swatershed=elevsinkw1 accumulation=elevacc1 tci=elevtci1 memory=900
MFD flow direction
D8CUT=999999986991104.000000
Memory size: 900.00M (943718400) bytes
Memory manager registering memory in MM_WARN_ON_MEMORY_EXCEEDED mode.
D2/3: G__read_Cell_head
D2/3: G__read_Cell_head_array
D3/3: region item: proj:       99
D3/3: region item: zone:       0
D3/3: region item: north:      228500
D3/3: region item: south:      215000
D3/3: region item: east:       645000
D3/3: region item: west:       630000
D3/3: region item: cols:       1500
D3/3: region item: rows:       1350
D3/3: region item: e-w resol:  10
D3/3: region item: n-s resol:  10
D3/3: region item: top:        1
D3/3: region item: bottom:     0
D3/3: region item: cols3:      1500
D3/3: region item: rows3:      1350
D3/3: region item: depths:     1
D3/3: region item: e-w resol3: 10
D3/3: region item: n-s resol3: 10
D3/3: region item: t-b resol:  1
D2/3: G__read_Cell_head
D2/3: G__read_Cell_head_array
D3/3: region item: proj:       99
D3/3: region item: zone:       0
D3/3: region item: north:      228500
D3/3: region item: south:      215000
D3/3: region item: east:       645000
D3/3: region item: west:       630000
D3/3: region item: cols:       1500
D3/3: region item: rows:       1350
D3/3: region item: e-w resol:  10
D3/3: region item: n-s resol:  10
D3/3: region item: format:     -1
D3/3: region item: compressed: 1
cell elevation header compatible with region header
Elevation stored as FLOAT (4B)
Region size is 1350 x 1500
STREAM temporary files in
C:/Users/syringia/AppData/Local/Temp  (THESE INTERMEDIATE
STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION
OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES
MANUALLY!)
reading data from elevation to stream
C:/Users/syringia/AppData/Local/Temp/STREAM_a06112: D2/3:
G__read_Cell_head
D2/3: G__read_Cell_head_array
D3/3: region item: proj:       99
D3/3: region item: zone:       0
D3/3: region item: north:      228500
D3/3: region item: south:      215000
D3/3: region item: east:       645000
D3/3: region item: west:       630000
D3/3: region item: cols:       1500
D3/3: region item: rows:       1350
D3/3: region item: e-w resol:  10
D3/3: region item: n-s resol:  10
D3/3: region item: format:     -1
D3/3: region item: compressed: 1
D3/3: create window mapping (1500 columns)
D3/3: nrows=1350   ncols=1500    stream_len()=0
'''Assertion failed: nrows * ncols == str->stream_len(), file
grass2str.h, line 146'''
This application has requested the Runtime to terminate it
in an unusual way.
Please contact the application's support team for more
information.
stats.out:
Free Memory=-1
[0.0] Thu Mar 25 20:26:58 2010

Command Line: 
r.terraflow --verbose elevation=elevation@PERMANENT filled=elevflood1 direction=elevdir1 swatershed=elevsinkw1 accumulation=elevacc1 tci=elevtci1 memory=900 
input elevation grid: elevation
output (flooded) elevations grid: elevflood1
output directions grid: elevdir1
output sinkwatershed grid: elevsinkw1
output accumulation grid: elevacc1
output tci grid: elevtci1
[0.0] MFD flow direction
[0.0] D8CUT=999999986991104.000000
[0.0] Memory size: 900.00M (943718400) bytes
region size = 1.93M (2025000) elts (1350 rows x 1500 cols)
internal nodata value: -9999
reading raster map elevationto stream C:/Users/syringia/AppData/Local/Temp/STREAM_a06112

Helmut

comment:10 in reply to:  9 ; Changed 10 years ago by hellik

Replying to hellik:

Replying to hamish:

Replying to hamish:

Note the Free Memory=-1 at the top of the file ...

that's nothing. freeMem() is hardcoded to that on Windows.

in r41535 I've added a new debug message to devbr6.5 to show the values tested in that assert(). So we'll see what tomorrow's nightly build says about it.

On linux for elevation.dem@spearfish I get:

D3/3: nrows=466   ncols=633    stream_len()=294978

as expected.

Hamish

grass65, self compiled in osgeo4w-stack, WinVista32 r41535

g.gisenv set=DEBUG=3 

now it's

Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 146

Helmut

comment:11 in reply to:  10 Changed 10 years ago by hamish

Hamish:

in r41535 I've added a new debug message to devbr6.5 to show the values tested in that assert().

Replying to hellik:

now it's

Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 146

yes, because I added two lines to the source code so now it fails two lines later.

turn on debug messages by running g.gisenv set="DEBUG=3". (set that back to 0 to turn them off again once you are done)

Hamish

comment:12 in reply to:  9 ; Changed 10 years ago by hamish

Replying to hellik:

D3/3: nrows=1350 ncols=1500 stream_len()=0
Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 146

(assert() complains because the two sides are not the same)

ok, so str->stream_len() is not getting set correctly.

I wonder if this works:

  /* create output stream */
  AMI_STREAM<T>* str = new AMI_STREAM<T>();

stream_len() is defined in include/iostream/ami_stream.h (number of items in each stream)

more debug messages now turned on in that r41548 (6.5svn) ...

hmmm I get "Free Memory=-1" in stats.out on 32bit linux too.

Hamish

comment:13 in reply to:  12 Changed 10 years ago by hellik

Replying to hamish:

[...] more debug messages now turned on in that r41548 (6.5svn) ...

r41567, self compiled in the osgeo4w/

g.gisenv set=DEBUG=3
r.terraflow --verbose elevation=elevation@PERMANENT filled=elevflood direction=elevdir swatershed=elevsinkwater accumulation=elevflowacc tci=elevtci memory=900
D2/3: G__read_Cell_head
D2/3: G__read_Cell_head_array
D3/3: region item: proj:       99
D3/3: region item: zone:       0
D3/3: region item: north:      228500
D3/3: region item: south:      215000
D3/3: region item: east:       645000
D3/3: region item: west:       630000
D3/3: region item: cols:       1500
D3/3: region item: rows:       1350
D3/3: region item: e-w resol:  10
D3/3: region item: n-s resol:  10
D3/3: region item: top:        1
D3/3: region item: bottom:     0
D3/3: region item: cols3:      1500
D3/3: region item: rows3:      1350
D3/3: region item: depths:     1
D3/3: region item: e-w resol3: 10
D3/3: region item: n-s resol3: 10
D3/3: region item: t-b resol:  1
D2/3: G__read_Cell_head
D2/3: G__read_Cell_head_array
D3/3: region item: proj:       99
D3/3: region item: zone:       0
D3/3: region item: north:      228500
D3/3: region item: south:      215000
D3/3: region item: east:       645000
D3/3: region item: west:       630000
D3/3: region item: cols:       1500
D3/3: region item: rows:       1350
D3/3: region item: e-w resol:  10
D3/3: region item: n-s resol:  10
D3/3: region item: format:     -1
D3/3: region item: compressed: 1
cell elevation header compatible with region header
Elevation stored as FLOAT (4B)
Region size is 1350 x 1500
STREAM temporary files in
C:/Users/syringia/AppData/Local/Temp  (THESE INTERMEDIATE
STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION
OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES
MANUALLY!)
Reading data from <elevation> to stream <C:/Users/syringia/AppData/Local/Temp/STREAM_a07132>
D2/3: G__read_Cell_head
D2/3: G__read_Cell_head_array
D3/3: region item: proj:       99
D3/3: region item: zone:       0
D3/3: region item: north:      228500
D3/3: region item: south:      215000
D3/3: region item: east:       645000
D3/3: region item: west:       630000
D3/3: region item: cols:       1500
D3/3: region item: rows:       1350
D3/3: region item: e-w resol:  10
D3/3: region item: n-s resol:  10
D3/3: region item: format:     -1
D3/3: region item: compressed: 1
D3/3: create window mapping (1500 columns)
MFD flow direction
D8CUT=999999986991104.000000
Memory size: 900.00M (943718400) bytes
Memory manager registering memory in MM_WARN_ON_MEMORY_EXCEEDED mode.
C:/Users/syringia/AppData/Local/Temp/STREAM_a07132: length =
17179869184   sizeof(T)=1
D3/3: nrows=1350   ncols=1500    stream_len()=0
C:/Users/syringia/AppData/Local/Temp/STREAM_a07132: length =
17179869184   sizeof(T)=0
Assertion failed: nrows * ncols == str->stream_len(), file
grass2str.h, line 146
This application has requested the Runtime to terminate it
in an unusual way.
Please contact the application's support team for more
information.
Free Memory=-1
[0.0] Sat Mar 27 03:38:21 2010

Command Line: 
r.terraflow --verbose elevation=elevation@PERMANENT filled=elevflood direction=elevdir swatershed=elevsinkwater accumulation=elevflowacc tci=elevtci memory=900 
input elevation grid: elevation
output (flooded) elevations grid: elevflood
output directions grid: elevdir
output sinkwatershed grid: elevsinkwater
output accumulation grid: elevflowacc
output tci grid: elevtci
[0.0] MFD flow direction
[0.0] D8CUT=999999986991104.000000
[0.0] Memory size: 900.00M (943718400) bytes
region size = 1.93M (2025000) elts (1350 rows x 1500 cols)
internal nodata value: -9999
Reading raster map <elevation> to stream <C:/Users/syringia/AppData/Local/Temp/STREAM_a07132>.
g.gisenv set=DEBUG=0
r.terraflow --verbose elevation=elevation@PERMANENT filled=elevflood direction=elevdir swatershed=elevsinkwater accumulation=elevflowacc tci=elevtci memory=900
MFD flow direction
D8CUT=999999986991104.000000
Memory size: 900.00M (943718400) bytes
Memory manager registering memory in MM_WARN_ON_MEMORY_EXCEEDED mode.
cell elevation header compatible with region header
Elevation stored as FLOAT (4B)
Region size is 1350 x 1500
STREAM temporary files in
C:/Users/syringia/AppData/Local/Temp  (THESE INTERMEDIATE
STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION
OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES
MANUALLY!)
Reading data from <elevation> to stream <C:/Users/syringia/AppData/Local/Temp/STREAM_a01828>
C:/Users/syringia/AppData/Local/Temp/STREAM_a01828: length =
17179869184   sizeof(T)=1
C:/Users/syringia/AppData/Local/Temp/STREAM_a01828: length =
17179869184   sizeof(T)=1
Assertion failed: nrows * ncols == str->stream_len(), file
grass2str.h, line 146
This application has requested the Runtime to terminate it
in an unusual way.
Please contact the application's support team for more
information.
Free Memory=-1
[0.0] Sat Mar 27 03:40:39 2010

Command Line: 
r.terraflow --verbose elevation=elevation@PERMANENT filled=elevflood direction=elevdir swatershed=elevsinkwater accumulation=elevflowacc tci=elevtci memory=900 
input elevation grid: elevation
output (flooded) elevations grid: elevflood
output directions grid: elevdir
output sinkwatershed grid: elevsinkwater
output accumulation grid: elevflowacc
output tci grid: elevtci
[0.0] MFD flow direction
[0.0] D8CUT=999999986991104.000000
[0.0] Memory size: 900.00M (943718400) bytes
region size = 1.93M (2025000) elts (1350 rows x 1500 cols)
internal nodata value: -9999
Reading raster map <elevation> to stream <C:/Users/syringia/AppData/Local/Temp/STREAM_a01828>.

best regards Helmut

comment:14 Changed 10 years ago by hamish

are you using the elevation raster map from the NC dataset?

Helmut wrote:

Reading data from <elevation> to stream <C:/Users/syringia/AppData/Local/Temp/STREAM_a01828>
C:/Users/syringia/AppData/Local/Temp/STREAM_a01828: length =
17179869184   sizeof(T)=1
C:/Users/syringia/AppData/Local/Temp/STREAM_a01828: length =
17179869184   sizeof(T)=1
Assertion failed: nrows * ncols == str->stream_len(), file
grass2str.h, line 146

so the value of the buf.st_size variable is filled with garbage. and sizeof(T) is different from what I get on linux.

for me on 32bit linux I get this output for that line (spearfish's elevation.dem)

evelation.dem:
rows:       466
cols:       633
...
MFD flow direction
D8CUT=999999986991104.000000
Memory size: 300.00M (314572800) bytes
Memory manager registering memory in MM_WARN_ON_MEMORY_EXCEEDED mode.
Reading data from <elevation.dem> to stream </var/tmp//STREAM_qylbyz>
 100%

/var/tmp//STREAM_qylbyz: length = 1179912   sizeof(T)=4
/var/tmp//STREAM_qylbyz: length = 1179912   sizeof(T)=4

total elements=294978, nodata elements=2661
largest temporary files:
FILL: 15.75M (16518768) [294978 elements, 56B each]
FLOW: 22.30M (23385360) [292317 elements, 80B each]
Will need at least 44.60M (46770720) space available in /var/tmp/
------------------------------
COMPUTING FLOW DIRECTIONS
...

is stat() on Windows the same as it is on UNIX?

on my version of linux .st_size should be type off_t: "total size, in bytes", and stat() should return 0 on success and -1 on failure.

is %lld the right printf() setting for off_t?

I'm not sure how big <class T> should be, but probably bigger than 1 byte.. ?

Hamish

comment:15 in reply to:  14 Changed 10 years ago by hellik

Replying to hamish:

are you using the elevation raster map from the NC dataset?

yes, it is the NC-dataset.

Helmut

comment:16 in reply to:  14 Changed 10 years ago by hellik

Replying to hamish:

for me on 32bit linux I get this output for that line (spearfish's elevation.dem)

evelation.dem:
rows:       466
cols:       633
...
MFD flow direction
D8CUT=999999986991104.000000
Memory size: 300.00M (314572800) bytes
Memory manager registering memory in MM_WARN_ON_MEMORY_EXCEEDED mode.
Reading data from <elevation.dem> to stream </var/tmp//STREAM_qylbyz>
 100%

/var/tmp//STREAM_qylbyz: length = 1179912   sizeof(T)=4
/var/tmp//STREAM_qylbyz: length = 1179912   sizeof(T)=4

total elements=294978, nodata elements=2661
largest temporary files:
FILL: 15.75M (16518768) [294978 elements, 56B each]
FLOW: 22.30M (23385360) [292317 elements, 80B each]
Will need at least 44.60M (46770720) space available in /var/tmp/
------------------------------
COMPUTING FLOW DIRECTIONS
...

and with spearfish's elevation.dem on WinVista32

r.terraflow --verbose elevation=elevation.dem@PERMANENT filled=floodedelevdem direction=flowdirrast swatershed=sinkwatershrast accumulation=flowaccrast tci=topoconvrast memory=900
...
MFD flow direction
D8CUT=999999986991104.000000
Memory size: 900.00M (943718400) bytes
Memory manager registering memory in MM_WARN_ON_MEMORY_EXCEEDED mode.
C:/Users/syringia/AppData/Local/Temp/STREAM_a03800: length =
17179869184   sizeof(T)=1
D3/3: nrows=466   ncols=633    stream_len()=0
C:/Users/syringia/AppData/Local/Temp/STREAM_a03800: length =
17179869184   sizeof(T)=0
Assertion failed: nrows * ncols == str->stream_len(), file
grass2str.h, line 146
...

Helmut

comment:17 Changed 10 years ago by hamish

could you try r.in.bin? it also uses stat()'s .st_size.

(I notice it uses int,%d for FILE_SIZE too, so will not be LFS-safe)

Hamish

comment:18 in reply to:  17 Changed 10 years ago by hellik

Replying to hamish:

could you try r.in.bin? it also uses stat()'s .st_size.

I'm not at my laptop at the moment, so I've tried WinGRASS-6.5.SVN-r41581-1-Setup.exe on a WinVista32-box with the nc-dataset:

r.out.bin -h --verbose input=elevation@PERMANENT output=J:\wd\grassoutbin\elevroutbin3.bin
r.in.bin -h input=J:\wd\grassoutbin\elevroutbin3.bin output=elevrinbin bytes=4  
D2/3: G__read_Cell_head
D2/3: G__read_Cell_head_array
D3/3: region item: proj:       99
D3/3: region item: zone:       0
D3/3: region item: north:      228500
D3/3: region item: south:      215000
D3/3: region item: east:       645000
D3/3: region item: west:       630000
D3/3: region item: cols:       1500
D3/3: region item: rows:       1350
D3/3: region item: e-w resol:  10
D3/3: region item: n-s resol:  10
D3/3: region item: top:        1
D3/3: region item: bottom:     0
D3/3: region item: cols3:      1500
D3/3: region item: rows3:      1350
D3/3: region item: depths:     1
D3/3: region item: e-w resol3: 10
D3/3: region item: n-s resol3: 10
D3/3: region item: t-b resol:  1
D1/3: Creating support files for elevrinbin
D1/3: close elevrinbin compressed
D3/3: G_recreate_command()

import with r.in.bin is working

+----------------------------------------------------------------------------+
 | Layer:    elevrinbin@user1               Date: Sun Mar 28 20:14:13 2010    |
 | Mapset:   user1                          Login of Creator: normal          |
 | Location: nc_spm_07                                                        |
 | DataBase: J:\gisdata                                                       |
 | Title:     ( elevrinbin )                                                  |
 | Timestamp: none                                                            |
 |----------------------------------------------------------------------------|
 |                                                                            |
 |   Type of Map:  raster               Number of Categories: 1125930098      |
 |   Data Type:    CELL                                                       |
 |   Rows:         1350                                                       |
 |   Columns:      1500                                                       |
 |   Total Cells:  2025000                                                    |
 |        Projection: Lambert Conformal Conic                                 |
 |            N:     228500    S:     215000   Res:    10                     |
 |            E:     645000    W:     630000   Res:    10                     |
 |   Range of data:    min = 1113477295  max = 1125930098                     |
 |                                                                            |
 |   Data Description:                                                        |
 |    generated by r.in.bin                                                   |
 |                                                                            |
 |   Comments:                                                                |
 |    r.in.bin -h input="J:\wd\grassoutbin\elevroutbin3.bin" output="elevr\   |
 |    inbin" bytes=4                                                          |
 |                                                                            |
 +----------------------------------------------------------------------------+

Helmut

comment:19 Changed 10 years ago by hamish

Priority: majorcritical

comment:20 Changed 10 years ago by hamish

Summary: r.terraflow fails to create temp file on Windows when launched from wxguir.terraflow fails to stat() stream file on Windows

comment:21 Changed 10 years ago by marisn

WinGRASS-6.4.SVN-r41749-1 running Vista with DEBUG=3:

GRASS 6.4.0svn (spearfish60)> r.terraflow --verbose elevation=elevation.dem@PERM
ANENT filled=elevflood direction=elevdir swatershed=elevsinkwater accumulation=e
levflowacc tci=elevtci memory=900
D2/3: G__read_Cell_head
D2/3: G__read_Cell_head_array
D3/3: region item: proj:       1
D3/3: region item: zone:       13
D3/3: region item: north:      4928010
D3/3: region item: south:      4913700
D3/3: region item: east:       609000
D3/3: region item: west:       589980
D3/3: region item: cols:       634
D3/3: region item: rows:       477
D3/3: region item: e-w resol:  30
D3/3: region item: n-s resol:  30
D3/3: region item: top:        1
D3/3: region item: bottom:     0
D3/3: region item: cols3:      634
D3/3: region item: rows3:      477
D3/3: region item: depths:     1
D3/3: region item: e-w resol3: 30
D3/3: region item: n-s resol3: 30
D3/3: region item: t-b resol:  1
D2/3: G__read_Cell_head
D2/3: G__read_Cell_head_array
D3/3: region item: proj:       1
D3/3: region item: zone:       13
D3/3: region item: north:      4928000
D3/3: region item: south:      4914020
D3/3: region item: east:       609000
D3/3: region item: west:       590010
D3/3: region item: cols:       633
D3/3: region item: rows:       466
D3/3: region item: e-w resol:  30
D3/3: region item: n-s resol:  30
D3/3: region item: format:     1
D3/3: region item: compressed: 1
cell elevation.dem header compatible with region header
Elevation stored as FLOAT (4B)
BR╬DIN┬JUMS:raster elevation.dem is of type CELL_TYPE --you should use
            r.terraflow.short
Region size is 477 x 634
STREAM temporary files in C:/Users/Maris/AppData/Local/Temp  (THESE INTERMEDIATE
 STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO
SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!)
stats.out: Permission denied

comment:22 in reply to:  14 Changed 10 years ago by glynn

Replying to hamish:

is stat() on Windows the same as it is on UNIX?

Sort of. However: if the file is open, it reports the size as zero bytes (or possibly the original size if the file was opened for update without truncation). Once the file is closed, it reports the correct size. Also, fstat() works correctly on open files.

Test case:

$ cat test.c
#include <stdio.h>
#include <sys/stat.h>

int main(void)
{
	char buf[1024];
	struct stat st;
	FILE *fp;
	int i;
	fp = fopen("test.dat", "wb");
	for (i = 0; i < 64; i++)
		fwrite(buf, 1, sizeof(buf), fp);
	fflush(fp);
	stat("test.dat", &st);
	printf("%d\n", (int)st.st_size);
	fstat(fileno(fp), &st);
	printf("%d\n", (int)st.st_size);
	fclose(fp);
	stat("test.dat", &st);
	printf("%d\n", (int)st.st_size);
	return 0;
}

$ gcc -Wall test.c
$ ./a.exe
0
65536
65536

A more portable way to determine the length of a file is to use fseek(SEEK_END), e.g.:

	long save = ftell(fp);
	fseek(fp, 0, SEEK_END);
	long size = ftell(fp);
	fseek(fp, save, SEEK_SET);
	return size;

comment:23 Changed 10 years ago by hamish

Keywords: r.terraflow added; wxgui terraflow removed
  • removed wxgui keyword, it also happens from the msys prompt.

Maris: fyi extra debug messages only added in 6.5svn.

STREAM temporary files in C:/Users/Maris/AppData/Local/Temp
...
stats.out: Permission denied

stats.out tries to write to the current working directory, try running

cd C:/Users/Maris/AppData/Local/Temp

or your home dir before starting the program. (or change the stats= filename option)

(IIUC stats.out file just contains extra debug messages; STREAM is a copy of the raster map stored on disk for random access)

testing with the latest 6.5 nightly wingrass build (r41761), it looks like the stat() on an open file is in fact the problem:

#spearfish
g.region rast=elevation.dem
r.terraflow ele=elevation.dem filled=rterra.fill dir=rterra.dir \
  swat=rterra.swat acc=rterra.acc tci=rterra.tci mem=600
...
D3/3: create window mapping (633 columns)
C:/DOCUME~1/OPG/LOCALS~1/Temp/STREAM_a03788: length = 0   sizeof(T)=4
D3/3: nrows=466   ncols=633    stream_len()=0
C:/DOCUME~1/OPG/LOCALS~1/Temp/STREAM_a03788: length = 0   sizeof(T)=4
Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 146

# file size on disk:
wc -c C:/DOCUME~1/OPG/LOCALS~1/Temp/STREAM_a03788 
1179912
633*466 = 294978 cells
1179912 / 294978 = 4 bytes per cell

Hamish

comment:24 Changed 10 years ago by hamish

In comment:12, Hamish wrote:

ok, so str->stream_len() is not getting set correctly.

...

stream_len() is defined in include/iostream/ami_stream.h (number of items in each stream)

In comment:22, Glynn wrote:

A more portable way to determine the length of a file is to use fseek(SEEK_END), e.g.:

	long save = ftell(fp);
	fseek(fp, 0, SEEK_END);
	long size = ftell(fp);
	fseek(fp, save, SEEK_SET);
	return size;

ok, thanks. how's this:

(patch vs. 6.5svn)

Index: include/iostream/ami_stream.h
===================================================================
--- include/iostream/ami_stream.h       (revision 41749)
+++ include/iostream/ami_stream.h       (working copy)
@@ -351,21 +351,32 @@
 template<class T>
 off_t AMI_STREAM<T>::stream_len(void) {
 
+  long posn_save, st_size;
+
   fflush(fp);
 
-  struct stat buf;
-  if (stat(path, &buf) == -1) {
-    perror("AMI_STREAM::stream_len(): fstat failed ");
-       perror(path);
-    assert(0);
-    exit(1);
+  posn_save = ftell(fp);
+  if(posn_save == -1) {
+     perror("ERROR: AMI_STREAM::stream_len(): ftell(fp) failed ");
+     perror(path);
+     exit(1);
   }
 
+  fseek(fp, 0, SEEK_END);
+  st_size = ftell(fp);
+  if(st_size == -1) {
+     perror("ERROR: AMI_STREAM::stream_len(): ftell[SEEK_END] failed ");
+     perror(path);
+     exit(1);
+  }
+
+  fseek(fp, posn_save, SEEK_SET);
+
   //DEBUG:
   fprintf(stderr, "%s: length = %lld   sizeof(T)=%d\n",
-         path, buf.st_size, sizeof(T));
+         path, st_size, sizeof(T));
 
-  return (buf.st_size / sizeof(T));
+  return (st_size / sizeof(T));
 };
 
 

Hamish

comment:25 in reply to:  24 ; Changed 10 years ago by adanner

Replying to hamish:

In comment:12, Hamish wrote:

ok, so str->stream_len() is not getting set correctly.

...

stream_len() is defined in include/iostream/ami_stream.h (number of items in each stream)

In comment:22, Glynn wrote:

A more portable way to determine the length of a file is to use fseek(SEEK_END), e.g.:

	long save = ftell(fp);
	fseek(fp, 0, SEEK_END);
	long size = ftell(fp);
	fseek(fp, save, SEEK_SET);
	return size;

ok, thanks. how's this:

(patch vs. 6.5svn)

Index: include/iostream/ami_stream.h
===================================================================
--- include/iostream/ami_stream.h       (revision 41749)
+++ include/iostream/ami_stream.h       (working copy)
@@ -351,21 +351,32 @@
 template<class T>
 off_t AMI_STREAM<T>::stream_len(void) {
 
+  long posn_save, st_size;
+
   fflush(fp);
 
-  struct stat buf;
-  if (stat(path, &buf) == -1) {
-    perror("AMI_STREAM::stream_len(): fstat failed ");
-       perror(path);
-    assert(0);
-    exit(1);
+  posn_save = ftell(fp);
+  if(posn_save == -1) {
+     perror("ERROR: AMI_STREAM::stream_len(): ftell(fp) failed ");
+     perror(path);
+     exit(1);
   }
 
+  fseek(fp, 0, SEEK_END);
+  st_size = ftell(fp);
+  if(st_size == -1) {
+     perror("ERROR: AMI_STREAM::stream_len(): ftell[SEEK_END] failed ");
+     perror(path);
+     exit(1);
+  }
+
+  fseek(fp, posn_save, SEEK_SET);
+
   //DEBUG:
   fprintf(stderr, "%s: length = %lld   sizeof(T)=%d\n",
-         path, buf.st_size, sizeof(T));
+         path, st_size, sizeof(T));
 
-  return (buf.st_size / sizeof(T));
+  return (st_size / sizeof(T));
 };
 
 

Hamish

fseek and ftell only work for 32-bit file offsets on 32-bit system as the offset for fseek and the return value of ftell is a long type. At least in linux, the proper return type is an off_t which can be set to 32 or 64 bits at compile time and fseeko and ftello use off_t types. A real fix should support 64 bit file offsets on 32 bit and 64-bit systems in Windows and Linux, but this gets a bit complicated. Is there a guaranteed portable 64-bit type in Grass? A portable 64-bit G_fseek might avoid future problems of this type.

comment:26 in reply to:  25 Changed 10 years ago by glynn

Replying to adanner:

fseek and ftell only work for 32-bit file offsets on 32-bit system as the offset for fseek and the return value of ftell is a long type. At least in linux, the proper return type is an off_t which can be set to 32 or 64 bits at compile time and fseeko and ftello use off_t types. A real fix should support 64 bit file offsets on 32 bit and 64-bit systems in Windows and Linux, but this gets a bit complicated. Is there a guaranteed portable 64-bit type in Grass? A portable 64-bit G_fseek might avoid future problems of this type.

In 7.0, G_fseek() and G_ftell() use off_t and use fseeko/ftello where available. But Windows doesn't have an equivalent of _FILE_OFFSET_BITS; you have to explicitly use off64_t, lseek64(), fseeko64() etc.

Also, "long" is still only 32 bits on Win64.

Finally, C89 doesn't guarantee the existence of a 64-bit type; "long long" is C99, which isn't widely supported. gcc and MSVC support "most" of C99, but not all of it. Which means that you can either code for C89, code for whichever subset of C99 is supported by some specific set of compilers, or code for C99 and be unable to find a compiler which will accept it.

GRASS aims to compile with any sane C89 compiler (excluding technically conforming yet obscure cases, e.g. 60-bit "int", ones-complement representation, etc). Anything which uses extensions should have a fall-back.

comment:27 Changed 10 years ago by hamish

as LFS support for WinGrass is not a goer for 6.4.0, how about if we #ifdef the ftell() patch for MINGW32 in the release branch, and leave the UNIXes running the non-portable stat() version.

Lack of large file support especially sucks for r.terraflow which likes to have very large temp files, but it's better than the current situation on WinGrass which is that the module won't run at all.

Maybe we can figure out something better to include with 6.4.1.

Hamish

Changed 10 years ago by hamish

Attachment: ami_stream_size65.diff added

include/iostream/ami_stream.h vs. grass65 for WinGrass support

comment:28 in reply to:  27 ; Changed 10 years ago by hamish

Replying to hamish:

as LFS support for WinGrass is not a goer for 6.4.0, how about if we #ifdef the ftell() patch for MINGW32 in the release branch, and leave the UNIXes running the non-portable stat() version.

Lack of large file support especially sucks for r.terraflow which likes to have very large temp files, but it's better than the current situation on WinGrass which is that the module won't run at all.

Maybe we can figure out something better to include with 6.4.1.

patch attached for consideration. debug messages should be removed after testing.

Hamish

Changed 10 years ago by hellik

Attachment: r.terraflow_patch_log2 added

Error Log

comment:29 in reply to:  28 Changed 10 years ago by hellik

Replying to hamish:

Replying to hamish:

as LFS support for WinGrass is not a goer for 6.4.0, how about if we #ifdef the ftell() patch for MINGW32 in the release branch, and leave the UNIXes running the non-portable stat() version.

Lack of large file support especially sucks for r.terraflow which likes to have very large temp files, but it's better than the current situation on WinGrass which is that the module won't run at all.

Maybe we can figure out something better to include with 6.4.1.

patch attached for consideration. debug messages should be removed after testing.

Hamish

patch tested on a self compiled grass65 on WinVista? 32. shortened error log attached.

r.terraflow --verbose elevation=elevation@PERMANENT filled=filflood direction=flowdir swatershed=sinkwatershed accumulation=flowacc tci=tpoconv memory=800
MFD flow direction
D8CUT=999999986991104.000000
Memory size: 800.00M (838860800) bytes
Memory manager registering memory in MM_WARN_ON_MEMORY_EXCEEDED mode.
cell elevation header compatible with region header
Elevation stored as FLOAT (4B)
Region size is 1350 x 1500
STREAM temporary files in
C:/Users/syringia/AppData/Local/Temp  (THESE INTERMEDIATE
STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION
OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES
MANUALLY!)
Reading data from <elevation> to stream <C:/Users/syringia/AppData/Local/Temp/STREAM_a06204>
[...]
Assertion failed: ae == AMI_ERROR_NO_ERROR, file
filldepr.cc, line 136
This application has requested the Runtime to terminate it
in an unusual way.
Please contact the application's support team for more
information.
UnionFind::makeSet: reallocate double 2000
UnionFind::makeSet: reallocate double 4000

comment:30 Changed 9 years ago by marisn

Tested with WinGRASS-6.4.SVN-r42668-1 on Vista. It's better now, as in CLI it fails with same message as in wxGUI (stats.out: Permission denied). No more assertion errors. Horray! Still r.terraflow is not working for windows users.

comment:31 in reply to:  30 Changed 9 years ago by mmetz

Milestone: 6.4.06.4.1
Priority: criticalmajor

Replying to marisn:

Tested with WinGRASS-6.4.SVN-r42668-1 on Vista. It's better now, as in CLI it fails with same message as in wxGUI (stats.out: Permission denied). No more assertion errors. Horray! Still r.terraflow is not working for windows users.

Still getting the same assertion error on 6.4 r42714:

Assertion failed: nrows * ncols == str->stream_len(), file
grass2str.h, line 144

About stats.out: Permission denied, stats.out is not written to STREAM_DIR because everything in STREAM_DIR is deleted after successful completion, stats.out is written to the current working directory and remains there for inspection. On my test Windows system, the current directory is /c/Dokumente und Einstellungen/Markus? and I can write to it.

r.terraflow is on windows crippled by missing LFS in GRASS. Its strength is its performance with massive datasets, but these can't be processed on windows because of the 2GB file size limit. Grass with LFS on windows will likely only be available in grass 7, planned for next month, but not in grass 6. r.terraflow is working fine on Linux and its functionality is covered by other modules, therefore it is IMHO not critical to have a working r.terraflow on windows. It would be very nice though.

Considering this as well as previous comments, I think it's more realistic to get it fixed in 6.4.1 or 6.5, but under windows it will be really useful only in grass 7.0 with LFS.

Markus M

comment:32 Changed 9 years ago by hamish

Milestone: 6.4.16.4.0
Priority: majorcritical

putting this back on the table for 6.4.0, without hesitation. see plan in comment:27.

I don't think we are far off having this working -- it may be working in 6.5 already and just need a backport.

r.terraflow (and v.buffer) are flagship modules for GRASS and frankly it's an embarrassment not to have them working properly (even if their full glory is crippled as in the case of LFS + r.terraflow).

Markus Metz wrote:

Still getting the same assertion error on 6.4 r42714:

well, yes, because the stat() work-around has not been backported to the 6.4 branch pending testing in 6.5. Please test it in 6.5 (run from a dir with write permissions :). I hope that stats.out error is not fatal to the process.

regards, Hamish

comment:33 Changed 9 years ago by hamish

sorry, it seems I never applied the patch for include/iostream/ami_stream.h in this ticket to 6.5svn as intended. I will do so ASAP so it can be tested from the daily binary snapshot build this weekend.

Hamish

comment:34 Changed 9 years ago by hamish

patch applied to 6.5svn in r42716.

comment:35 in reply to:  32 ; Changed 9 years ago by mmetz

Replying to hamish:

putting this back on the table for 6.4.0, without hesitation. see plan in comment:27.

Hmph. The ticket is however not a blocker ;-)

I don't think we are far off having this working -- it may be working in 6.5 already and just need a backport.

r.terraflow (and v.buffer) are flagship modules for GRASS and frankly it's an embarrassment not to have them working properly (even if their full glory is crippled as in the case of LFS + r.terraflow).

Its flagship character comes from its ability to process very large datasets, but all its functionality including the ability to process large datasets is also covered by other modules. Without LFS, r.terraflow is limited to less than 27 million non-NULL cells, converting the module from a flagship to a dinghy. I am very much inclined to change milestone to 7.0 and set priority to normal.

Tested 6.5svn r42716 on XP, same error as in comment:29 in filldepr.cc. Once that is fixed, we might have a look at why r.terraflow refuses to run with output from r.volcano... No, seriously, r.terraflow and iostream is written for Linux, as also previous tickets have shown, no reason to expect that this is the last problem with Windows. Please leave it for 6.5 or 7.0.

Markus M

comment:36 in reply to:  35 ; Changed 9 years ago by hamish

Replying to mmetz:

the ability to process large datasets is also covered by other modules.

I know that, but the (perhaps recently historic) way the modules are presented in the menus does not make that clear to users.

Tested 6.5svn r42716 on XP, same error as in comment:29 in filldepr.cc.

thanks. so the error is now:

Reading data from <elevation> to stream <C:/Users/syringia/AppData/Local/Temp/STREAM_a06204>
[...]
Assertion failed: ae == AMI_ERROR_NO_ERROR, file
filldepr.cc, line 136
This application has requested the Runtime to terminate it
in an unusual way.

from raster/r.terraflow/filldepr.cc

    /*read next edge*/
    ae = boundaryStr->read_item(&nextedge);
    assert(ae == AMI_ERROR_NO_ERROR);

which calls include/iostream/mem_stream.h

template<class T>
AMI_err MEM_STREAM<T>::read_item(T **elt)  {

  assert(data);

  if(curr == dataend) {
    return AMI_ERROR_END_OF_STREAM;
  }
  *elt = curr;
  curr++;
  return AMI_ERROR_NO_ERROR;
};

so it seems that it is prematurely reaching the end of the stream?

Once that is fixed, we might have a look at why r.terraflow refuses to run with output from r.volcano...

does it not?? how about a sum=r.volcano+r.surf.fractal surface? perhaps because there are no basins except the small one in the crater in that fake landscape? is there a ticket for this?

No, seriously, r.terraflow and iostream is written for Linux, as also previous tickets have shown, no reason to expect that this is the last problem with Windows. Please leave it for 6.5 or 7.0.

we could easily say the same for all of GRASS. some parts will be easier to port than others, but if we didn't care about MS-Windows at all we could have released 6.4 mid-last year. in this case it sucks, is not known to be fundamentally deep, and I'd like to try and fix it. even if it is crippled due to LFS (you might say 32bit XP is too with its 3.2gb RAM limit), at least people can still work on tutorials/examples and learn how to use it for when they graduate to a gruntier system.

in pursuit of a technical fix I will add debug messages to mem_stream.h. (r42731)

regards, Hamish

comment:37 in reply to:  36 ; Changed 9 years ago by mmetz

Replying to hamish:

Replying to mmetz:

the ability to process large datasets is also covered by other modules.

I know that, but the (perhaps recently historic) way the modules are presented in the menus does not make that clear to users.

Well, then we have to change the way modules are presented in the menus.

from raster/r.terraflow/filldepr.cc

    /*read next edge*/
    ae = boundaryStr->read_item(&nextedge);
    assert(ae == AMI_ERROR_NO_ERROR);

which calls include/iostream/mem_stream.h

template<class T>
AMI_err MEM_STREAM<T>::read_item(T **elt)  {

  assert(data);

  if(curr == dataend) {
    return AMI_ERROR_END_OF_STREAM;
  }
  *elt = curr;
  curr++;
  return AMI_ERROR_NO_ERROR;
};

so it seems that it is prematurely reaching the end of the stream?

I never thought I would agree to fix depression filling... The mingw32 version of stream_len() does not work 100%, workaround in r42734 and r42735. Worksforme now on Windows. On Linux, no difference in results. Can I now put warning messages in r.terraflow that its hydrological algorithms from the late 80ies and early 90ies do not produce very accurate results (its emphasis is on the computational complexity of the algorithms, rather than on modeling realistic flow)?

r.terraflow and iostream is written for Linux, as also previous tickets have shown, no reason to expect that this is the last problem with Windows. Please leave it for 6.5 or 7.0.

we could easily say the same for all of GRASS.

Most parts of grass are not giving problems with mingw32, maybe because these adhere to C89? C++ doesn't, so problems might be expected here.

[snip] even if it is crippled due to LFS (you might say 32bit XP is too with its 3.2gb RAM limit),

You might say that mingw32 is the problem (apart from the msys problems). IIUC, stuff compiled on e.g. Windows7 64bit with mingw32 is still 32bit and has all the 32bit limits, even if the OS is 64bit. Plus other annoyances like sizeof(long) = 4 on Windows 64bit.

in pursuit of a technical fix I will add debug messages to mem_stream.h. (r42731)

IMHO, the debug messages can be removed now, then the fix can be backported to 6.4.1 ;-)

Markus M

comment:38 in reply to:  37 ; Changed 9 years ago by hamish

Replying to mmetz:

Well, then we have to change the way modules are presented in the menus.

[aside] can we have a README.howto file in the gui/wxpython/xml/ dir? Every time I edit menudata.xml it seems to get overwritten some time later by an automated update which I don't see documented anywhere. It sort of kills my motivation to work on improving the menus anymore. tx.

so it seems that it is prematurely reaching the end of the stream?

I never thought I would agree to fix depression filling...

:)

The mingw32 version of stream_len() does not work 100%,

could you explain for us humble c++ students?

workaround in r42734 and r42735. Worksforme now on Windows. On Linux, no difference in results.

excellent!

Can I now put warning messages in r.terraflow that its hydrological algorithms from the late 80ies and early 90ies do not produce very accurate results (its emphasis is on the computational complexity of the algorithms, rather than on modeling realistic flow)?

I'd rather you got together with Andrew and/or Laura and co-authored a journal article racingW comparing the two with the current offerings from the other major proprietary GISs. :-) [seriously]

http://www.cs.duke.edu/geo*/terraflow/speedup.html

Are the core r.watershed algorithms by Ehlschlaeger any more modern? or have you now replaced and upgraded them to 'state of the art'? (whatever that means)

r.terraflow and iostream is written for Linux,

well I knew a version for Arc existed, so thought it possible,

http://wwwmath1.uni-muenster.de:8010/u/jan/TerraFlowExtension/

IMHO, the debug messages can be removed now, then the fix can be backported to 6.4.1 ;-)

I removed the noisiest one, will do the rest after testing tomorrow's autobuild. (and of course merge with trunk too)

If it doesn't change results on UNIX it may as well go into 6.4.0. It doesn't matter how risky it is on Windows as it can't be worse than before. If you feel there's a risk for breakage on UNIX from this we could wait til 6.4.1 for the backport & disable building it in 6.4.0, but we should perhaps look over the menus in that case to make sure that the emphasis and descriptions are all up to date.

cheers, Hamish

comment:39 in reply to:  37 ; Changed 9 years ago by hellik

Replying to mmetz: [...]

IMHO, the debug messages can be removed now, then the fix can be backported to 6.4.1 ;-)

Markus M

I've tried a self compiled Grass65 (r42743) in the osgeo4w-tree on my WinVista32-box with nc-sample-dataset started from the gui.

r.terraflow elevation=elevation filled=ff direction=dd accumulation=aa tci=tt memory=800

r.terraflow starts and seems to calculate, but after a while, the gui freezes at the following message in the command output

[...]
assining preliminary directions
finding flat areas (plateaus and depressions)

Helmut

comment:40 in reply to:  38 ; Changed 9 years ago by martinl

Replying to hamish:

[aside] can we have a README.howto file in the gui/wxpython/xml/ dir? Every time I edit menudata.xml it seems to get overwritten some time later by an automated update which I don't see documented anywhere. It sort of kills my motivation to work on improving the menus anymore. tx.

menudata.xml is maintained by source:grass/trunk/gui/wxpython/support/update_menudata.py which touches only description and keywords (based on command usage desc). Other items should be untouched, if not please report it as a bug.

Martin

comment:41 in reply to:  39 ; Changed 9 years ago by mmetz

Replying to hellik:

I've tried a self compiled Grass65 (r42743) in the osgeo4w-tree on my WinVista32-box with nc-sample-dataset started from the gui.

r.terraflow elevation=elevation filled=ff direction=dd accumulation=aa tci=tt memory=800

r.terraflow starts and seems to calculate, but after a while, the gui freezes

Try less memory, e.g. default 300 (are there currently 800MB free on your vista 32bit box?). Or wait for it complete...

Markus M

comment:42 in reply to:  41 ; Changed 9 years ago by hellik

Replying to mmetz:

Replying to hellik:

I've tried a self compiled Grass65 (r42743) in the osgeo4w-tree on my WinVista32-box with nc-sample-dataset started from the gui.

r.terraflow elevation=elevation filled=ff direction=dd accumulation=aa tci=tt memory=800

r.terraflow starts and seems to calculate, but after a while, the gui freezes

Try less memory, e.g. default 300 (are there currently 800MB free on your vista 32bit box?). Or wait for it complete...

Markus M

the whole wx-gui freezes also with 300mb.

starting r.terraflow from the msys-command-line, it's finishing all calculations.

Helmut

comment:43 in reply to:  42 ; Changed 9 years ago by mmetz

Replying to hellik:

Replying to mmetz:

Replying to hellik:

I've tried a self compiled Grass65 (r42743) in the osgeo4w-tree on my WinVista32-box with nc-sample-dataset started from the gui.

r.terraflow elevation=elevation filled=ff direction=dd accumulation=aa tci=tt memory=800

r.terraflow starts and seems to calculate, but after a while, the gui freezes

Try less memory, e.g. default 300 (are there currently 800MB free on your vista 32bit box?). Or wait for it complete...

Markus M

the whole wx-gui freezes also with 300mb.

starting r.terraflow from the msys-command-line, it's finishing all calculations.

Hmm. Good to hear that it works on CLI. In wxGUI, it might become better once the debug messages have been removed, I had to clear output regularly to get it going again. Too much for the wxGUI output window?

Markus M

comment:44 in reply to:  40 Changed 9 years ago by hamish

Replying to martinl:

menudata.xml is maintained by source:grass/trunk/gui/wxpython/support/update_menudata.py which touches only description and keywords (based on command usage desc).

thanks, added some notes as

source:grass/trunk/gui/wxpython/xml/menudata.README

how to get that into the programmer's manual? doxygen in header comments of update_menudata.py?

Hamish

comment:45 in reply to:  43 Changed 9 years ago by hamish

Replying to mmetz:

Hmm. Good to hear that it works on CLI. In wxGUI, it might become better once the debug messages have been removed, I had to clear output regularly to get it going again. Too much for the wxGUI output window?

probably. You'll get similar GUI lockup failures if you try and run vector modules while DEBUG=5. It can't handle the incoming message volume. Tcl/Tk? had the same problem.

Hamish

comment:46 Changed 9 years ago by hamish

Hi, just tested 6.5svn r42750 nightly build on XP. And I'm back to the original error message & exit code 3:

#spearfish
g.region rast=elevation.10m
r.terraflow elev=elevation.10m ...

Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 144

the left over .../Temp/STREAM_a02688 file is 10,619,208 bytes.

  • (and .../Temp/ is full of 100+ old tmp*.ppm files at a MB each... :-/ new ones even after exiting the wxGUI cleanly. now that I look, it's the same in /tmp/ in linux)
D2/3: G__read_Cell_head
D2/3: G__read_Cell_head_array
D3/3: region item: proj:       1
D3/3: region item: zone:       13
D3/3: region item: north:      4928000
D3/3: region item: south:      4914020
D3/3: region item: east:       609000
D3/3: region item: west:       590010
D3/3: region item: cols:       1899
D3/3: region item: rows:       1398
D3/3: region item: e-w resol:  10
D3/3: region item: n-s resol:  10
D3/3: region item: format:     -1
D3/3: region item: compressed: 1
D3/3: create window mapping (1899 columns)
C:/DOCUME~1/Hamish/LOCALS~1/Temp/STREAM_a02688: length = 9175040   sizeof(T)=4
D3/3: nrows=1398   ncols=1899    stream_len()=2293760
C:/DOCUME~1/Hamish/LOCALS~1/Temp/STREAM_a02688: length = 9175040   sizeof(T)=4
Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 146

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

the length = 9175040 reported by the debug message jumps around every time I run it, sometimes it's 0, sometimes its ~550000 or so.

  • also often it dies just after the "stats.out file found, renaming." message with error code 1. rm stats.out* at the msys prompt gets it working again.

Hamish

comment:47 in reply to:  46 ; Changed 9 years ago by mmetz

Replying to hamish:

Hi, just tested 6.5svn r42750 nightly build on XP. And I'm back to the original error message & exit code 3:

#spearfish
g.region rast=elevation.10m
r.terraflow elev=elevation.10m ...

That same command works for me.

Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 144 }}}

Close to an insult to ask, but have you done make distclean (package.sh doesn't)? You could check the headers in include/grass/iostream in the installation dir (not the source dir) to see if the patches are really in there.

Markus M

comment:48 in reply to:  47 Changed 9 years ago by hellik

Replying to mmetz:

Replying to hamish:

Hi, just tested 6.5svn r42750 nightly build on XP. And I'm back to the original error message & exit code 3:

#spearfish
g.region rast=elevation.10m
r.terraflow elev=elevation.10m ...

That same command works for me.

Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 144 }}}

Close to an insult to ask, but have you done make distclean (package.sh doesn't)? You could check the headers in include/grass/iostream in the installation dir (not the source dir) to see if the patches are really in there.

Markus M

I've tried a completly fresh svn-checkout of Grass65 in my WinVista32. r.terraflow works and finishes also now in the wx-gui.

Helmut

comment:49 in reply to:  47 ; Changed 9 years ago by hamish

Replying to mmetz:

Close to an insult to ask,

not at all,

but have you done make distclean (package.sh doesn't)?

I tried the nightly binary snapshot from josef.fsv.cvut.cz. Even if package.sh does not, I don't know if the build script does (or does not).

You could check the headers in include/grass/iostream in the installation dir (not the source dir) to see if the patches are really in there.

When back at work with access I will check that, hopefully it explains it...

Hamish

comment:50 in reply to:  49 ; Changed 9 years ago by hamish

Replying to hamish:

Replying to mmetz:

but have you done make distclean (package.sh doesn't)?

I tried the nightly binary snapshot from josef.fsv.cvut.cz.

I have now added "make distclean" to source:grass-addons/tools/wingrass-packager/grass_compile.sh

maybe it should better go into mswindows/osgeo4w/package.sh, changing the 'make clean' after ./configure there to 'make distclean' before. I leave that decision to the script's authors & runners. (I assume they update this by hand anyway)

Hamish

comment:51 in reply to:  50 Changed 9 years ago by hamish

Replying to hamish:

Replying to hamish:

Replying to mmetz:

but have you done make distclean (package.sh doesn't)?

I tried the nightly binary snapshot from josef.fsv.cvut.cz.

I have now added "make distclean" to source:grass-addons/tools/wingrass-packager/grass_compile.sh

maybe it should better go into mswindows/osgeo4w/package.sh, changing the 'make clean' after ./configure there to 'make distclean' before. I leave that decision to the script's authors & runners. (I assume they update this by hand anyway)

I had a look changing it in package.sh, but I think that needs more fundamental change. Right now it is only re-running ./configure and 'make clean' on fresh checkouts. Given that, I'm surprised that trunk still builds actually.

Hamish

comment:52 in reply to:  28 ; Changed 9 years ago by mmetz

Replying to hamish:

patch attached for consideration. debug messages should be removed after testing.

The patch seems to work in combination with the later patch, results are identical between windows grass65, Linux grass65 patched and Linux grass65 unpatched.

Note that the patch for stream_len() contradicts the principle of r.terraflow to avoid random seek at all costs, fseek should only be called to reset the file pointer to the beginning of the file. Each call to the patched stream_len in mingw32 calls fseek twice, and stream_len itself is called very often as the debug messages illustrate (so often that wxGUI might freeze...). As a result, r.terraflow is heavily slowed down on windows: using elevation.10m from spearfish60 as input (2.6 million cells) takes with r.terraflow twice as long as with r.watershed in disk swap mode, and that even though all temp files also of r.terraflow are small enough to be cached by my test system.

Markus M

comment:53 in reply to:  52 Changed 9 years ago by glynn

Replying to mmetz:

stream_len itself is called very often

In which case, it shouldn't be calling stat() either. This isn't exactly fast on Unix, is quite slow on Windows, and is extremely slow on Cygwin.

On Unix, fstat() is typically significantly faster than stat(), as stat() needs to resolve the pathname to an inode, which requires lookups and permission checks on each component in the path.

However: both fstat() and stat() reflect the size of the underlying file, which isn't necessarily the same as the size of the stream (FILE*) if buffering is in use. The only reliable way to determine the size of a stream is fseek(SEEK_END)+ftell().

If stream_len() is called often, it would be better to make the AMI_STREAM class keep track of its own length, updating it in the write_item() and write_array() methods.

comment:54 Changed 9 years ago by hamish

6.4.0svn nightly build r42815 and 6.5svn r42819 on XP: still dies with the assertion mismatch in grass2str.h line 144 as in original report.

perhaps 'make distclean' is still not making it through to the nightly builds??

if I try again it also still dies if stats.out already exists in the pwd after giving you a 'renaming stats.out' message.

comment:55 in reply to:  54 ; Changed 9 years ago by hellik

Replying to hamish:

6.4.0svn nightly build r42815 and 6.5svn r42819 on XP: still dies with the assertion mismatch in grass2str.h line 144 as in original report.

perhaps 'make distclean' is still not making it through to the nightly builds??

if I try again it also still dies if stats.out already exists in the pwd after giving you a 'renaming stats.out' message.

I can confirm this with 6.4.0svn nightly build r42815 on my WinVista32-box.

in my self-compiled grass64 and grass65 r.terraflow is working. also in a self built WinGrass-installer of grass64 and grass65.

but for compiling I have to comment out the lines added in r42799, r42797 because package.sh isn't working for me.

Helmut

comment:56 in reply to:  55 ; Changed 9 years ago by mmetz

Replying to hamish:

6.4.0svn nightly build r42815 and 6.5svn r42819 on XP: still dies with the assertion mismatch in grass2str.h line 144 as in original report.

perhaps 'make distclean' is still not making it through to the nightly builds??

Maybe because configure-stamp has not been removed before running package.sh again [1]

Replying to hellik:

in my self-compiled grass64 and grass65 r.terraflow is working. also in a self built WinGrass-installer of grass64 and grass65.

I can confirm.

Markus M

[1] https://trac.osgeo.org/grass/browser/grass/branches/releasebranch_6_4/mswindows/osgeo4w/package.sh#L31

comment:57 in reply to:  56 Changed 9 years ago by martinl

Replying to mmetz:

perhaps 'make distclean' is still not making it through to the nightly builds??

Should be fixed by r42837. Please test the next build (this night). Martin

comment:58 Changed 9 years ago by hamish

Resolution: fixed
Status: newclosed

nightly package builds now working; module limited to 32bit files on WinGrass moved to #1106; D8CUT variable type confusion moved to #1107.

now works on WinGrass; closing bug.

Note: See TracTickets for help on using tickets.