Opened 14 years ago
Closed 14 years ago
#1006 closed defect (fixed)
r.terraflow fails to stat() stream file on Windows
| Reported by: | marisn | Owned by: | |
|---|---|---|---|
| Priority: | critical | Milestone: | 6.4.0 |
| Component: | Raster | Version: | svn-releasebranch64 |
| Keywords: | wingrass, r.terraflow | Cc: | |
| CPU: | x86-32 | Platform: | MSWindows Vista |
Description
Temporary file folder (specified as STREAM_DIR) exists and is writeable for current user.
Run r.terraflow with default STREAM_DIR option within wxgui:
Sun Mar 14 18:47:32 2010) r.terraflow elevation=reljefs_rst_t100@zonde filled=floddeed direction=dirrectionss swatershed=sinkss accumulation=akumulacija2 tci=tcii stats.out: Permission denied STREAM temporary files in C:/Users/Maris/AppData/Local/Temp (THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!) (Sun Mar 14 18:47:32 2010) Command finished (0 sec)
Running the same from CLI:
GRASS 6.4.0svn (latvija3)> r.terraflow elevation=reljefs_rst_t100 filled=fffllll l direction=diiiirrr swatershed=siiiknk accumulation=aacccum tci=tttopoc STREAM temporary files in C:/Users/Maris/AppData/Local/Temp (THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!) MFD flow direction D8CUT=999999986991104.000000 Memory size: 300.00M (314572800) bytes Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode. Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 144 This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information.
WinGRASS-6.4.SVN-r41428 on Windows Vista
Attachments (2)
Change History (60)
comment:1 by , 14 years ago
comment:2 by , 14 years ago
Replying to marisn:
Temporary file folder (specified as STREAM_DIR) exists and is writeable for current user.
Run r.terraflow with default STREAM_DIR option within wxgui:
Sun Mar 14 18:47:32 2010) r.terraflow elevation=reljefs_rst_t100@zonde filled=floddeed direction=dirrectionss swatershed=sinkss accumulation=akumulacija2 tci=tcii stats.out: Permission denied STREAM temporary files in C:/Users/Maris/AppData/Local/Temp (THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!) (Sun Mar 14 18:47:32 2010) Command finished (0 sec)Running the same from CLI:
GRASS 6.4.0svn (latvija3)> r.terraflow elevation=reljefs_rst_t100 filled=fffllll l direction=diiiirrr swatershed=siiiknk accumulation=aacccum tci=tttopoc STREAM temporary files in C:/Users/Maris/AppData/Local/Temp (THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!) MFD flow direction D8CUT=999999986991104.000000 Memory size: 300.00M (314572800) bytes Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode. Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 144 This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information.WinGRASS-6.4.SVN-r41428 on Windows Vista
I've tried this on my WinVista32-box, from command line and wx-gui, the intermediate temp files are partly written in C:\Users\syringia\AppData\Local\Temp, but r.terraflow fails with
Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 144
Helmut
follow-up: 4 comment:3 by , 14 years ago
what does "r.terraflow --help" say ?
(specifically the STREAM_DIR section)
/* temporary STREAM path */
struct Option *streamdir;
streamdir = G_define_option() ;
streamdir->key = "STREAM_DIR";
streamdir->type = TYPE_STRING;
streamdir->required = NO;
#ifdef __MINGW32__
streamdir->answer = G_convert_dirseps_from_host(G_store(getenv("TEMP")));
#else
streamdir->answer = G_store("/var/tmp/");
#endif
streamdir->description=
_("Directory to hold temporary files (they can be large)");
Maris:
Aaah - just noticed - could this be related to #520?
probably not as that should be already fixed. (just not ideal)
can you try on a small (file size) raster like elevation.dem @ Spearfish? (existing 300mb doesn't seem so bad though)
can you try with --verbose so percent done is shown for nrows and we get more details?
Hamish
comment:4 by , 14 years ago
Replying to hamish:
what does "r.terraflow --help" say ?
GRASS 6.4> r.terraflow --help
Beschreibung:
Flow computation for massive grids (Float version).
Schlüsselwörter:
Raster
Benutzung:
r.terraflow [-sq] elevation=name filled=name direction=name
swatershed=name accumulation=name tci=name [d8cut=value]
[memory=value] [STREAM_DIR=string] [stats=string] [--overwrite]
[--verbose] [--quiet]
Flags:
-s SFD (D8) flow (default is MFD)
-q schweigsam
--o Ausgabedateien dürfen bereits existierende Dateien überschreiben.
--v Detaillierter Ausgabemodus
--q Schweigsame Modulausgabe
Parameter:
elevation Name der Höhenkarte.
filled Output filled (flooded) elevation raster map
direction Output flow direction raster map
swatershed Output sink-watershed raster map
accumulation Output flow accumulation raster map
tci Output topographic convergence index (tci) raster map
d8cut If flow accumulation is larger than this value it is routed using SFD (D8) direction
(meaningfull only for MFD flow)
Standard: infinity
memory Maximum runtime memory size (in MB)
Standard: 300
STREAM_DIR Directory to hold temporary files (they can be large)
Standard: C:/Users/syringia/AppData/Local/Temp
stats Name of file containing runtime statistics
Standard: stats.out
(specifically the STREAM_DIR section)
/* temporary STREAM path */ struct Option *streamdir; streamdir = G_define_option() ; streamdir->key = "STREAM_DIR"; streamdir->type = TYPE_STRING; streamdir->required = NO; #ifdef __MINGW32__ streamdir->answer = G_convert_dirseps_from_host(G_store(getenv("TEMP"))); #else streamdir->answer = G_store("/var/tmp/"); #endif streamdir->description= _("Directory to hold temporary files (they can be large)");Maris:
Aaah - just noticed - could this be related to #520?
probably not as that should be already fixed. (just not ideal)
can you try on a small (file size) raster like elevation.dem @ Spearfish? (existing 300mb doesn't seem so bad though)
nc-data set:
g.region -p -a rast=el_D782_6m@PERMANENT projection: 99 (Lambert Conformal Conic) zone: 0 datum: nad83 ellipsoid: a=6378137 es=0.006694380022900787 north: 222504.44509601 south: 219456.439 west: 633985.268 east: 637033.27409601 nsres: 6.09601219 ewres: 6.09601219 rows: 500 cols: 500 cells: 250000
can you try with --verbose so percent done is shown for nrows and we get more details?
Hamish
g.gisenv set=DEBUG=3
r.terraflow --verbose elevation=el_D782_6m@PERMANENT filled=floodeda direction=dirr swatershed=swat accumulation=accr tci=topgr D2/3: G__read_Cell_head D2/3: G__read_Cell_head_array D3/3: region item: proj: 99 D3/3: region item: zone: 0 D3/3: region item: north: 222504.44509601 D3/3: region item: south: 219456.439 D3/3: region item: east: 637033.27409601 D3/3: region item: west: 633985.268 D3/3: region item: cols: 500 D3/3: region item: rows: 500 D3/3: region item: e-w resol: 6.09601219 D3/3: region item: n-s resol: 6.09601219 D3/3: region item: top: 1 D3/3: region item: bottom: 0 D3/3: region item: cols3: 500 D3/3: region item: rows3: 500 D3/3: region item: depths: 1 D3/3: region item: e-w resol3: 6.09601219 D3/3: region item: n-s resol3: 6.09601219 D3/3: region item: t-b resol: 1 D2/3: G__read_Cell_head D2/3: G__read_Cell_head_array D3/3: region item: proj: 99 D3/3: region item: zone: 0 D3/3: region item: north: 222504.44509601 D3/3: region item: south: 219456.439 D3/3: region item: east: 637033.27409601 D3/3: region item: west: 633985.268 D3/3: region item: cols: 500 D3/3: region item: rows: 500 D3/3: region item: e-w resol: 6.09601219 D3/3: region item: n-s resol: 6.09601219 D3/3: region item: format: -1 D3/3: region item: compressed: 1 cell el_D782_6m header compatible with region header Elevation stored as FLOAT (4B) Region size is 500 x 500 STREAM temporary files in C:/Users/syringia/AppData/Local/Temp (THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!) file stats.out exists - renaming. reading data from el_D782_6m to stream C:/Users/syringia/AppData/Local/Temp/STREAM_a05332: D2/3: G__read_Cell_head D2/3: G__read_Cell_head_array D3/3: region item: proj: 99 D3/3: region item: zone: 0 D3/3: region item: north: 222504.44509601 D3/3: region item: south: 219456.439 D3/3: region item: east: 637033.27409601 D3/3: region item: west: 633985.268 D3/3: region item: cols: 500 D3/3: region item: rows: 500 D3/3: region item: e-w resol: 6.09601219 D3/3: region item: n-s resol: 6.09601219 D3/3: region item: format: -1 D3/3: region item: compressed: 1 D3/3: create window mapping (500 columns) MFD flow direction D8CUT=999999986991104.000000 Memory size: 300.00M (314572800) bytes Memory manager registering memory in MM_WARN_ON_MEMORY_EXCEEDED mode. Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 144 This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information.
the only maybe usefull information?
Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 144
Helmut
follow-up: 6 comment:5 by , 14 years ago
STREAM temporary files in C:/Users/syringia/AppData/Local/Temp (THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!) file stats.out exists - renaming.
could you clean out the temp dir? in the original report you had:
stats.out: Permission denied
at that stage. does your user account have write permission in that dir?
this isn't related to the bug AFAIK, but I notice the D8CUT variable can't seem to decide if it wants to be an integer, float, or double:
module output: D8CUT=999999986991104.000000 option.h: float d8cut; /* flow value where flow accu comp switches to D8 */ main.cc: d8cut->type = TYPE_DOUBLE; main.cc: opt->d8cut = atoi(d8cut->answer); main.cc: sprintf(buf, "D8CUT=%f", opt->d8cut); types.h:static const flowaccumulation_type MAX_ACCU = 1e+15;
Hamish
comment:6 by , 14 years ago
Replying to hamish:
could you clean out the temp dir? in the original report you had:
stats.out: Permission deniedat that stage. does your user account have write permission in that dir?
This isn't Slashdot where You don't have to read original report before replying.
(Thu Mar 18 13:55:51 2010) r.terraflow elevation=elevation.dem@PERMANENT filled=flo1 direction=dir1 swatershed=sink1 accumulation=accu1 tci=tci1 stats.out: Permission denied raster elevation.dem is of type CELL_TYPE --you should use r.terraflow.short STREAM temporary files in C:/Users/Maris/AppData/Local/Temp (THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!) (Thu Mar 18 13:55:51 2010) Command finished (0 sec) (Thu Mar 18 13:55:58 2010) r.terraflow --verbose elevation=elevation.dem@PERMANENT filled=flo1 direction=dir1 swatershed=sink1 accumulation=accu1 tci=tci1 cell elevation.dem header compatible with region header stats.out: Permission denied Elevation stored as FLOAT (4B) raster elevation.dem is of type CELL_TYPE --you should use r.terraflow.short Region size is 466 x 633 STREAM temporary files in C:/Users/Maris/AppData/Local/Temp (THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!) (Thu Mar 18 13:55:58 2010) Command finished (0 sec)
GRASS 6.4.0svn (Spearfish60)> cd C:/Users/Maris/AppData/Local/Temp GRASS 6.4.0svn (Spearfish60)> pwd /tmp GRASS 6.4.0svn (Spearfish60)> echo "11111">stats.out GRASS 6.4.0svn (Spearfish60)> ls Comodo plugtmp tmp8dmank tmphq36zm.ppm tmpu1igd0.ppm Low stats.out tmp8dmank.pgm tmpjwcn5a tmpulgzoq.ppm Maris.bmp tmp38qqbr.ppm tmp8dmank.ppm tmpmvlbga tmpv1ri6t.ppm STREAM_a03888 tmp6qna7p tmp_qcrpo.ppm tmpqnwipi tmpx9g0rx.ppm WPDNSE tmp6qna7p.pgm tmpg45dva.ppm tmpqnwipi.pgm tmpyxrqeo.ppm jusched.log tmp6qna7p.ppm tmpghekic.ppm tmpqnwipi.ppm ~nsu.tmp GRASS 6.4.0svn (Spearfish60)> cat stats.out "11111"
follow-up: 8 comment:7 by , 14 years ago
| Priority: | normal → major |
|---|
Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 144
Hi, I can reproduce this in Windows XP Prof. with the 6.4.0rc6 native wingrass installer.
The temp dir option seems to be respected.
Interestingly the stats.out file ends up in C:\Program Files\GRASS-6.4\msys
it contains:
Free Memory=-1 [0.0] Fri Mar 26 00:57:42 2010 Command Line: r.terraflow --verbose elevation=elevation.dem@PERMANENT filled=terra.flooded \ direction=terra.dir swatershed=terra.sink accumulation=terra.accum \ tci=terra.tci memory=600 input elevation grid: elevation.dem output (flooded) elevations grid: terra.flooded output directions grid: terra.dir output sinkwatershed grid: terra.sink output accumulation grid: terra.accum output tci grid: terra.tci [0.0] MFD flow direction [0.0] D8CUT=999999986991104.000000 [0.0] Memory size: 600.00M (629145600) bytes region size = 288.06K (294978) elts (466 rows x 633 cols) internal nodata value: -9999 reading raster map elevation.demto stream C:/DOCUME~1/HAMISH/LOCALS~1/Temp/STREAM_a02052
Note the Free Memory=-1 at the top of the file ...
Hamish
follow-up: 9 comment:8 by , 14 years ago
Replying to hamish:
Note the Free Memory=-1 at the top of the file ...
that's nothing. freeMem() is hardcoded to that on Windows.
in r41535 I've added a new debug message to devbr6.5 to show the values tested in that assert(). So we'll see what tomorrow's nightly build says about it.
On linux for elevation.dem@spearfish I get:
D3/3: nrows=466 ncols=633 stream_len()=294978
as expected.
Hamish
follow-ups: 10 12 comment:9 by , 14 years ago
Replying to hamish:
Replying to hamish:
Note the Free Memory=-1 at the top of the file ...
that's nothing. freeMem() is hardcoded to that on Windows.
in r41535 I've added a new debug message to devbr6.5 to show the values tested in that assert(). So we'll see what tomorrow's nightly build says about it.
On linux for elevation.dem@spearfish I get:
D3/3: nrows=466 ncols=633 stream_len()=294978as expected.
Hamish
grass65, self compiled in osgeo4w-stack, WinVista32 r41535
g.gisenv set=DEBUG=3
g.region -p -a rast=elevation@PERMANENT projection: 99 (Lambert Conformal Conic) zone: 0 datum: nad83 ellipsoid: a=6378137 es=0.006694380022900787 north: 228500 south: 215000 west: 630000 east: 645000 nsres: 10 ewres: 10 rows: 1350 cols: 1500 cells: 2025000
r.terraflow --verbose elevation=elevation@PERMANENT filled=elevflood1 direction=elevdir1 swatershed=elevsinkw1 accumulation=elevacc1 tci=elevtci1 memory=900 MFD flow direction D8CUT=999999986991104.000000 Memory size: 900.00M (943718400) bytes Memory manager registering memory in MM_WARN_ON_MEMORY_EXCEEDED mode. D2/3: G__read_Cell_head D2/3: G__read_Cell_head_array D3/3: region item: proj: 99 D3/3: region item: zone: 0 D3/3: region item: north: 228500 D3/3: region item: south: 215000 D3/3: region item: east: 645000 D3/3: region item: west: 630000 D3/3: region item: cols: 1500 D3/3: region item: rows: 1350 D3/3: region item: e-w resol: 10 D3/3: region item: n-s resol: 10 D3/3: region item: top: 1 D3/3: region item: bottom: 0 D3/3: region item: cols3: 1500 D3/3: region item: rows3: 1350 D3/3: region item: depths: 1 D3/3: region item: e-w resol3: 10 D3/3: region item: n-s resol3: 10 D3/3: region item: t-b resol: 1 D2/3: G__read_Cell_head D2/3: G__read_Cell_head_array D3/3: region item: proj: 99 D3/3: region item: zone: 0 D3/3: region item: north: 228500 D3/3: region item: south: 215000 D3/3: region item: east: 645000 D3/3: region item: west: 630000 D3/3: region item: cols: 1500 D3/3: region item: rows: 1350 D3/3: region item: e-w resol: 10 D3/3: region item: n-s resol: 10 D3/3: region item: format: -1 D3/3: region item: compressed: 1 cell elevation header compatible with region header Elevation stored as FLOAT (4B) Region size is 1350 x 1500 STREAM temporary files in C:/Users/syringia/AppData/Local/Temp (THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!) reading data from elevation to stream C:/Users/syringia/AppData/Local/Temp/STREAM_a06112: D2/3: G__read_Cell_head D2/3: G__read_Cell_head_array D3/3: region item: proj: 99 D3/3: region item: zone: 0 D3/3: region item: north: 228500 D3/3: region item: south: 215000 D3/3: region item: east: 645000 D3/3: region item: west: 630000 D3/3: region item: cols: 1500 D3/3: region item: rows: 1350 D3/3: region item: e-w resol: 10 D3/3: region item: n-s resol: 10 D3/3: region item: format: -1 D3/3: region item: compressed: 1 D3/3: create window mapping (1500 columns) D3/3: nrows=1350 ncols=1500 stream_len()=0 '''Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 146''' This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information.
stats.out: Free Memory=-1 [0.0] Thu Mar 25 20:26:58 2010 Command Line: r.terraflow --verbose elevation=elevation@PERMANENT filled=elevflood1 direction=elevdir1 swatershed=elevsinkw1 accumulation=elevacc1 tci=elevtci1 memory=900 input elevation grid: elevation output (flooded) elevations grid: elevflood1 output directions grid: elevdir1 output sinkwatershed grid: elevsinkw1 output accumulation grid: elevacc1 output tci grid: elevtci1 [0.0] MFD flow direction [0.0] D8CUT=999999986991104.000000 [0.0] Memory size: 900.00M (943718400) bytes region size = 1.93M (2025000) elts (1350 rows x 1500 cols) internal nodata value: -9999 reading raster map elevationto stream C:/Users/syringia/AppData/Local/Temp/STREAM_a06112
Helmut
follow-up: 11 comment:10 by , 14 years ago
Replying to hellik:
Replying to hamish:
Replying to hamish:
Note the Free Memory=-1 at the top of the file ...
that's nothing. freeMem() is hardcoded to that on Windows.
in r41535 I've added a new debug message to devbr6.5 to show the values tested in that assert(). So we'll see what tomorrow's nightly build says about it.
On linux for elevation.dem@spearfish I get:
D3/3: nrows=466 ncols=633 stream_len()=294978as expected.
Hamish
grass65, self compiled in osgeo4w-stack, WinVista32 r41535
g.gisenv set=DEBUG=3
now it's
Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 146
Helmut
comment:11 by , 14 years ago
Hamish:
in r41535 I've added a new debug message to devbr6.5 to show the values tested in that assert().
Replying to hellik:
now it's
Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 146
yes, because I added two lines to the source code so now it fails two lines later.
turn on debug messages by running g.gisenv set="DEBUG=3".
(set that back to 0 to turn them off again once you are done)
Hamish
follow-up: 13 comment:12 by , 14 years ago
Replying to hellik:
D3/3: nrows=1350 ncols=1500 stream_len()=0
Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 146
(assert() complains because the two sides are not the same)
ok, so str->stream_len() is not getting set correctly.
I wonder if this works:
/* create output stream */ AMI_STREAM<T>* str = new AMI_STREAM<T>();
stream_len() is defined in include/iostream/ami_stream.h (number of items in each stream)
more debug messages now turned on in that r41548 (6.5svn) ...
hmmm I get "Free Memory=-1" in stats.out on 32bit linux too.
Hamish
comment:13 by , 14 years ago
Replying to hamish:
[...] more debug messages now turned on in that r41548 (6.5svn) ...
r41567, self compiled in the osgeo4w/
g.gisenv set=DEBUG=3
r.terraflow --verbose elevation=elevation@PERMANENT filled=elevflood direction=elevdir swatershed=elevsinkwater accumulation=elevflowacc tci=elevtci memory=900 D2/3: G__read_Cell_head D2/3: G__read_Cell_head_array D3/3: region item: proj: 99 D3/3: region item: zone: 0 D3/3: region item: north: 228500 D3/3: region item: south: 215000 D3/3: region item: east: 645000 D3/3: region item: west: 630000 D3/3: region item: cols: 1500 D3/3: region item: rows: 1350 D3/3: region item: e-w resol: 10 D3/3: region item: n-s resol: 10 D3/3: region item: top: 1 D3/3: region item: bottom: 0 D3/3: region item: cols3: 1500 D3/3: region item: rows3: 1350 D3/3: region item: depths: 1 D3/3: region item: e-w resol3: 10 D3/3: region item: n-s resol3: 10 D3/3: region item: t-b resol: 1 D2/3: G__read_Cell_head D2/3: G__read_Cell_head_array D3/3: region item: proj: 99 D3/3: region item: zone: 0 D3/3: region item: north: 228500 D3/3: region item: south: 215000 D3/3: region item: east: 645000 D3/3: region item: west: 630000 D3/3: region item: cols: 1500 D3/3: region item: rows: 1350 D3/3: region item: e-w resol: 10 D3/3: region item: n-s resol: 10 D3/3: region item: format: -1 D3/3: region item: compressed: 1 cell elevation header compatible with region header Elevation stored as FLOAT (4B) Region size is 1350 x 1500 STREAM temporary files in C:/Users/syringia/AppData/Local/Temp (THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!) Reading data from <elevation> to stream <C:/Users/syringia/AppData/Local/Temp/STREAM_a07132> D2/3: G__read_Cell_head D2/3: G__read_Cell_head_array D3/3: region item: proj: 99 D3/3: region item: zone: 0 D3/3: region item: north: 228500 D3/3: region item: south: 215000 D3/3: region item: east: 645000 D3/3: region item: west: 630000 D3/3: region item: cols: 1500 D3/3: region item: rows: 1350 D3/3: region item: e-w resol: 10 D3/3: region item: n-s resol: 10 D3/3: region item: format: -1 D3/3: region item: compressed: 1 D3/3: create window mapping (1500 columns) MFD flow direction D8CUT=999999986991104.000000 Memory size: 900.00M (943718400) bytes Memory manager registering memory in MM_WARN_ON_MEMORY_EXCEEDED mode. C:/Users/syringia/AppData/Local/Temp/STREAM_a07132: length = 17179869184 sizeof(T)=1 D3/3: nrows=1350 ncols=1500 stream_len()=0 C:/Users/syringia/AppData/Local/Temp/STREAM_a07132: length = 17179869184 sizeof(T)=0 Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 146 This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information.
Free Memory=-1 [0.0] Sat Mar 27 03:38:21 2010 Command Line: r.terraflow --verbose elevation=elevation@PERMANENT filled=elevflood direction=elevdir swatershed=elevsinkwater accumulation=elevflowacc tci=elevtci memory=900 input elevation grid: elevation output (flooded) elevations grid: elevflood output directions grid: elevdir output sinkwatershed grid: elevsinkwater output accumulation grid: elevflowacc output tci grid: elevtci [0.0] MFD flow direction [0.0] D8CUT=999999986991104.000000 [0.0] Memory size: 900.00M (943718400) bytes region size = 1.93M (2025000) elts (1350 rows x 1500 cols) internal nodata value: -9999 Reading raster map <elevation> to stream <C:/Users/syringia/AppData/Local/Temp/STREAM_a07132>.
g.gisenv set=DEBUG=0
r.terraflow --verbose elevation=elevation@PERMANENT filled=elevflood direction=elevdir swatershed=elevsinkwater accumulation=elevflowacc tci=elevtci memory=900 MFD flow direction D8CUT=999999986991104.000000 Memory size: 900.00M (943718400) bytes Memory manager registering memory in MM_WARN_ON_MEMORY_EXCEEDED mode. cell elevation header compatible with region header Elevation stored as FLOAT (4B) Region size is 1350 x 1500 STREAM temporary files in C:/Users/syringia/AppData/Local/Temp (THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!) Reading data from <elevation> to stream <C:/Users/syringia/AppData/Local/Temp/STREAM_a01828> C:/Users/syringia/AppData/Local/Temp/STREAM_a01828: length = 17179869184 sizeof(T)=1 C:/Users/syringia/AppData/Local/Temp/STREAM_a01828: length = 17179869184 sizeof(T)=1 Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 146 This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information.
Free Memory=-1 [0.0] Sat Mar 27 03:40:39 2010 Command Line: r.terraflow --verbose elevation=elevation@PERMANENT filled=elevflood direction=elevdir swatershed=elevsinkwater accumulation=elevflowacc tci=elevtci memory=900 input elevation grid: elevation output (flooded) elevations grid: elevflood output directions grid: elevdir output sinkwatershed grid: elevsinkwater output accumulation grid: elevflowacc output tci grid: elevtci [0.0] MFD flow direction [0.0] D8CUT=999999986991104.000000 [0.0] Memory size: 900.00M (943718400) bytes region size = 1.93M (2025000) elts (1350 rows x 1500 cols) internal nodata value: -9999 Reading raster map <elevation> to stream <C:/Users/syringia/AppData/Local/Temp/STREAM_a01828>.
best regards Helmut
follow-ups: 15 16 22 comment:14 by , 14 years ago
are you using the elevation raster map from the NC dataset?
Helmut wrote:
Reading data from <elevation> to stream <C:/Users/syringia/AppData/Local/Temp/STREAM_a01828> C:/Users/syringia/AppData/Local/Temp/STREAM_a01828: length = 17179869184 sizeof(T)=1 C:/Users/syringia/AppData/Local/Temp/STREAM_a01828: length = 17179869184 sizeof(T)=1 Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 146
so the value of the buf.st_size variable is filled with garbage. and sizeof(T) is different from what I get on linux.
for me on 32bit linux I get this output for that line (spearfish's elevation.dem)
evelation.dem: rows: 466 cols: 633 ... MFD flow direction D8CUT=999999986991104.000000 Memory size: 300.00M (314572800) bytes Memory manager registering memory in MM_WARN_ON_MEMORY_EXCEEDED mode. Reading data from <elevation.dem> to stream </var/tmp//STREAM_qylbyz> 100% /var/tmp//STREAM_qylbyz: length = 1179912 sizeof(T)=4 /var/tmp//STREAM_qylbyz: length = 1179912 sizeof(T)=4 total elements=294978, nodata elements=2661 largest temporary files: FILL: 15.75M (16518768) [294978 elements, 56B each] FLOW: 22.30M (23385360) [292317 elements, 80B each] Will need at least 44.60M (46770720) space available in /var/tmp/ ------------------------------ COMPUTING FLOW DIRECTIONS ...
is stat() on Windows the same as it is on UNIX?
on my version of linux .st_size should be type off_t: "total size, in bytes", and stat() should return 0 on success and -1 on failure.
is %lld the right printf() setting for off_t?
I'm not sure how big <class T> should be, but probably bigger than 1 byte.. ?
Hamish
comment:15 by , 14 years ago
Replying to hamish:
are you using the elevation raster map from the NC dataset?
yes, it is the NC-dataset.
Helmut
comment:16 by , 14 years ago
Replying to hamish:
for me on 32bit linux I get this output for that line (spearfish's elevation.dem)
evelation.dem: rows: 466 cols: 633 ... MFD flow direction D8CUT=999999986991104.000000 Memory size: 300.00M (314572800) bytes Memory manager registering memory in MM_WARN_ON_MEMORY_EXCEEDED mode. Reading data from <elevation.dem> to stream </var/tmp//STREAM_qylbyz> 100% /var/tmp//STREAM_qylbyz: length = 1179912 sizeof(T)=4 /var/tmp//STREAM_qylbyz: length = 1179912 sizeof(T)=4 total elements=294978, nodata elements=2661 largest temporary files: FILL: 15.75M (16518768) [294978 elements, 56B each] FLOW: 22.30M (23385360) [292317 elements, 80B each] Will need at least 44.60M (46770720) space available in /var/tmp/ ------------------------------ COMPUTING FLOW DIRECTIONS ...
and with spearfish's elevation.dem on WinVista32
r.terraflow --verbose elevation=elevation.dem@PERMANENT filled=floodedelevdem direction=flowdirrast swatershed=sinkwatershrast accumulation=flowaccrast tci=topoconvrast memory=900 ... MFD flow direction D8CUT=999999986991104.000000 Memory size: 900.00M (943718400) bytes Memory manager registering memory in MM_WARN_ON_MEMORY_EXCEEDED mode. C:/Users/syringia/AppData/Local/Temp/STREAM_a03800: length = 17179869184 sizeof(T)=1 D3/3: nrows=466 ncols=633 stream_len()=0 C:/Users/syringia/AppData/Local/Temp/STREAM_a03800: length = 17179869184 sizeof(T)=0 Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 146 ...
Helmut
follow-up: 18 comment:17 by , 14 years ago
could you try r.in.bin? it also uses stat()'s .st_size.
(I notice it uses int,%d for FILE_SIZE too, so will not be LFS-safe)
Hamish
comment:18 by , 14 years ago
Replying to hamish:
could you try r.in.bin? it also uses stat()'s .st_size.
I'm not at my laptop at the moment, so I've tried WinGRASS-6.5.SVN-r41581-1-Setup.exe on a WinVista32-box with the nc-dataset:
r.out.bin -h --verbose input=elevation@PERMANENT output=J:\wd\grassoutbin\elevroutbin3.bin
r.in.bin -h input=J:\wd\grassoutbin\elevroutbin3.bin output=elevrinbin bytes=4 D2/3: G__read_Cell_head D2/3: G__read_Cell_head_array D3/3: region item: proj: 99 D3/3: region item: zone: 0 D3/3: region item: north: 228500 D3/3: region item: south: 215000 D3/3: region item: east: 645000 D3/3: region item: west: 630000 D3/3: region item: cols: 1500 D3/3: region item: rows: 1350 D3/3: region item: e-w resol: 10 D3/3: region item: n-s resol: 10 D3/3: region item: top: 1 D3/3: region item: bottom: 0 D3/3: region item: cols3: 1500 D3/3: region item: rows3: 1350 D3/3: region item: depths: 1 D3/3: region item: e-w resol3: 10 D3/3: region item: n-s resol3: 10 D3/3: region item: t-b resol: 1 D1/3: Creating support files for elevrinbin D1/3: close elevrinbin compressed D3/3: G_recreate_command()
import with r.in.bin is working
+----------------------------------------------------------------------------+ | Layer: elevrinbin@user1 Date: Sun Mar 28 20:14:13 2010 | | Mapset: user1 Login of Creator: normal | | Location: nc_spm_07 | | DataBase: J:\gisdata | | Title: ( elevrinbin ) | | Timestamp: none | |----------------------------------------------------------------------------| | | | Type of Map: raster Number of Categories: 1125930098 | | Data Type: CELL | | Rows: 1350 | | Columns: 1500 | | Total Cells: 2025000 | | Projection: Lambert Conformal Conic | | N: 228500 S: 215000 Res: 10 | | E: 645000 W: 630000 Res: 10 | | Range of data: min = 1113477295 max = 1125930098 | | | | Data Description: | | generated by r.in.bin | | | | Comments: | | r.in.bin -h input="J:\wd\grassoutbin\elevroutbin3.bin" output="elevr\ | | inbin" bytes=4 | | | +----------------------------------------------------------------------------+
Helmut
comment:19 by , 14 years ago
| Priority: | major → critical |
|---|
comment:20 by , 14 years ago
| Summary: | r.terraflow fails to create temp file on Windows when launched from wxgui → r.terraflow fails to stat() stream file on Windows |
|---|
comment:21 by , 14 years ago
WinGRASS-6.4.SVN-r41749-1 running Vista with DEBUG=3:
GRASS 6.4.0svn (spearfish60)> r.terraflow --verbose elevation=elevation.dem@PERM
ANENT filled=elevflood direction=elevdir swatershed=elevsinkwater accumulation=e
levflowacc tci=elevtci memory=900
D2/3: G__read_Cell_head
D2/3: G__read_Cell_head_array
D3/3: region item: proj: 1
D3/3: region item: zone: 13
D3/3: region item: north: 4928010
D3/3: region item: south: 4913700
D3/3: region item: east: 609000
D3/3: region item: west: 589980
D3/3: region item: cols: 634
D3/3: region item: rows: 477
D3/3: region item: e-w resol: 30
D3/3: region item: n-s resol: 30
D3/3: region item: top: 1
D3/3: region item: bottom: 0
D3/3: region item: cols3: 634
D3/3: region item: rows3: 477
D3/3: region item: depths: 1
D3/3: region item: e-w resol3: 30
D3/3: region item: n-s resol3: 30
D3/3: region item: t-b resol: 1
D2/3: G__read_Cell_head
D2/3: G__read_Cell_head_array
D3/3: region item: proj: 1
D3/3: region item: zone: 13
D3/3: region item: north: 4928000
D3/3: region item: south: 4914020
D3/3: region item: east: 609000
D3/3: region item: west: 590010
D3/3: region item: cols: 633
D3/3: region item: rows: 466
D3/3: region item: e-w resol: 30
D3/3: region item: n-s resol: 30
D3/3: region item: format: 1
D3/3: region item: compressed: 1
cell elevation.dem header compatible with region header
Elevation stored as FLOAT (4B)
BR╬DIN┬JUMS:raster elevation.dem is of type CELL_TYPE --you should use
r.terraflow.short
Region size is 477 x 634
STREAM temporary files in C:/Users/Maris/AppData/Local/Temp (THESE INTERMEDIATE
STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO
SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!)
stats.out: Permission denied
comment:22 by , 14 years ago
Replying to hamish:
is stat() on Windows the same as it is on UNIX?
Sort of. However: if the file is open, it reports the size as zero bytes (or possibly the original size if the file was opened for update without truncation). Once the file is closed, it reports the correct size. Also, fstat() works correctly on open files.
Test case:
$ cat test.c
#include <stdio.h>
#include <sys/stat.h>
int main(void)
{
char buf[1024];
struct stat st;
FILE *fp;
int i;
fp = fopen("test.dat", "wb");
for (i = 0; i < 64; i++)
fwrite(buf, 1, sizeof(buf), fp);
fflush(fp);
stat("test.dat", &st);
printf("%d\n", (int)st.st_size);
fstat(fileno(fp), &st);
printf("%d\n", (int)st.st_size);
fclose(fp);
stat("test.dat", &st);
printf("%d\n", (int)st.st_size);
return 0;
}
$ gcc -Wall test.c
$ ./a.exe
0
65536
65536
A more portable way to determine the length of a file is to use fseek(SEEK_END), e.g.:
long save = ftell(fp); fseek(fp, 0, SEEK_END); long size = ftell(fp); fseek(fp, save, SEEK_SET); return size;
comment:23 by , 14 years ago
| Keywords: | r.terraflow added; wxgui terraflow removed |
|---|
- removed wxgui keyword, it also happens from the msys prompt.
Maris: fyi extra debug messages only added in 6.5svn.
STREAM temporary files in C:/Users/Maris/AppData/Local/Temp ... stats.out: Permission denied
stats.out tries to write to the current working directory, try running
cd C:/Users/Maris/AppData/Local/Temp
or your home dir before starting the program. (or change the stats= filename option)
(IIUC stats.out file just contains extra debug messages; STREAM is a copy of the raster map stored on disk for random access)
testing with the latest 6.5 nightly wingrass build (r41761), it looks like the stat() on an open file is in fact the problem:
#spearfish g.region rast=elevation.dem r.terraflow ele=elevation.dem filled=rterra.fill dir=rterra.dir \ swat=rterra.swat acc=rterra.acc tci=rterra.tci mem=600 ... D3/3: create window mapping (633 columns) C:/DOCUME~1/OPG/LOCALS~1/Temp/STREAM_a03788: length = 0 sizeof(T)=4 D3/3: nrows=466 ncols=633 stream_len()=0 C:/DOCUME~1/OPG/LOCALS~1/Temp/STREAM_a03788: length = 0 sizeof(T)=4 Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 146 # file size on disk: wc -c C:/DOCUME~1/OPG/LOCALS~1/Temp/STREAM_a03788 1179912
633*466 = 294978 cells 1179912 / 294978 = 4 bytes per cell
Hamish
follow-up: 25 comment:24 by , 14 years ago
In comment:12, Hamish wrote:
ok, so str->stream_len() is not getting set correctly.
...
stream_len() is defined in include/iostream/ami_stream.h (number of items in each stream)
In comment:22, Glynn wrote:
A more portable way to determine the length of a file is to use fseek(SEEK_END), e.g.:
long save = ftell(fp); fseek(fp, 0, SEEK_END); long size = ftell(fp); fseek(fp, save, SEEK_SET); return size;
ok, thanks. how's this:
(patch vs. 6.5svn)
Index: include/iostream/ami_stream.h
===================================================================
--- include/iostream/ami_stream.h (revision 41749)
+++ include/iostream/ami_stream.h (working copy)
@@ -351,21 +351,32 @@
template<class T>
off_t AMI_STREAM<T>::stream_len(void) {
+ long posn_save, st_size;
+
fflush(fp);
- struct stat buf;
- if (stat(path, &buf) == -1) {
- perror("AMI_STREAM::stream_len(): fstat failed ");
- perror(path);
- assert(0);
- exit(1);
+ posn_save = ftell(fp);
+ if(posn_save == -1) {
+ perror("ERROR: AMI_STREAM::stream_len(): ftell(fp) failed ");
+ perror(path);
+ exit(1);
}
+ fseek(fp, 0, SEEK_END);
+ st_size = ftell(fp);
+ if(st_size == -1) {
+ perror("ERROR: AMI_STREAM::stream_len(): ftell[SEEK_END] failed ");
+ perror(path);
+ exit(1);
+ }
+
+ fseek(fp, posn_save, SEEK_SET);
+
//DEBUG:
fprintf(stderr, "%s: length = %lld sizeof(T)=%d\n",
- path, buf.st_size, sizeof(T));
+ path, st_size, sizeof(T));
- return (buf.st_size / sizeof(T));
+ return (st_size / sizeof(T));
};
Hamish
follow-up: 26 comment:25 by , 14 years ago
Replying to hamish:
In comment:12, Hamish wrote:
ok, so str->stream_len() is not getting set correctly.
...
stream_len() is defined in include/iostream/ami_stream.h (number of items in each stream)
In comment:22, Glynn wrote:
A more portable way to determine the length of a file is to use fseek(SEEK_END), e.g.:
long save = ftell(fp); fseek(fp, 0, SEEK_END); long size = ftell(fp); fseek(fp, save, SEEK_SET); return size;ok, thanks. how's this:
(patch vs. 6.5svn)
Index: include/iostream/ami_stream.h =================================================================== --- include/iostream/ami_stream.h (revision 41749) +++ include/iostream/ami_stream.h (working copy) @@ -351,21 +351,32 @@ template<class T> off_t AMI_STREAM<T>::stream_len(void) { + long posn_save, st_size; + fflush(fp); - struct stat buf; - if (stat(path, &buf) == -1) { - perror("AMI_STREAM::stream_len(): fstat failed "); - perror(path); - assert(0); - exit(1); + posn_save = ftell(fp); + if(posn_save == -1) { + perror("ERROR: AMI_STREAM::stream_len(): ftell(fp) failed "); + perror(path); + exit(1); } + fseek(fp, 0, SEEK_END); + st_size = ftell(fp); + if(st_size == -1) { + perror("ERROR: AMI_STREAM::stream_len(): ftell[SEEK_END] failed "); + perror(path); + exit(1); + } + + fseek(fp, posn_save, SEEK_SET); + //DEBUG: fprintf(stderr, "%s: length = %lld sizeof(T)=%d\n", - path, buf.st_size, sizeof(T)); + path, st_size, sizeof(T)); - return (buf.st_size / sizeof(T)); + return (st_size / sizeof(T)); };Hamish
fseek and ftell only work for 32-bit file offsets on 32-bit system as the offset for fseek and the return value of ftell is a long type. At least in linux, the proper return type is an off_t which can be set to 32 or 64 bits at compile time and fseeko and ftello use off_t types. A real fix should support 64 bit file offsets on 32 bit and 64-bit systems in Windows and Linux, but this gets a bit complicated. Is there a guaranteed portable 64-bit type in Grass? A portable 64-bit G_fseek might avoid future problems of this type.
comment:26 by , 14 years ago
Replying to adanner:
fseek and ftell only work for 32-bit file offsets on 32-bit system as the offset for fseek and the return value of ftell is a long type. At least in linux, the proper return type is an off_t which can be set to 32 or 64 bits at compile time and fseeko and ftello use off_t types. A real fix should support 64 bit file offsets on 32 bit and 64-bit systems in Windows and Linux, but this gets a bit complicated. Is there a guaranteed portable 64-bit type in Grass? A portable 64-bit G_fseek might avoid future problems of this type.
In 7.0, G_fseek() and G_ftell() use off_t and use fseeko/ftello where available. But Windows doesn't have an equivalent of _FILE_OFFSET_BITS; you have to explicitly use off64_t, lseek64(), fseeko64() etc.
Also, "long" is still only 32 bits on Win64.
Finally, C89 doesn't guarantee the existence of a 64-bit type; "long long" is C99, which isn't widely supported. gcc and MSVC support "most" of C99, but not all of it. Which means that you can either code for C89, code for whichever subset of C99 is supported by some specific set of compilers, or code for C99 and be unable to find a compiler which will accept it.
GRASS aims to compile with any sane C89 compiler (excluding technically conforming yet obscure cases, e.g. 60-bit "int", ones-complement representation, etc). Anything which uses extensions should have a fall-back.
follow-up: 28 comment:27 by , 14 years ago
as LFS support for WinGrass is not a goer for 6.4.0, how about if we #ifdef the ftell() patch for MINGW32 in the release branch, and leave the UNIXes running the non-portable stat() version.
Lack of large file support especially sucks for r.terraflow which likes to have very large temp files, but it's better than the current situation on WinGrass which is that the module won't run at all.
Maybe we can figure out something better to include with 6.4.1.
Hamish
by , 14 years ago
| Attachment: | ami_stream_size65.diff added |
|---|
include/iostream/ami_stream.h vs. grass65 for WinGrass support
follow-ups: 29 52 comment:28 by , 14 years ago
Replying to hamish:
as LFS support for WinGrass is not a goer for 6.4.0, how about if we #ifdef the ftell() patch for MINGW32 in the release branch, and leave the UNIXes running the non-portable stat() version.
Lack of large file support especially sucks for r.terraflow which likes to have very large temp files, but it's better than the current situation on WinGrass which is that the module won't run at all.
Maybe we can figure out something better to include with 6.4.1.
patch attached for consideration. debug messages should be removed after testing.
Hamish
comment:29 by , 14 years ago
Replying to hamish:
Replying to hamish:
as LFS support for WinGrass is not a goer for 6.4.0, how about if we #ifdef the ftell() patch for MINGW32 in the release branch, and leave the UNIXes running the non-portable stat() version.
Lack of large file support especially sucks for r.terraflow which likes to have very large temp files, but it's better than the current situation on WinGrass which is that the module won't run at all.
Maybe we can figure out something better to include with 6.4.1.
patch attached for consideration. debug messages should be removed after testing.
Hamish
patch tested on a self compiled grass65 on WinVista 32. shortened error log attached.
r.terraflow --verbose elevation=elevation@PERMANENT filled=filflood direction=flowdir swatershed=sinkwatershed accumulation=flowacc tci=tpoconv memory=800 MFD flow direction D8CUT=999999986991104.000000 Memory size: 800.00M (838860800) bytes Memory manager registering memory in MM_WARN_ON_MEMORY_EXCEEDED mode. cell elevation header compatible with region header Elevation stored as FLOAT (4B) Region size is 1350 x 1500 STREAM temporary files in C:/Users/syringia/AppData/Local/Temp (THESE INTERMEDIATE STREAMS WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!) Reading data from <elevation> to stream <C:/Users/syringia/AppData/Local/Temp/STREAM_a06204> [...] Assertion failed: ae == AMI_ERROR_NO_ERROR, file filldepr.cc, line 136 This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information. UnionFind::makeSet: reallocate double 2000 UnionFind::makeSet: reallocate double 4000
follow-up: 31 comment:30 by , 14 years ago
Tested with WinGRASS-6.4.SVN-r42668-1 on Vista. It's better now, as in CLI it fails with same message as in wxGUI (stats.out: Permission denied). No more assertion errors. Horray! Still r.terraflow is not working for windows users.
comment:31 by , 14 years ago
| Milestone: | 6.4.0 → 6.4.1 |
|---|---|
| Priority: | critical → major |
Replying to marisn:
Tested with WinGRASS-6.4.SVN-r42668-1 on Vista. It's better now, as in CLI it fails with same message as in wxGUI (stats.out: Permission denied). No more assertion errors. Horray! Still r.terraflow is not working for windows users.
Still getting the same assertion error on 6.4 r42714:
Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 144
About stats.out: Permission denied, stats.out is not written to STREAM_DIR because everything in STREAM_DIR is deleted after successful completion, stats.out is written to the current working directory and remains there for inspection. On my test Windows system, the current directory is /c/Dokumente und Einstellungen/Markus and I can write to it.
r.terraflow is on windows crippled by missing LFS in GRASS. Its strength is its performance with massive datasets, but these can't be processed on windows because of the 2GB file size limit. Grass with LFS on windows will likely only be available in grass 7, planned for next month, but not in grass 6. r.terraflow is working fine on Linux and its functionality is covered by other modules, therefore it is IMHO not critical to have a working r.terraflow on windows. It would be very nice though.
Considering this as well as previous comments, I think it's more realistic to get it fixed in 6.4.1 or 6.5, but under windows it will be really useful only in grass 7.0 with LFS.
Markus M
follow-up: 35 comment:32 by , 14 years ago
| Milestone: | 6.4.1 → 6.4.0 |
|---|---|
| Priority: | major → critical |
putting this back on the table for 6.4.0, without hesitation. see plan in comment:27.
I don't think we are far off having this working -- it may be working in 6.5 already and just need a backport.
r.terraflow (and v.buffer) are flagship modules for GRASS and frankly it's an embarrassment not to have them working properly (even if their full glory is crippled as in the case of LFS + r.terraflow).
Markus Metz wrote:
Still getting the same assertion error on 6.4 r42714:
well, yes, because the stat() work-around has not been backported to the 6.4 branch pending testing in 6.5. Please test it in 6.5 (run from a dir with write permissions :). I hope that stats.out error is not fatal to the process.
regards, Hamish
comment:33 by , 14 years ago
sorry, it seems I never applied the patch for include/iostream/ami_stream.h in this ticket to 6.5svn as intended. I will do so ASAP so it can be tested from the daily binary snapshot build this weekend.
Hamish
follow-up: 36 comment:35 by , 14 years ago
Replying to hamish:
putting this back on the table for 6.4.0, without hesitation. see plan in comment:27.
Hmph. The ticket is however not a blocker ;-)
I don't think we are far off having this working -- it may be working in 6.5 already and just need a backport.
r.terraflow (and v.buffer) are flagship modules for GRASS and frankly it's an embarrassment not to have them working properly (even if their full glory is crippled as in the case of LFS + r.terraflow).
Its flagship character comes from its ability to process very large datasets, but all its functionality including the ability to process large datasets is also covered by other modules. Without LFS, r.terraflow is limited to less than 27 million non-NULL cells, converting the module from a flagship to a dinghy. I am very much inclined to change milestone to 7.0 and set priority to normal.
Tested 6.5svn r42716 on XP, same error as in comment:29 in filldepr.cc. Once that is fixed, we might have a look at why r.terraflow refuses to run with output from r.volcano... No, seriously, r.terraflow and iostream is written for Linux, as also previous tickets have shown, no reason to expect that this is the last problem with Windows. Please leave it for 6.5 or 7.0.
Markus M
follow-up: 37 comment:36 by , 14 years ago
Replying to mmetz:
the ability to process large datasets is also covered by other modules.
I know that, but the (perhaps recently historic) way the modules are presented in the menus does not make that clear to users.
Tested 6.5svn r42716 on XP, same error as in comment:29 in filldepr.cc.
thanks. so the error is now:
Reading data from <elevation> to stream <C:/Users/syringia/AppData/Local/Temp/STREAM_a06204> [...] Assertion failed: ae == AMI_ERROR_NO_ERROR, file filldepr.cc, line 136 This application has requested the Runtime to terminate it in an unusual way.
from raster/r.terraflow/filldepr.cc
/*read next edge*/
ae = boundaryStr->read_item(&nextedge);
assert(ae == AMI_ERROR_NO_ERROR);
which calls include/iostream/mem_stream.h
template<class T>
AMI_err MEM_STREAM<T>::read_item(T **elt) {
assert(data);
if(curr == dataend) {
return AMI_ERROR_END_OF_STREAM;
}
*elt = curr;
curr++;
return AMI_ERROR_NO_ERROR;
};
so it seems that it is prematurely reaching the end of the stream?
Once that is fixed, we might have a look at why r.terraflow refuses to run with output from r.volcano...
does it not?? how about a sum=r.volcano+r.surf.fractal surface? perhaps because there are no basins except the small one in the crater in that fake landscape? is there a ticket for this?
No, seriously, r.terraflow and iostream is written for Linux, as also previous tickets have shown, no reason to expect that this is the last problem with Windows. Please leave it for 6.5 or 7.0.
we could easily say the same for all of GRASS. some parts will be easier to port than others, but if we didn't care about MS-Windows at all we could have released 6.4 mid-last year. in this case it sucks, is not known to be fundamentally deep, and I'd like to try and fix it. even if it is crippled due to LFS (you might say 32bit XP is too with its 3.2gb RAM limit), at least people can still work on tutorials/examples and learn how to use it for when they graduate to a gruntier system.
in pursuit of a technical fix I will add debug messages to mem_stream.h. (r42731)
regards, Hamish
follow-ups: 38 39 comment:37 by , 14 years ago
Replying to hamish:
Replying to mmetz:
the ability to process large datasets is also covered by other modules.
I know that, but the (perhaps recently historic) way the modules are presented in the menus does not make that clear to users.
Well, then we have to change the way modules are presented in the menus.
from raster/r.terraflow/filldepr.cc
/*read next edge*/ ae = boundaryStr->read_item(&nextedge); assert(ae == AMI_ERROR_NO_ERROR);which calls include/iostream/mem_stream.h
template<class T> AMI_err MEM_STREAM<T>::read_item(T **elt) { assert(data); if(curr == dataend) { return AMI_ERROR_END_OF_STREAM; } *elt = curr; curr++; return AMI_ERROR_NO_ERROR; };so it seems that it is prematurely reaching the end of the stream?
I never thought I would agree to fix depression filling... The mingw32 version of stream_len() does not work 100%, workaround in r42734 and r42735. Worksforme now on Windows. On Linux, no difference in results. Can I now put warning messages in r.terraflow that its hydrological algorithms from the late 80ies and early 90ies do not produce very accurate results (its emphasis is on the computational complexity of the algorithms, rather than on modeling realistic flow)?
r.terraflow and iostream is written for Linux, as also previous tickets have shown, no reason to expect that this is the last problem with Windows. Please leave it for 6.5 or 7.0.
we could easily say the same for all of GRASS.
Most parts of grass are not giving problems with mingw32, maybe because these adhere to C89? C++ doesn't, so problems might be expected here.
[snip] even if it is crippled due to LFS (you might say 32bit XP is too with its 3.2gb RAM limit),
You might say that mingw32 is the problem (apart from the msys problems). IIUC, stuff compiled on e.g. Windows7 64bit with mingw32 is still 32bit and has all the 32bit limits, even if the OS is 64bit. Plus other annoyances like sizeof(long) = 4 on Windows 64bit.
in pursuit of a technical fix I will add debug messages to mem_stream.h. (r42731)
IMHO, the debug messages can be removed now, then the fix can be backported to 6.4.1 ;-)
Markus M
follow-up: 40 comment:38 by , 14 years ago
Replying to mmetz:
Well, then we have to change the way modules are presented in the menus.
[aside] can we have a README.howto file in the gui/wxpython/xml/ dir? Every time I edit menudata.xml it seems to get overwritten some time later by an automated update which I don't see documented anywhere. It sort of kills my motivation to work on improving the menus anymore. tx.
so it seems that it is prematurely reaching the end of the stream?
I never thought I would agree to fix depression filling...
:)
The mingw32 version of stream_len() does not work 100%,
could you explain for us humble c++ students?
workaround in r42734 and r42735. Worksforme now on Windows. On Linux, no difference in results.
excellent!
Can I now put warning messages in r.terraflow that its hydrological algorithms from the late 80ies and early 90ies do not produce very accurate results (its emphasis is on the computational complexity of the algorithms, rather than on modeling realistic flow)?
I'd rather you got together with Andrew and/or Laura and co-authored a journal article racingW comparing the two with the current offerings from the other major proprietary GISs. :-) [seriously]
http://www.cs.duke.edu/geo*/terraflow/speedup.html
Are the core r.watershed algorithms by Ehlschlaeger any more modern? or have you now replaced and upgraded them to 'state of the art'? (whatever that means)
r.terraflow and iostream is written for Linux,
well I knew a version for Arc existed, so thought it possible,
http://wwwmath1.uni-muenster.de:8010/u/jan/TerraFlowExtension/
IMHO, the debug messages can be removed now, then the fix can be backported to 6.4.1 ;-)
I removed the noisiest one, will do the rest after testing tomorrow's autobuild. (and of course merge with trunk too)
If it doesn't change results on UNIX it may as well go into 6.4.0. It doesn't matter how risky it is on Windows as it can't be worse than before. If you feel there's a risk for breakage on UNIX from this we could wait til 6.4.1 for the backport & disable building it in 6.4.0, but we should perhaps look over the menus in that case to make sure that the emphasis and descriptions are all up to date.
cheers, Hamish
follow-up: 41 comment:39 by , 14 years ago
Replying to mmetz: [...]
IMHO, the debug messages can be removed now, then the fix can be backported to 6.4.1 ;-)
Markus M
I've tried a self compiled Grass65 (r42743) in the osgeo4w-tree on my WinVista32-box with nc-sample-dataset started from the gui.
r.terraflow elevation=elevation filled=ff direction=dd accumulation=aa tci=tt memory=800
r.terraflow starts and seems to calculate, but after a while, the gui freezes at the following message in the command output
[...] assining preliminary directions finding flat areas (plateaus and depressions)
Helmut
follow-up: 44 comment:40 by , 14 years ago
Replying to hamish:
[aside] can we have a README.howto file in the gui/wxpython/xml/ dir? Every time I edit menudata.xml it seems to get overwritten some time later by an automated update which I don't see documented anywhere. It sort of kills my motivation to work on improving the menus anymore. tx.
menudata.xml is maintained by source:grass/trunk/gui/wxpython/support/update_menudata.py which touches only description and keywords (based on command usage desc). Other items should be untouched, if not please report it as a bug.
Martin
follow-up: 42 comment:41 by , 14 years ago
Replying to hellik:
I've tried a self compiled Grass65 (r42743) in the osgeo4w-tree on my WinVista32-box with nc-sample-dataset started from the gui.
r.terraflow elevation=elevation filled=ff direction=dd accumulation=aa tci=tt memory=800r.terraflow starts and seems to calculate, but after a while, the gui freezes
Try less memory, e.g. default 300 (are there currently 800MB free on your vista 32bit box?). Or wait for it complete...
Markus M
follow-up: 43 comment:42 by , 14 years ago
Replying to mmetz:
Replying to hellik:
I've tried a self compiled Grass65 (r42743) in the osgeo4w-tree on my WinVista32-box with nc-sample-dataset started from the gui.
r.terraflow elevation=elevation filled=ff direction=dd accumulation=aa tci=tt memory=800r.terraflow starts and seems to calculate, but after a while, the gui freezes
Try less memory, e.g. default 300 (are there currently 800MB free on your vista 32bit box?). Or wait for it complete...
Markus M
the whole wx-gui freezes also with 300mb.
starting r.terraflow from the msys-command-line, it's finishing all calculations.
Helmut
follow-up: 45 comment:43 by , 14 years ago
Replying to hellik:
Replying to mmetz:
Replying to hellik:
I've tried a self compiled Grass65 (r42743) in the osgeo4w-tree on my WinVista32-box with nc-sample-dataset started from the gui.
r.terraflow elevation=elevation filled=ff direction=dd accumulation=aa tci=tt memory=800r.terraflow starts and seems to calculate, but after a while, the gui freezes
Try less memory, e.g. default 300 (are there currently 800MB free on your vista 32bit box?). Or wait for it complete...
Markus M
the whole wx-gui freezes also with 300mb.
starting r.terraflow from the msys-command-line, it's finishing all calculations.
Hmm. Good to hear that it works on CLI. In wxGUI, it might become better once the debug messages have been removed, I had to clear output regularly to get it going again. Too much for the wxGUI output window?
Markus M
comment:44 by , 14 years ago
Replying to martinl:
menudata.xml is maintained by source:grass/trunk/gui/wxpython/support/update_menudata.py which touches only description and keywords (based on command usage desc).
thanks, added some notes as
how to get that into the programmer's manual? doxygen in header comments of update_menudata.py?
Hamish
comment:45 by , 14 years ago
Replying to mmetz:
Hmm. Good to hear that it works on CLI. In wxGUI, it might become better once the debug messages have been removed, I had to clear output regularly to get it going again. Too much for the wxGUI output window?
probably. You'll get similar GUI lockup failures if you try and run vector modules while DEBUG=5. It can't handle the incoming message volume. Tcl/Tk had the same problem.
Hamish
follow-up: 47 comment:46 by , 14 years ago
Hi, just tested 6.5svn r42750 nightly build on XP. And I'm back to the original error message & exit code 3:
#spearfish g.region rast=elevation.10m r.terraflow elev=elevation.10m ... Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 144
the left over .../Temp/STREAM_a02688 file is 10,619,208 bytes.
- (and .../Temp/ is full of 100+ old tmp*.ppm files at a MB each... :-/ new ones even after exiting the wxGUI cleanly. now that I look, it's the same in /tmp/ in linux)
D2/3: G__read_Cell_head D2/3: G__read_Cell_head_array D3/3: region item: proj: 1 D3/3: region item: zone: 13 D3/3: region item: north: 4928000 D3/3: region item: south: 4914020 D3/3: region item: east: 609000 D3/3: region item: west: 590010 D3/3: region item: cols: 1899 D3/3: region item: rows: 1398 D3/3: region item: e-w resol: 10 D3/3: region item: n-s resol: 10 D3/3: region item: format: -1 D3/3: region item: compressed: 1 D3/3: create window mapping (1899 columns) C:/DOCUME~1/Hamish/LOCALS~1/Temp/STREAM_a02688: length = 9175040 sizeof(T)=4 D3/3: nrows=1398 ncols=1899 stream_len()=2293760 C:/DOCUME~1/Hamish/LOCALS~1/Temp/STREAM_a02688: length = 9175040 sizeof(T)=4 Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 146 This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information.
the length = 9175040 reported by the debug message jumps around every time I run it, sometimes it's 0, sometimes its ~550000 or so.
- also often it dies just after the "stats.out file found, renaming." message with error code 1.
rm stats.out*at the msys prompt gets it working again.
Hamish
follow-ups: 48 49 comment:47 by , 14 years ago
Replying to hamish:
Hi, just tested 6.5svn r42750 nightly build on XP. And I'm back to the original error message & exit code 3:
#spearfish g.region rast=elevation.10m r.terraflow elev=elevation.10m ...
That same command works for me.
Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 144 }}}
Close to an insult to ask, but have you done make distclean (package.sh doesn't)? You could check the headers in include/grass/iostream in the installation dir (not the source dir) to see if the patches are really in there.
Markus M
comment:48 by , 14 years ago
Replying to mmetz:
Replying to hamish:
Hi, just tested 6.5svn r42750 nightly build on XP. And I'm back to the original error message & exit code 3:
#spearfish g.region rast=elevation.10m r.terraflow elev=elevation.10m ...That same command works for me.
Assertion failed: nrows * ncols == str->stream_len(), file grass2str.h, line 144 }}}
Close to an insult to ask, but have you done make distclean (package.sh doesn't)? You could check the headers in include/grass/iostream in the installation dir (not the source dir) to see if the patches are really in there.
Markus M
I've tried a completly fresh svn-checkout of Grass65 in my WinVista32. r.terraflow works and finishes also now in the wx-gui.
Helmut
follow-up: 50 comment:49 by , 14 years ago
Replying to mmetz:
Close to an insult to ask,
not at all,
but have you done make distclean (package.sh doesn't)?
I tried the nightly binary snapshot from josef.fsv.cvut.cz. Even if package.sh does not, I don't know if the build script does (or does not).
You could check the headers in include/grass/iostream in the installation dir (not the source dir) to see if the patches are really in there.
When back at work with access I will check that, hopefully it explains it...
Hamish
follow-up: 51 comment:50 by , 14 years ago
Replying to hamish:
Replying to mmetz:
but have you done make distclean (package.sh doesn't)?
I tried the nightly binary snapshot from josef.fsv.cvut.cz.
I have now added "make distclean" to source:grass-addons/tools/wingrass-packager/grass_compile.sh
maybe it should better go into mswindows/osgeo4w/package.sh, changing the 'make clean' after ./configure there to 'make distclean' before. I leave that decision to the script's authors & runners. (I assume they update this by hand anyway)
Hamish
comment:51 by , 14 years ago
Replying to hamish:
Replying to hamish:
Replying to mmetz:
but have you done make distclean (package.sh doesn't)?
I tried the nightly binary snapshot from josef.fsv.cvut.cz.
I have now added "make distclean" to source:grass-addons/tools/wingrass-packager/grass_compile.sh
maybe it should better go into mswindows/osgeo4w/package.sh, changing the 'make clean' after ./configure there to 'make distclean' before. I leave that decision to the script's authors & runners. (I assume they update this by hand anyway)
I had a look changing it in package.sh, but I think that needs more fundamental change. Right now it is only re-running ./configure and 'make clean' on fresh checkouts. Given that, I'm surprised that trunk still builds actually.
Hamish
follow-up: 53 comment:52 by , 14 years ago
Replying to hamish:
patch attached for consideration. debug messages should be removed after testing.
The patch seems to work in combination with the later patch, results are identical between windows grass65, Linux grass65 patched and Linux grass65 unpatched.
Note that the patch for stream_len() contradicts the principle of r.terraflow to avoid random seek at all costs, fseek should only be called to reset the file pointer to the beginning of the file. Each call to the patched stream_len in mingw32 calls fseek twice, and stream_len itself is called very often as the debug messages illustrate (so often that wxGUI might freeze...). As a result, r.terraflow is heavily slowed down on windows: using elevation.10m from spearfish60 as input (2.6 million cells) takes with r.terraflow twice as long as with r.watershed in disk swap mode, and that even though all temp files also of r.terraflow are small enough to be cached by my test system.
Markus M
comment:53 by , 14 years ago
Replying to mmetz:
stream_len itself is called very often
In which case, it shouldn't be calling stat() either. This isn't exactly fast on Unix, is quite slow on Windows, and is extremely slow on Cygwin.
On Unix, fstat() is typically significantly faster than stat(), as stat() needs to resolve the pathname to an inode, which requires lookups and permission checks on each component in the path.
However: both fstat() and stat() reflect the size of the underlying file, which isn't necessarily the same as the size of the stream (FILE*) if buffering is in use. The only reliable way to determine the size of a stream is fseek(SEEK_END)+ftell().
If stream_len() is called often, it would be better to make the AMI_STREAM class keep track of its own length, updating it in the write_item() and write_array() methods.
follow-up: 55 comment:54 by , 14 years ago
6.4.0svn nightly build r42815 and 6.5svn r42819 on XP: still dies with the assertion mismatch in grass2str.h line 144 as in original report.
perhaps 'make distclean' is still not making it through to the nightly builds??
if I try again it also still dies if stats.out already exists in the pwd after giving you a 'renaming stats.out' message.
follow-up: 56 comment:55 by , 14 years ago
Replying to hamish:
6.4.0svn nightly build r42815 and 6.5svn r42819 on XP: still dies with the assertion mismatch in grass2str.h line 144 as in original report.
perhaps 'make distclean' is still not making it through to the nightly builds??
if I try again it also still dies if stats.out already exists in the pwd after giving you a 'renaming stats.out' message.
I can confirm this with 6.4.0svn nightly build r42815 on my WinVista32-box.
in my self-compiled grass64 and grass65 r.terraflow is working. also in a self built WinGrass-installer of grass64 and grass65.
but for compiling I have to comment out the lines added in r42799, r42797 because package.sh isn't working for me.
Helmut
follow-up: 57 comment:56 by , 14 years ago
Replying to hamish:
6.4.0svn nightly build r42815 and 6.5svn r42819 on XP: still dies with the assertion mismatch in grass2str.h line 144 as in original report.
perhaps 'make distclean' is still not making it through to the nightly builds??
Maybe because configure-stamp has not been removed before running package.sh again [1]
Replying to hellik:
in my self-compiled grass64 and grass65 r.terraflow is working. also in a self built WinGrass-installer of grass64 and grass65.
I can confirm.
Markus M
[1] https://trac.osgeo.org/grass/browser/grass/branches/releasebranch_6_4/mswindows/osgeo4w/package.sh#L31
comment:57 by , 14 years ago
comment:58 by , 14 years ago
| Resolution: | → fixed |
|---|---|
| Status: | new → closed |
Aaah - just noticed - could this be related to #520?