r.viewshed fails on large raster: mktemp provides only 26 unique file names

[wenzeslaus]

Citing original report from mailing list ​r.viewshed fails on large raster: temp file already exists (​nabble):

I was running r.viewshed in GRASS 7.0 under Win7 on a large raster with 11000 rows by 11000 columns. The algorithm ran in external memory mode with temporary files being written to a local directory (on C:). On starting sweeping, two temporary files were created, “STREAM_a06524” and “STREAM_b06524”, of which the first never increased in size beyond 0 KB, and the second grew to about 11 GB during sweeping. When sorting events, additional temp files were created, each of about 255 MB size, which were all named in the same logic, i.e. “STREAM_c06524”, “STREAM_d06524” etc, with only the letter after the underscore in the file name changing to the next letter in the alphabet. This continued until the file “STREAM_z06524” had been created, then the algorithm crashed with “File exists” (see sh output below). Obviously it had tried to create a file that already existed, probably named “STREAM_a06524”. It seems that either the programmer never counted on more files being necessary than the alphabet has letters, or maybe something gets mixed up in the loop and the code assumes that “STREAM_a06524” does not exist because it never wrote any data to it.

...
ami_single_temp_name: mktemp failed: : File exists
Assertion failed: 0, file ami_stream.cpp, line 97

As found by ​MarkusN, MS Windows' ​_mktemp supports only 26 unique file names:

_mktemp can create a maximum of 26 unique file names for any given
combination of base and template values. Therefore, FNZ12345 is the
last unique file name _mktemp can create for the base and template
values used in this example.

Because it ​works on Linux, the problem is probably connected only to MS Windows' _mktemp function.

References:

• ​http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47439 (similar report for gfortran)
• ​http://gcc.gnu.org/ml/fortran/2011-03/msg00102/mktemp.diff (otential patch to be re-used)
• ​http://msdn.microsoft.com/en-us/library/34wc6k1f%28v=VS.80%29.aspx (_mktemp at MSDN)
• ​http://msdn.microsoft.com/en-us/library/t8ex5e91%28v=vs.80%29.aspx (_mktemp_s at MSDN)
• ​http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2011/n3242.pdf (C++ standard defines tmpfile and tmpnam)
• ​http://www.gnu.org/software/libc/manual/html_node/Temporary-Files.html (GNU documentation)

[neteler]

In case a new G_mktemp() should also be used in

general/g.tempfile/main.c

See ticket #1902

[neteler]

A recent tempfile() implementation which does not have the 26 char limitation can be found here:

​http://code.metager.de/source/xref/tudresden/tudos/l4/pkg/libgfortran/lib/contrib/io/unix.c#1021

License:

"This file is part of the GNU Fortran runtime library (libgfortran).

Libgfortran is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3, or (at your option) any later version. ... "

[neteler]

r58740 to be discussed and potentially used here

[neteler]

Replying to neteler:

r58740 to be discussed and potentially used here

G_mkstemp() is now activated in libiostream (used by r.terraflow, r.viewshed) in r58887. Please test in the winGRASS binaries which will be generated in approx. 9 hours from now.

(Ticket #1902 not yet addressed)

[hellik]

Replying to neteler:

Replying to neteler:

r58740 to be discussed and potentially used here

G_mkstemp() is now activated in libiostream (used by r.terraflow, r.viewshed) in r58887. Please test in the winGRASS binaries which will be generated in approx. 9 hours from now.

(Ticket #1902 not yet addressed)

tested with following region

g.region -p                                                                     
projection: 99 (Lambert Azimuthal Equal Area)
zone:       0
datum:      etrs89
ellipsoid:  grs80
north:      2889850
south:      2595725
west:       4284725
east:       4853725
nsres:      25
ewres:      25
rows:       11765
cols:       22760
cells:      267771400

and

System Info                                                                     
GRASS Version: 7.0.svn                                                          
GRASS SVN Revision: 59343M                                                      
Erstellungsdatum: 2014-03-26                                                    
Build Platform: i686-pc-mingw32                                                 
GDAL/OGR: 1.10.1                                                                
PROJ.4: 4.8.0                                                                   
GEOS: 3.4.2                                                                     
SQLite: 3.7.17                                                                  
Python: 2.7.4                                                                   
wxPython: 2.8.12.1                                                              
Platform: Windows-7-6.1.7601-SP1 (OSGeo4W)     

the temp-files are created and named according the new scheme.

closing ticket