r.external / r.external.out: support paths relative to the mapset, and variables

[sbl]

Currently, r.external.out uses absolute paths which may be incorrect if the database is accessed via a networked filesystem (or even a local filesystem if the mount point changes).

If the directory given to r.external.out is relative (doesn't begin with a "/"), it's converted to an absolute path relative to the current mapset directory.

Thereafter, any created maps will have the absolute path in their GDAL link (i.e. the cell_misc/<map>/gdal file).

Thus, maps created with r.external.out can't be read if the database directory has "moved" relative to its location when the map was created.

It would be helpful if paths relative to the mapset could be supported, and maybe additional options, e.g. the ability to use environment variables in paths

Here is a test case in order to reproduce the issue:

First I created a new and empty GRASS Location "test" on a network storage (NFS) from the Linux Server. This GRASS DB is mounted on LINUX (through mount.cifs) to `/home/stefan/R_raw_data/test'. On Windows the "Network drive" is mapped to " R:\Raw_data\test "

Within this Location I crated a mapset "linux" (from the Linux Server) and a mapset "windows" from my Windows 7 workstation.

1st test case: Linux to Windows: On the Linux server, and in the "linux" mapset I did:

r.external.out --verbose directory="/home/stefan/R_raw_data/test" extension=".tif" format="GTiff"
g.region -p n=1 s=0 e=1 w=0 res=0.1
r.mapcalc expression="linux_map=1"

On my Win 7 box I did:

d.rast map= linux_map @linux Here I got the following error message:

Command 'd.rast map=linux_map@linux' failed
Details: ERROR 4: `/home/stefan/R_raw_data/test/linux_map.tif' does not exist in the file system, and is not recognised as a supported dataset name.

Then I tried relative paths: On Linux:

r.external.out --verbose directory="../../" extension=".tif" format="GTiff"
r.mapcalc expression="linux_map=1" --o

Error message on Windows changes to:

Command 'd.rast map=linux_map@linux' failed
Details: ERROR 4: `/home/stefan/R_raw_data/test/newLocation/linux/../..//linux_map.tif' does not exist in the file system, and is not recognised as a supported dataset name.

2nd test case: Windows to Linux: On my Windows box (and in the "windows" mapset) I did:

r.external.out --verbose directory="R:\Raw_data\test" extension=".tif" format="GTiff"
r.mapcalc expression=windows_map=1

Which results in the following error message:

ERROR: Unable to make mapset element R:\Raw_data\test (R:\Raw_data\test/newLocation/windows/R:\Raw_data\test): No such file or directory

Then I tried relative paths:

r.external.out --verbose directory="../../" extension=.tif format=GTiff

The relative paths work for the Windows side. However, back on Linux I tried: d.rast map=windows_map@windows and got the following error message:

Command 'd.rast map=windows_map@windows' failed
Details: ERROR 4: `R:\Raw_data\test/newLocation/windows/../..//windows_map.tif' does not exist in the file system, and is not recognised as a supported dataset name.

[neteler]

Ticket retargeted after 7.0.1 milestone closed

[neteler]

Ticket retargeted after milestone closed

[sbl]

This issue applies equally to r.external, where links are allways converted to absolute paths (​https://trac.osgeo.org/grass/browser/grass/trunk/raster/r.external/main.c lines: 162-170).

Both r.external and r.external.out are therefore practically unusable in mixed environments where data can / should be accessed from both Windows and Linux. Maps linked with r.external / r.external.out will not be readable across platforms, because absolute paths differ (of course directory delimiters differ as well).

[dnewcomb]

Replying to sbl:

This issue applies equally to r.external, where links are allways converted to absolute paths (​https://trac.osgeo.org/grass/browser/grass/trunk/raster/r.external/main.c lines: 162-170).

Both r.external and r.external.out are therefore practically unusable in mixed environments where data can / should be accessed from both Windows and Linux. Maps linked with r.external / r.external.out will not be readable across platforms, because absolute paths differ (of course directory delimiters differ as well).

It seems to me that there is a limit to what GRASS can do to solve this issue. Relative paths can also be disrupted between OS's due to the difference in directory delimiters ( / vs \ ) . Perhaps the best option is a method for detection and "repair" that allows one to clone the settings for the original link and create a new link with a user specified path? Perhaps one that creates the new link with a suffix of _w for windows or a suffix of _l for linux?

[glynn]

Replying to sbl:

If the directory given to r.external.out is relative (doesn't begin with a "/"), it's converted to an absolute path relative to the current mapset directory.

This isn't quite correct (although the distinction doesn't matter much in practice). r.external.out stores the path exactly as given, but when a linked map is created, a relative directory is converted to an absolute directory based upon the current mapset directory.

For linked maps created with r.external, a relative filename is converted to an absolute filename relative to the current working directory. When accessing a linked map, no conversion is performed; the filename is passed directly to GDALOpen(). If it was possible to create a link with a relative filename, it would be interpreted relative to the process' current directory.

It would be helpful if paths relative to the mapset could be supported, and maybe additional options, e.g. the ability to use environment variables in paths

The first option shouldn't be that hard. It would just require moving the relative-to-absolute conversion from read_gdal_options() (used by Rast_create_gdal_link()) to Rast_get_gdal_link(), so that relative paths are stored as relative then converted when accessing a linked map.

The case where a link is created manually with r.external is a bit harder. Currently, relative paths are relative to the current working directory, not the mapset directory, so deferring the conversion to read time would change the behaviour. Another option is to have a GRASS variable holding the directory for which relative paths would be interpreted.

And none of this does anything for the situation where a linked map is outside of the mapset directory (which is the intended usage; the GRASS database isn't supposed to contain arbitrary data files). For that, the GDAL-link code would need some kind of remapping table. Or we would have to allow pathnames to contain some kind of variables (not necessarily environment variables; GRASS variables would probably be more convenient).

[sbl]

Thanks for looking into this!

Let`s say my GRASS DB is /grassdata/LOCATION/MAPSET and my external data is on /raw_data/... The Windows equivalent is R:\grassdata\LOCATION\MAPSET and R:\raw_data\... (lets assume storage areas are mounted equally on Windows and Linux (meaning in particular Windowss drive letters do not change).

If one was able to store the links relative to the current mapset, like this ../../../raw_data/my_time_series/lots_of_maps.tif Then the path would be identical for both Windows and Linux, if the code or link storage could handle the directory separators OS dependent. In the given example the data would be outside the GRASS DB.

If current behavior should not be changed, maybe a new flag "store link relative to current mapset" could be added?

P.S.: Sorry for not being precise in my first post. I was actually thinking of the GRASS variables GISDBASE, LOCATION, MAPSET.

[glynn]

Replying to sbl:

Let`s say my GRASS DB is /grassdata/LOCATION/MAPSET and my external data is on /raw_data/... The Windows equivalent is R:\grassdata\LOCATION\MAPSET and R:\raw_data\... (lets assume storage areas are mounted equally on Windows and Linux (meaning in particular Windowss drive letters do not change).

The specific case where both the database and GDAL files are on the same filesystem probably isn't all that common. It's more likely that the two will move relative to each other. E.g the database in a fixed location on an internal disk or network share, the GDAL files on an external drive or a different network share.

IMHO, any fix needs to be more general.