r.external to accept gdal config options

[perrygeo]

When linking to an external GDAL raster source, it would be useful to pass GDAL configuration options (​http://trac.osgeo.org/gdal/wiki/ConfigOptions)

Consider this scenario: I need to specify a GDAL config option to read an NetCDF file correctly. I can specify GDAL_NETCDF_BOTTOMUP=NO as an environment variable which works for most cases but when using a multiprocessing approach to parallelization (as e.g. t.aggregate does) the newly spawned processes don't inherit the same environment and will fail.

The solution might be for r.external to accept GDAL config options that can be applied regardless of the environment variables and can allow externally linked rasters to function properly across processes.

[neteler]

The request sounds reasonable but I didn't figure out how to pass "papszOptions" to GDAL as used in r.external. The magic line in r.out.gdal is this:

# r.in.gdal/main.c

	hDstDS =
	    GDALCreate(hDriver, output->answer, cellhead.cols, cellhead.rows,
		       ref.nfiles, datatype, papszOptions);

but in r.external GDALOpen() is used. Perhaps a GDAL expert can tell us the trick.

[glynn]

Replying to neteler:

The request sounds reasonable but I didn't figure out how to pass "papszOptions" to GDAL as used in r.external. The magic line in r.out.gdal is this:

but in r.external GDALOpen() is used. Perhaps a GDAL expert can tell us the trick.

Note that r.external is the analogue of r.in.gdal, which doesn't accept any configuration options.

The analogue of r.out.gdal is r.external.out, which has an options= option.

If options are needed for reading, a similar option should be added to both r.in.gdal and r.external, presumably using GDALOpenEx() instead of GDALOpen(). The latter will require extending the GDAL "link" format (lib/raster/gdal.c).

[neteler]

Replying to glynn:

Replying to neteler:

The request sounds reasonable but I didn't figure out how to pass "papszOptions" to GDAL as used in r.external. The magic line in r.out.gdal is this:

but in r.external GDALOpen() is used. Perhaps a GDAL expert can tell us the trick.

Note that r.external is the analogue of r.in.gdal, which doesn't accept any configuration options.

While it doesn't directly, I had added a larger cache some time ago by setting GDALSetCacheMax() to 300MB rather than the tiny 40MB default GDAL cache size. This speeds up import tremendously:

r.in.gdal ...
memory=integer
    Maximum memory to be used (in MB)
    Cache size for raster rows
    Options: 0-2047
    Default: 300

I wonder how to get that into r.external (I suppose that it would benefit as well).

If options are needed for reading, a similar option should be added to both r.in.gdal and r.external, presumably using GDALOpenEx() instead of GDALOpen(). The latter will require extending the GDAL "link" format (lib/raster/gdal.c).

OK (no idea how to implement that).

[dylan]

Replying to neteler:

Replying to glynn:

Replying to neteler:

The request sounds reasonable but I didn't figure out how to pass "papszOptions" to GDAL as used in r.external. The magic line in r.out.gdal is this:

but in r.external GDALOpen() is used. Perhaps a GDAL expert can tell us the trick.

Note that r.external is the analogue of r.in.gdal, which doesn't accept any configuration options.

While it doesn't directly, I had added a larger cache some time ago by setting GDALSetCacheMax() to 300MB rather than the tiny 40MB default GDAL cache size. This speeds up import tremendously:

r.in.gdal ...
memory=integer
    Maximum memory to be used (in MB)
    Cache size for raster rows
    Options: 0-2047
    Default: 300

I wonder how to get that into r.external (I suppose that it would benefit as well).

If options are needed for reading, a similar option should be added to both r.in.gdal and r.external, presumably using GDALOpenEx() instead of GDALOpen(). The latter will require extending the GDAL "link" format (lib/raster/gdal.c).

OK (no idea how to implement that).

Finding this thread after searching for some ways to speed-up file access to maps linked via r.external.

The adjustable cache solution in r.in.gdal appears to be:

if (parm.memory->answer && *parm.memory->answer) {
	   /* TODO: GDALGetCacheMax() overflows at 2GiB, implement use of GDALSetCacheMax64() */
           GDALSetCacheMax(atol(parm.memory->answer) * 1024 * 1024);
           G_verbose_message(_("Using memory cache size: %.1f MiB"), GDALGetCacheMax()/1024.0/1024.0);
    }

Could this same block of code be used within r.external? I don't fully understand how r.external works, so I suppose that it is more complicated than this.

Or another option, is there an environmental variable that could be used to control the GDAL cache size?

[glynn]

Replying to dylan:

Could this same block of code be used within r.external? I don't fully understand how r.external works, so I suppose that it is more complicated than this.

r.external itself just sets up the "link" between GRASS and the data file. The actual I/O occurs in lib/raster (gdal.c, open.c, get_row.c, close.c) when a GRASS module reads the map.

But the data which r.external controls is per-map, while this appears to be a global setting. What happens when a module reads multiple GDAL-linked maps with different settings? It might make more sense to set this in Rast_init_gdal() from an environment variable or $GISRC variable.

[dylan]

Replying to glynn:

Replying to dylan:

Could this same block of code be used within r.external? I don't fully understand how r.external works, so I suppose that it is more complicated than this.

r.external itself just sets up the "link" between GRASS and the data file. The actual I/O occurs in lib/raster (gdal.c, open.c, get_row.c, close.c) when a GRASS module reads the map.

But the data which r.external controls is per-map, while this appears to be a global setting. What happens when a module reads multiple GDAL-linked maps with different settings? It might make more sense to set this in Rast_init_gdal() from an environment variable or $GISRC variable.

Thank you for the clarification Glynn. I think that a suitable environmental or GRASS variable would be ideal. Something that isn't widely used but *very* important when working with massive, numerous, or massive and numerous files. I am unable to implement but happy to test and document.

[dylan]

Checking-in, any progress?

[neteler]

Milestone renamed

[neteler]

Ticket retargeted after milestone closed

[martinl]

All enhancement tickets should be assigned to 7.4 milestone.

[neteler]

Ticket retargeted after milestone closed

[martinl]

All enhancement tickets should be assigned to 7.6 milestone.

[martinl]

Ticket retargeted after milestone closed

[martinl]

Ticket retargeted after milestone closed