netCDF driver ignores the "scale_factor"

[jluis]

The scale factor in netcdf files is ignored. I'll use a GMT generated file to show this.

- Generate one dumb file grdmath -R-10/10/-10/10 -I1 X Y MUL = lixo.grd

- Change the scale_factor from its default value of 1 to 0.01 grdedit lixo.grd -D=/=/=/0.01/=/=/=/=

- Convert to tiff gdal_translate lixo.grd lixo.tiff

Ask GMT & GDAL about data min/max

grdinfo lixo.grd <...> lixo.grd: x_min: -10 x_max: 10 x_inc: 1 name: x nx: 21 lixo.grd: y_min: -10 y_max: 10 y_inc: 1 name: y ny: 21 lixo.grd: z_min: -1 z_max: 1 name: z lixo.grd: scale_factor: 0.01 add_offset: 0

gdalinfo lixo.tiff -mm <...> Band 1 Block=21x21 Type=Float32, ColorInterp?=Gray

Computed Min/Max?=-100.000,100.000

NoData? Value=nan

What happens is that while converting the scale_factor was not applied.

[kyle]

I should be able to get to this this week.

[kyle]

I have added Set/Get? functions for Offset and Scale in the netcdf driver. Ticket #1660 was submitted some time ago and was in reference to this issue too. As Frank mentioned in that ticket:

Note that GDALRasterBand has methods to get the offset and scale. The normal GDAL practice would be to return them via those methods - not to apply them on the fly. Then it is up to the caller to do so if they wish.

I agree with that, in that GDAL is allowing access to the data. This avoids data type issues. In the case above, gdalinfo is the caller. I will commit this after some testing. jluis, do you have any other datasets that are scaled i could use for testing? if not, i will build a few. I will also add a test to autotest.

kss

[kyle]

I am going to close this and call it fixed. There may be another issue, but I think it deserves another ticket. Scale and Offset implemented in trunk r20997

[etourigny]

I'm reopening this ticket, because the resolution is not satisfactory.

The result of the discussion at  http://www.osgeo.org/pipermail/gdal-dev/2010-October/026544.html seems to be that gdalinfo shouldn't be modified to account for Offset and Scale.

However, this begs the question: what is the use of these RasterBand? variables if nothing uses them? Allegedly, one can call gdal_translate -unscale but that is somewhat inconvenient. Further, other apps like QGis do not necessarily support Offset and Scale.

Perhaps a better solution in the netcdf driver is to use internal offset/scale values, and calculate the unscaled valeus in RasterIO() (as suggested in bug #1660). Downside of this, is that code calling RasterIO will probably be expecting Integer (scaled data).

An other option is to unscale the data when creating the file, but this defeats the purpose of scaled data.

[etourigny]

Perhaps I do not understand the use and meaning of GDAL's Offset and Scale variables. In netcdf they are intended to be transparent to the user, whereas in GDAL it is up to the user to manage it. Very different from compression packing (i.e. GTIFF DEFLATE) which is transparent at least in reading.

[kyle]

Etienne, I believe that GDAL is for low level access of files, not analysis. It does matter what the unscaled values are in that regard. If the user wants to store 8 bit values and specify scale and offset, they can. If a user of the library wants to unscale values they can. gdalinfo aren't the only apps that use gdal. Unscaling values should be left to the user, imho.

kss

[etourigny]

Kyle,

Thanks for your input.

I understand your arguments, however it would be much easier if GDAL (gdalinfo) allowed to get the unscaled values for a rapid analysis, like other command-line tools such as nco and cdo. However, I understand that the GDAL philosophy is to keep it as low-level as possible, and not mess with higher-level details.

In light of this I will close re-close this bug, submit a report to QGIS and consider adding an option to unscale in the netcdf driver (-co or config).