Opened 12 years ago

Closed 5 years ago

#3085 closed enhancement (fixed)

[PATCH] Incorrect nodata values when using the -scale option in gdal_translate

Reported by: ersts Owned by: Even Rouault
Priority: normal Milestone:
Component: GDAL_Raster Version: 1.5.2
Severity: normal Keywords: gdal_translate nodata ubuntu
Cc: warmerdam, antonio, Kyle Shannon

Description (last modified by warmerdam)

When scaling data and converting between data types, I am experiencing unexpected results, specifically with the nodata values in the resulting image. I am running Ubuntu 9.04 with stock GDAL 1.5.2

Example ascii grid:

ncols        5 
nrows        5 
xllcorner     -2610321.0830204
yllcorner     -3330321.0830204
cellsize      4500.0
NODATA_value  -9999.0
-9999.0 -9999.0 1.2E-4 1.2E-4 1.2E-4
-9999.0 -9999.0 1.2E-3 1.2E-3 1.2E-3
-9999.0 -9999.0 1.2E-2 1.2E-2 1.2E-2
-9999.0 -9999.0 1.2E-1 1.2E-1 1.2E-1
-9999.0 -9999.0 5.2E-1 5.2E-1 5.2E-1

with the command:

gdal_translate -ot int32 -scale 0 1 0 1000 -a_nodata -9999 -of GTiff test.asc test.tif

The nodata value is reported as -9999 but in the image -9999 becomes -9999000

gdal_translate -ot Int16 -scale 0 1 0 1000 -a_nodata -9999 -of GTiff test.asc test.tif

The nodata value is reported as -9999 but in the image -9999 becomes -32768

If the scaling option is removed, the nodata values are correct in the destination image

Attachments (2)

ticket_3085.patch (4.3 KB) - added by Even Rouault 12 years ago.
ticket_3085.2.patch (6.1 KB) - added by Even Rouault 12 years ago.

Download all attachments as: .zip

Change History (19)

comment:1 Changed 12 years ago by Even Rouault

Owner: changed from warmerdam to Even Rouault
Status: newassigned

comment:2 Changed 12 years ago by warmerdam

Description: modified (diff)

comment:3 Changed 12 years ago by warmerdam

Cc: warmerdam added

Peter / Even,

I don't really see the issue. Did you anticipate that gdal_translate would somehow treat nodata values specially, and not rescale them? Did you think that -a_nodata would alter the actual pixel values?

comment:4 in reply to:  3 Changed 12 years ago by ersts

Replying to warmerdam:

Did you anticipate that gdal_translate would somehow treat nodata values specially, and not rescale them? Did you think that -a_nodata would alter the actual pixel values?

Yes, I guess so. I would have expected nodata in the src image to be remain nodata in the destination image no matter what the operation - except for of course setting a new nodata value.

With scaling, the -a_nodata flag was included thinking that any value falling out of the indicated src range would have either been set to the specified nodata value, or at least the min or max of dest range.

comment:5 Changed 12 years ago by Even Rouault

Frank,

I was a bit puzzled about if the current behaviour was expected or not at first. But I realized when looking more closely at the VRT driver that we have already a way to tell it that the source data has a nodata value so that it doesn't set the value in the output buffer (see VRTComplexSource::RasterIO()). This mechanism wasn't used by gdal_translate.

So I'm attaching a patch (for trunk) that will give a more natural result I think, that is to say that when a raster has a src nodata value, it is taken into account when translating each pixel value. If people absolutely want the old behaviour, they could use the new option -srcnodata NONE so that nodata pixels are considered as regular values, like currently.

Another advantage is that the new behaviour of gdal_translate will be consistant with gdalwarp and its -srcnodata and -dstnodata options.

A few examples :

"gdal_translate src_raster dst_raster -a_nodata -9999" :
- will just defined -9999 as nodata in the dst_raster if src_raster has no nodata value
- will defined -9999 as the new nodata value if the src_raster has a nodata value
and remap all pixels that had the old nodata value to the new nodata value

"gdal_translate src_raster dst_raster -srcnodata none -a_nodata -9999" will defined
-9999 as the new nodata value and not change any pixel value

"gdal_translate src_raster dst_raster -scale 0 1 0 1000" will scale all pixel
values, except those which are equal to the nodata value and preserve the source 
nodata value unscaled

"gdal_translate src_raster dst_raster -scale 0 1 0 1000 -a_nodata -32768" will
scale all pixel values, except those which are equal to the source nodata value
 (-9999 e.g.) and will set them to dest nodata value (-32768)

"gdal_translate src_raster dst_raster -scale 0 1 0 1000 -srcnodata none -a_nodata    -32768"
will scale all pixel values (including those which are equal to the source 
nodata value) and will set -32768 as the dest nodata value

"gdal_translate src_raster dst_raster -scale 0 1 0 1000 -srcnodata 12345 -a_nodata -32768"
will scale all pixel values, except those which are equal to 12345 and will set -them to -32768

Do you agree with that ?

Changed 12 years ago by Even Rouault

Attachment: ticket_3085.patch added

Changed 12 years ago by Even Rouault

Attachment: ticket_3085.2.patch added

comment:6 Changed 12 years ago by Even Rouault

Summary: Incorrect nodata values when using the -scale option in gdal_translate[PATCH] Incorrect nodata values when using the -scale option in gdal_translate
Type: defectenhancement

I won't apply the patch for the moment. New behaviour doesn't make consensus.

comment:7 Changed 12 years ago by ersts

rouault

The examples you provided sound logical. I hope other folks are interested in this enhancement and it is accepted. Thanks for taking the time to look into it.

-pete

comment:8 Changed 12 years ago by jctull

+1 to get this patch implemented. I am running into this very issue with the 1.6 branch trying to convert a dem from feet to meters. The nodata value is also scaled causing an incorrect min/max for the raster.

comment:9 Changed 10 years ago by antonio

Cc: antonio added

comment:10 Changed 10 years ago by Kyle Shannon

Cc: Kyle Shannon added

+1 for the patch.

comment:11 Changed 10 years ago by Kyle Shannon

I am running into similar results when I use -unscale in gdal_translate. I would think that nodata values would stay nodata values, but they become valid values and ruin my raster (stats):

C:\Users\k\Desktop\mmi_tmp\vrt_test>gdalinfo -stats vrt\mmi.vrt
Driver: VRT/Virtual Raster
Files: vrt\mmi.vrt
       csv\UShazard.20081231.pga.10pc50
Size is 1201, 509
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0,
        AUTHORITY["EPSG","8901"]],
    UNIT["degree",0.0174532925199433,
        AUTHORITY["EPSG","9122"]],
    AUTHORITY["EPSG","4326"]]
Origin = (-125.025000000000010,50.024999999999999)
Pixel Size = (0.050000000000000,-0.050000000000000)
Corner Coordinates:
Upper Left  (-125.0250000,  50.0250000) (125d 1'30.00"W, 50d 1'30.00"N)
Lower Left  (-125.0250000,  24.5750000) (125d 1'30.00"W, 24d34'30.00"N)
Upper Right ( -64.9750000,  50.0250000) ( 64d58'30.00"W, 50d 1'30.00"N)
Lower Right ( -64.9750000,  24.5750000) ( 64d58'30.00"W, 24d34'30.00"N)
Center      ( -95.0000000,  37.3000000) ( 95d 0' 0.00"W, 37d18' 0.00"N)
Band 1 Block=128x128 Type=Float32, ColorInterp=Undefined
  Minimum=0.005, Maximum=1.110, Mean=0.041, StdDev=0.064
  NoData Value=-9999
  Metadata:
    STATISTICS_MINIMUM=0.0049999998882413
    STATISTICS_MAXIMUM=1.1102999448776
    STATISTICS_MEAN=0.040921010072679
    STATISTICS_STDDEV=0.064469886712302

C:\Users\k\Desktop\mmi_tmp\vrt_test>gdal_translate -unscale vrt\mmi.vrt output\mmi.tif
Input file size is 1201, 509
0...10...20...30...40...50...60...70...80...90...100 - done.

C:\Users\k\Desktop\mmi_tmp\vrt_test>gdalinfo -stats output\mmi.tif
Driver: GTiff/GeoTIFF
Files: output\mmi.tif
Size is 1201, 509
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (-125.025000000000010,50.024999999999999)
Pixel Size = (0.050000000000000,-0.050000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (-125.0250000,  50.0250000) (125d 1'30.00"W, 50d 1'30.00"N)
Lower Left  (-125.0250000,  24.5750000) (125d 1'30.00"W, 24d34'30.00"N)
Upper Right ( -64.9750000,  50.0250000) ( 64d58'30.00"W, 50d 1'30.00"N)
Lower Right ( -64.9750000,  24.5750000) ( 64d58'30.00"W, 24d34'30.00"N)
Center      ( -95.0000000,  37.3000000) ( 95d 0' 0.00"W, 37d18' 0.00"N)
Band 1 Block=1201x1 Type=Float32, ColorInterp=Gray
  Minimum=0.005, Maximum=1.110, Mean=0.041, StdDev=0.064
  NoData Value=-9999
  Metadata:
    STATISTICS_MINIMUM=0.0049999998882413
    STATISTICS_MAXIMUM=1.1102999448776
    STATISTICS_MEAN=0.040921010072679
    STATISTICS_STDDEV=0.064469886712297

I don't believe this is expected. Using 1.8.1 from Tamas' site.

comment:12 Changed 10 years ago by Kyle Shannon

Sorry, I don't get it for a single band raster, but I do with a multiband raster, I will try to hunt down my issue.

comment:13 Changed 10 years ago by Kyle Shannon

I am getting the same issue with -unscale. I understand the default behavior should be to apply scale and offset to all data, regardless of the nodata value. Is it possible to add a config option? It doesn't seem like it fits the scope of most config options.

comment:14 Changed 7 years ago by mikrit

+1 to get this patch implemented, or some option to keep NODATA values unchanged by the -scale operation.

I am running into this issue in GDAL 1.9.0. Use-case: a free Lake Tahoe DEM is published as a Float32 raster with elevations in feet, and NODATA value is -32767.0. I translated it to meters by using

-scale 0 100000 0 30480.

But then, the NODATA value remains -32767.0, while the pixels that used to be undefined now contain -9987.3816, so no longer undefined. I suppose I could specify

-a_nodata -9987.3816,

but I dislike having this silly number as the NODATA value, and I worry that it is not representable exactly as a Float32 number, so a mismatch might occur.

Actually, I don't know what to do. How can I use GDAL Utilities to convert the elevation raster to meters, while keeping undefined values as undefined, and represented by an integer NODATA value (or represented by NaN, perhaps)?

Slightly related question: gdalinfo reported "Unit Type: ft" for Band 1 of the original file. Very nice, that's what alerted me (especially since the official document for the data says it's in meters!). However, after rescaling to meters with gdal_translate, the metadata "Unit Type: ft" remains. I understand that these metadata are not interpreted by gdal_translate, but how would I change it or remove it?

comment:15 Changed 7 years ago by Even Rouault

trunk r27928 gdal_translate: don't recopy band units if rescaling or unscaling is involved (mentionned in comment 14 of #3085)" (you can probably unset it by calling SetUnitType?("") on the band)

Regarding the patch, I don't remember why I wrote "I won't apply the patch for the moment. New behaviour doesn't make consensus" at that time. But would probably be material appropriate for GDAL 2.0

comment:16 Changed 7 years ago by mikrit

Thanks, Even.

For the record, I managed to do a workaround. First, I did:

gdalwarp -dstnodata -10000 tahoedems.tif tahoedems_nodata_m10k.tif

This will set the new NODATA value to be -10000.0, and pixels that used to contain the old NODATA value -32767.0 will be replaced by -10000.0.

For some reason, it seems that gdalwarp removes the "Unit Type: ft" metadata. I suppose this can be regarded as a bug in gdalwarp, but it's good for me!

Then, I did

gdal_translate -scale 0 100000 0 30480 -a_nodata -3048 tahoedems_nodata_m10k.tif tahoedems_meters.tif

This will convert from feet to meters, and pixels containing the undefined -10000.0 feet are converted to -3048.0 meters. It is necessary to specify -a_nodata -3048 in the command. Fortunately, -3048.0 is an integer, and as far as I can tell, the rescaling makes the pixel values exactly -3048.0. I can certainly live with -3048 as the undefined value for elevations (the original -32767 does not make more sense).

comment:17 Changed 5 years ago by Even Rouault

Resolution: fixed
Status: assignedclosed

trunk r37633 "gdal_translate: set nodata value on vrtsource on scale / unscale / expand cases (patch by Zac Flamig, https://github.com/OSGeo/gdal/pull/199)"

Note: See TracTickets for help on using tickets.