Opened 15 years ago
Closed 7 years ago
#3085 closed enhancement (fixed)
[PATCH] Incorrect nodata values when using the -scale option in gdal_translate
Reported by: | ersts | Owned by: | Even Rouault |
---|---|---|---|
Priority: | normal | Milestone: | |
Component: | GDAL_Raster | Version: | 1.5.2 |
Severity: | normal | Keywords: | gdal_translate nodata ubuntu |
Cc: | warmerdam, antonio, Kyle Shannon |
Description (last modified by )
When scaling data and converting between data types, I am experiencing unexpected results, specifically with the nodata values in the resulting image. I am running Ubuntu 9.04 with stock GDAL 1.5.2
Example ascii grid:
ncols 5 nrows 5 xllcorner -2610321.0830204 yllcorner -3330321.0830204 cellsize 4500.0 NODATA_value -9999.0 -9999.0 -9999.0 1.2E-4 1.2E-4 1.2E-4 -9999.0 -9999.0 1.2E-3 1.2E-3 1.2E-3 -9999.0 -9999.0 1.2E-2 1.2E-2 1.2E-2 -9999.0 -9999.0 1.2E-1 1.2E-1 1.2E-1 -9999.0 -9999.0 5.2E-1 5.2E-1 5.2E-1
with the command:
gdal_translate -ot int32 -scale 0 1 0 1000 -a_nodata -9999 -of GTiff test.asc test.tif
The nodata value is reported as -9999 but in the image -9999 becomes -9999000
gdal_translate -ot Int16 -scale 0 1 0 1000 -a_nodata -9999 -of GTiff test.asc test.tif
The nodata value is reported as -9999 but in the image -9999 becomes -32768
If the scaling option is removed, the nodata values are correct in the destination image
Attachments (2)
Change History (19)
comment:1 by , 15 years ago
Owner: | changed from | to
---|---|
Status: | new → assigned |
comment:2 by , 15 years ago
Description: | modified (diff) |
---|
follow-up: 4 comment:3 by , 15 years ago
Cc: | added |
---|
comment:4 by , 15 years ago
Replying to warmerdam:
Did you anticipate that gdal_translate would somehow treat nodata values specially, and not rescale them? Did you think that -a_nodata would alter the actual pixel values?
Yes, I guess so. I would have expected nodata in the src image to be remain nodata in the destination image no matter what the operation - except for of course setting a new nodata value.
With scaling, the -a_nodata flag was included thinking that any value falling out of the indicated src range would have either been set to the specified nodata value, or at least the min or max of dest range.
comment:5 by , 15 years ago
Frank,
I was a bit puzzled about if the current behaviour was expected or not at first. But I realized when looking more closely at the VRT driver that we have already a way to tell it that the source data has a nodata value so that it doesn't set the value in the output buffer (see VRTComplexSource::RasterIO()). This mechanism wasn't used by gdal_translate.
So I'm attaching a patch (for trunk) that will give a more natural result I think, that is to say that when a raster has a src nodata value, it is taken into account when translating each pixel value. If people absolutely want the old behaviour, they could use the new option -srcnodata NONE so that nodata pixels are considered as regular values, like currently.
Another advantage is that the new behaviour of gdal_translate will be consistant with gdalwarp and its -srcnodata and -dstnodata options.
A few examples :
"gdal_translate src_raster dst_raster -a_nodata -9999" : - will just defined -9999 as nodata in the dst_raster if src_raster has no nodata value - will defined -9999 as the new nodata value if the src_raster has a nodata value and remap all pixels that had the old nodata value to the new nodata value "gdal_translate src_raster dst_raster -srcnodata none -a_nodata -9999" will defined -9999 as the new nodata value and not change any pixel value "gdal_translate src_raster dst_raster -scale 0 1 0 1000" will scale all pixel values, except those which are equal to the nodata value and preserve the source nodata value unscaled "gdal_translate src_raster dst_raster -scale 0 1 0 1000 -a_nodata -32768" will scale all pixel values, except those which are equal to the source nodata value (-9999 e.g.) and will set them to dest nodata value (-32768) "gdal_translate src_raster dst_raster -scale 0 1 0 1000 -srcnodata none -a_nodata -32768" will scale all pixel values (including those which are equal to the source nodata value) and will set -32768 as the dest nodata value "gdal_translate src_raster dst_raster -scale 0 1 0 1000 -srcnodata 12345 -a_nodata -32768" will scale all pixel values, except those which are equal to 12345 and will set -them to -32768
Do you agree with that ?
by , 15 years ago
Attachment: | ticket_3085.patch added |
---|
by , 15 years ago
Attachment: | ticket_3085.2.patch added |
---|
comment:6 by , 15 years ago
Summary: | Incorrect nodata values when using the -scale option in gdal_translate → [PATCH] Incorrect nodata values when using the -scale option in gdal_translate |
---|---|
Type: | defect → enhancement |
I won't apply the patch for the moment. New behaviour doesn't make consensus.
comment:7 by , 15 years ago
rouault
The examples you provided sound logical. I hope other folks are interested in this enhancement and it is accepted. Thanks for taking the time to look into it.
-pete
comment:8 by , 15 years ago
+1 to get this patch implemented. I am running into this very issue with the 1.6 branch trying to convert a dem from feet to meters. The nodata value is also scaled causing an incorrect min/max for the raster.
comment:9 by , 12 years ago
Cc: | added |
---|
comment:11 by , 12 years ago
I am running into similar results when I use -unscale in gdal_translate. I would think that nodata values would stay nodata values, but they become valid values and ruin my raster (stats):
C:\Users\k\Desktop\mmi_tmp\vrt_test>gdalinfo -stats vrt\mmi.vrt Driver: VRT/Virtual Raster Files: vrt\mmi.vrt csv\UShazard.20081231.pga.10pc50 Size is 1201, 509 Coordinate System is: GEOGCS["WGS 84", DATUM["WGS_1984", SPHEROID["WGS 84",6378137,298.257223563, AUTHORITY["EPSG","7030"]], AUTHORITY["EPSG","6326"]], PRIMEM["Greenwich",0, AUTHORITY["EPSG","8901"]], UNIT["degree",0.0174532925199433, AUTHORITY["EPSG","9122"]], AUTHORITY["EPSG","4326"]] Origin = (-125.025000000000010,50.024999999999999) Pixel Size = (0.050000000000000,-0.050000000000000) Corner Coordinates: Upper Left (-125.0250000, 50.0250000) (125d 1'30.00"W, 50d 1'30.00"N) Lower Left (-125.0250000, 24.5750000) (125d 1'30.00"W, 24d34'30.00"N) Upper Right ( -64.9750000, 50.0250000) ( 64d58'30.00"W, 50d 1'30.00"N) Lower Right ( -64.9750000, 24.5750000) ( 64d58'30.00"W, 24d34'30.00"N) Center ( -95.0000000, 37.3000000) ( 95d 0' 0.00"W, 37d18' 0.00"N) Band 1 Block=128x128 Type=Float32, ColorInterp=Undefined Minimum=0.005, Maximum=1.110, Mean=0.041, StdDev=0.064 NoData Value=-9999 Metadata: STATISTICS_MINIMUM=0.0049999998882413 STATISTICS_MAXIMUM=1.1102999448776 STATISTICS_MEAN=0.040921010072679 STATISTICS_STDDEV=0.064469886712302 C:\Users\k\Desktop\mmi_tmp\vrt_test>gdal_translate -unscale vrt\mmi.vrt output\mmi.tif Input file size is 1201, 509 0...10...20...30...40...50...60...70...80...90...100 - done. C:\Users\k\Desktop\mmi_tmp\vrt_test>gdalinfo -stats output\mmi.tif Driver: GTiff/GeoTIFF Files: output\mmi.tif Size is 1201, 509 Coordinate System is: GEOGCS["WGS 84", DATUM["WGS_1984", SPHEROID["WGS 84",6378137,298.257223563, AUTHORITY["EPSG","7030"]], AUTHORITY["EPSG","6326"]], PRIMEM["Greenwich",0], UNIT["degree",0.0174532925199433], AUTHORITY["EPSG","4326"]] Origin = (-125.025000000000010,50.024999999999999) Pixel Size = (0.050000000000000,-0.050000000000000) Metadata: AREA_OR_POINT=Area Image Structure Metadata: INTERLEAVE=BAND Corner Coordinates: Upper Left (-125.0250000, 50.0250000) (125d 1'30.00"W, 50d 1'30.00"N) Lower Left (-125.0250000, 24.5750000) (125d 1'30.00"W, 24d34'30.00"N) Upper Right ( -64.9750000, 50.0250000) ( 64d58'30.00"W, 50d 1'30.00"N) Lower Right ( -64.9750000, 24.5750000) ( 64d58'30.00"W, 24d34'30.00"N) Center ( -95.0000000, 37.3000000) ( 95d 0' 0.00"W, 37d18' 0.00"N) Band 1 Block=1201x1 Type=Float32, ColorInterp=Gray Minimum=0.005, Maximum=1.110, Mean=0.041, StdDev=0.064 NoData Value=-9999 Metadata: STATISTICS_MINIMUM=0.0049999998882413 STATISTICS_MAXIMUM=1.1102999448776 STATISTICS_MEAN=0.040921010072679 STATISTICS_STDDEV=0.064469886712297
I don't believe this is expected. Using 1.8.1 from Tamas' site.
comment:12 by , 12 years ago
Sorry, I don't get it for a single band raster, but I do with a multiband raster, I will try to hunt down my issue.
comment:13 by , 12 years ago
I am getting the same issue with -unscale. I understand the default behavior should be to apply scale and offset to all data, regardless of the nodata value. Is it possible to add a config option? It doesn't seem like it fits the scope of most config options.
comment:14 by , 9 years ago
+1 to get this patch implemented, or some option to keep NODATA values unchanged by the -scale operation.
I am running into this issue in GDAL 1.9.0. Use-case: a free Lake Tahoe DEM is published as a Float32 raster with elevations in feet, and NODATA value is -32767.0. I translated it to meters by using
-scale 0 100000 0 30480.
But then, the NODATA value remains -32767.0, while the pixels that used to be undefined now contain -9987.3816, so no longer undefined. I suppose I could specify
-a_nodata -9987.3816,
but I dislike having this silly number as the NODATA value, and I worry that it is not representable exactly as a Float32 number, so a mismatch might occur.
Actually, I don't know what to do. How can I use GDAL Utilities to convert the elevation raster to meters, while keeping undefined values as undefined, and represented by an integer NODATA value (or represented by NaN, perhaps)?
Slightly related question: gdalinfo reported "Unit Type: ft" for Band 1 of the original file. Very nice, that's what alerted me (especially since the official document for the data says it's in meters!). However, after rescaling to meters with gdal_translate, the metadata "Unit Type: ft" remains. I understand that these metadata are not interpreted by gdal_translate, but how would I change it or remove it?
comment:15 by , 9 years ago
trunk r27928 gdal_translate: don't recopy band units if rescaling or unscaling is involved (mentionned in comment 14 of #3085)" (you can probably unset it by calling SetUnitType("") on the band)
Regarding the patch, I don't remember why I wrote "I won't apply the patch for the moment. New behaviour doesn't make consensus" at that time. But would probably be material appropriate for GDAL 2.0
comment:16 by , 9 years ago
Thanks, Even.
For the record, I managed to do a workaround. First, I did:
gdalwarp -dstnodata -10000 tahoedems.tif tahoedems_nodata_m10k.tif
This will set the new NODATA value to be -10000.0, and pixels that used to contain the old NODATA value -32767.0 will be replaced by -10000.0.
For some reason, it seems that gdalwarp removes the "Unit Type: ft" metadata. I suppose this can be regarded as a bug in gdalwarp, but it's good for me!
Then, I did
gdal_translate -scale 0 100000 0 30480 -a_nodata -3048 tahoedems_nodata_m10k.tif tahoedems_meters.tif
This will convert from feet to meters, and pixels containing the undefined -10000.0 feet are converted to -3048.0 meters. It is necessary to specify -a_nodata -3048 in the command. Fortunately, -3048.0 is an integer, and as far as I can tell, the rescaling makes the pixel values exactly -3048.0. I can certainly live with -3048 as the undefined value for elevations (the original -32767 does not make more sense).
comment:17 by , 7 years ago
Resolution: | → fixed |
---|---|
Status: | assigned → closed |
trunk r37633 "gdal_translate: set nodata value on vrtsource on scale / unscale / expand cases (patch by Zac Flamig, https://github.com/OSGeo/gdal/pull/199)"
Peter / Even,
I don't really see the issue. Did you anticipate that gdal_translate would somehow treat nodata values specially, and not rescale them? Did you think that -a_nodata would alter the actual pixel values?