Opened 17 years ago
Closed 16 years ago
#1625 closed defect (fixed)
problems reading etopo2v2 (arc grid, / GMT (netcdf) /HDF)
Reported by: | jonmu | Owned by: | Mateusz Łoskot |
---|---|---|---|
Priority: | normal | Milestone: | 1.4.4 |
Component: | GDAL_Raster | Version: | 1.4.1 |
Severity: | normal | Keywords: | hdf gmt etopo2 netcdf |
Cc: | warmerdam |
Description
The Etopo2 data set is a 2 minute topography/bathymetry grid covering the entire world provided by NGDC.
http://www.ngdc.noaa.gov/mgg/fliers/01mgg04.html
Download: http://www.ngdc.noaa.gov/mgg/global/relief/ETOPO2/ETOPO2v2-2006/
Version 2 of this grid comes in several different formats: I have tried to import them using the GDAL library, and it does not seem to work.
I have also tried to import the grids using the GW tools (file->import) but i seems to crash the program.
This is what happens: GMT: poDataset->GetRasterXSize(); poDataset->GetRasterYSize(); poDataset->GetRasterCount(); returns wrong values.
ARC, and HDF does not have the same problem. The values returned from the GDALDataset seem to be correct, but when i try to read the array using poDataset->RasterIO(...) it does not work, it returns errors.
Do anyone else have the same problem with these grids?
Attachments (1)
Change History (16)
comment:1 by , 17 years ago
Cc: | added |
---|---|
Component: | default → GDAL_Raster |
Milestone: | → 1.4.2 |
Owner: | changed from | to
comment:3 by , 16 years ago
Keywords: | hdf gmt etopo2 added |
---|---|
Status: | new → assigned |
comment:4 by , 16 years ago
jonmu,
I have a couple of questions:
- could you precise what dataset file exactly you used?
- what you mean saying you were trying to import them?
- could you provide me with exact command you used?
- what do you mean as GW tools?
- what you mean as GMT in This is what happens: GMT:? Is this GMT dataset or GMT tools (I'm quite sure it's the former, but it's always better to ask :-))?
Thanks in advance for your assistance!
follow-up: 8 comment:5 by , 16 years ago
Milestone: | 1.4.3 → 1.4.4 |
---|
Pushing off to 1.4.4 for lack of information.
follow-up: 9 comment:6 by , 16 years ago
Keywords: | netcdf added |
---|
Short tests of sample ETOPO2 grid ETOPO2v2g_f4_netCDF.zip using trunk and branches/1.4 give different results:
- using GDAL trunk
~/data/etopo2/ETOPO2v2-2006/ETOPO2v2g/netCDF $ ~/dev/gdal/_svn/trunk/gdal/apps/gdalinfo -stats ETOPO2v2g_f4.nc GDAL_netCDF: dim_count = 2 GDAL_netCDF: var_count = 3 GDAL_netCDF: NETCDFFilename = NETCDF:"ETOPO2v2g_f4.nc":z GDAL: GDALOpen(ETOPO2v2g_f4.nc) succeeds as netCDF. Driver: netCDF/Network Common Data Format Files: ETOPO2v2g_f4.nc ETOPO2v2g_f4.nc.aux.xml Size is 10801, 5401 Coordinate System is `' Origin = (-180.016666666666680,-90.016666666666666) Pixel Size = (0.033333333333333,0.033333333333333) Metadata: NC_GLOBAL#Conventions=COARDS NC_GLOBAL#title=z NC_GLOBAL#history=grdreformat ETOPO2v2g_GMT_int.grd ETOPO2v2g_GMT_flt2.grd NC_GLOBAL#node_offset=0 z#long_name=z z#_FillValue=nan z#actual_range=-1.072200e+048.046000e+03 x#long_name=x x#actual_range=-180180 y#long_name=y y#actual_range=-9090 Corner Coordinates: Upper Left (-180.0166667, -90.0166667) Lower Left (-180.0166667, 90.0166667) Upper Right ( 180.0166667, -90.0166667) Lower Right ( 180.0166667, 90.0166667) Center ( -0.0000000, 0.0000000) Band 1 Block=10801x1 Type=Float32, ColorInterp=Undefined Min=-10722.000 Max=8046.000 Minimum=-10722.000, Maximum=8046.000, Mean=-1887.982, StdDev=2649.789 NoData Value=nan Metadata: NETCDF_VARNAME=z STATISTICS_MINIMUM=-10722 STATISTICS_MAXIMUM=8046 STATISTICS_MEAN=-1887.9822298507 STATISTICS_STDDEV=2649.7888386366 GDAL: GDALClose(ETOPO2v2g_f4.nc) GDAL: GDALDeregister_GTiff() called. ~/data/etopo2/ETOPO2v2-2006/ETOPO2v2g/netCDF $ otool -L ~/dev/gdal/_svn/trunk/gdal/apps/gdalinfo /Users/mloskot/dev/gdal/_svn/trunk/gdal/apps/gdalinfo: /Users/mloskot/dev/gdal/_svn/trunk/gdal/libgdal.dylib (compatibility version 0.0.0, current version 0.0.0) ...
- using branches/1.4 (upcoming 1.4.3)
~/data/etopo2/ETOPO2v2-2006/ETOPO2v2g/netCDF $ ~/dev/gdal/_svn/branches/1.4/gdal/apps/gdalinfo -stats ETOPO2v2g_f4.nc GDAL_netCDF: dim_count = 2 GDAL_netCDF: var_count = 3 GDAL: GDALOpen(ETOPO2v2g_f4.nc) succeeds as netCDF. Driver: netCDF/Network Common Data Format Size is 512, 512 Coordinate System is `' Metadata: NC_GLOBAL#Conventions=COARDS NC_GLOBAL#title=z NC_GLOBAL#history=grdreformat ETOPO2v2g_GMT_int.grd ETOPO2v2g_GMT_flt2.grd NC_GLOBAL#node_offset=0 Subdatasets: SUBDATASET_1_NAME=NETCDF:"ETOPO2v2g_f4.nc":z SUBDATASET_1_DESC=[5401x10801] z (32-bit floating-point) Corner Coordinates: Upper Left ( 0.0, 0.0) Lower Left ( 0.0, 512.0) Upper Right ( 512.0, 0.0) Lower Right ( 512.0, 512.0) Center ( 256.0, 256.0) GDAL: GDALClose(ETOPO2v2g_f4.nc) GDAL: GDALDeregister_GTiff() called. ~/data/etopo2/ETOPO2v2-2006/ETOPO2v2g/netCDF $ otool -L ~/dev/gdal/_svn/branches/1.4/gdal/apps/gdalinfo /Users/mloskot/dev/gdal/_svn/branches/1.4/gdal/apps/gdalinfo: /Users/mloskot/dev/gdal/_svn/branches/1.4/gdal/libgdal.dylib (compatibility version 0.0.0, current version 0.0.0) ...
I have not noticed any crash.
According to my knowledge about status of the netCDF driver, these difference are explainable, because recently there have been a lot of changes and fixes applied to this driver.
Also, I've managed to import the ETOPO2v2g_f4_netCDF.zip file to OpenEV using File -> Import (using FWTools 1.3.9). See screenshot attached.
comment:7 by , 16 years ago
Mateusz,
I concur, everything here looks fine. If we don't hear back in a week or two with details to reproduce the original problem we should just close this.
comment:8 by , 16 years ago
Replying to warmerdam:
Pushing off to 1.4.4 for lack of information.
Hi, thanks for looking at my problem. I will give you a more precise description of whats wrong. I will answer your questions first:
could you precise what dataset file exactly you used?
I tried many of them, none of them worked. I will come back to you with concrete examples.
what you mean saying you were trying to import them?
The different formats stopped working at different stages. I will describe each of them.
could you provide me with exact command you used?
Yes I can, I will get back to you on this.
what do you mean as GW tools?
It has been a while since I looked at this, but I think i downloaded some of the utilities from the GDAL page, which had problems with the same files.
what you mean as GMT in This is what happens: GMT:? Is this GMT dataset or GMT tools (I'm quite sure it's the former, but it's always better to ask :-))?
I meant a GMT netcdf dataset. I have not used the GMT Tools.
Thanks in advance for your assistance!
follow-up: 15 comment:9 by , 16 years ago
When trying to read this file (ETOPO2v2g_f4_netCDF mentioned above my mloskot) I get the same error as he mentions in branches/1.4 (upcoming 1.4.3)
GDALDataset* poDataset; poDataset = (GDALDataset* ) GDALOpen(filename, GA_ReadOnly); ok so far... but then: int xsize = poDataset->GetRasterXSize(); xsize == 512 ...which is wrong int ysize = poDataset->GetRasterYSize(); ysize == 512 ...also wrong int count = poDataset->GetRasterCount(); count == 0 should be 1
gdal trunk seems to return a better result. This is a dataset covering the entire world with 2 minute grid, in lat lon coords, so the report from this code seems more reasonable:
(From 10/27/07 18:53:57 changed by mloskot )
Size is 10801, 5401 Upper Left (-180.0166667, -90.0166667) Lower Left (-180.0166667, 90.0166667) Upper Right ( 180.0166667, -90.0166667) Lower Right ( 180.0166667, 90.0166667) Center ( -0.0000000, 0.0000000)
This is correct!
Size is 512, 512 Upper Left ( 0.0, 0.0) Lower Left ( 0.0, 512.0) Upper Right ( 512.0, 0.0) Lower Right ( 512.0, 512.0) Center ( 256.0, 256.0)
This is wrong.
Replying to mloskot:
Short tests of sample ETOPO2 grid ETOPO2v2g_f4_netCDF.zip using trunk and branches/1.4 give different results:
- using GDAL trunk
~/data/etopo2/ETOPO2v2-2006/ETOPO2v2g/netCDF $ ~/dev/gdal/_svn/trunk/gdal/apps/gdalinfo -stats ETOPO2v2g_f4.nc GDAL_netCDF: dim_count = 2 GDAL_netCDF: var_count = 3 GDAL_netCDF: NETCDFFilename = NETCDF:"ETOPO2v2g_f4.nc":z GDAL: GDALOpen(ETOPO2v2g_f4.nc) succeeds as netCDF. Driver: netCDF/Network Common Data Format Files: ETOPO2v2g_f4.nc ETOPO2v2g_f4.nc.aux.xml Size is 10801, 5401 Coordinate System is `' Origin = (-180.016666666666680,-90.016666666666666) Pixel Size = (0.033333333333333,0.033333333333333) Metadata: NC_GLOBAL#Conventions=COARDS NC_GLOBAL#title=z NC_GLOBAL#history=grdreformat ETOPO2v2g_GMT_int.grd ETOPO2v2g_GMT_flt2.grd NC_GLOBAL#node_offset=0 z#long_name=z z#_FillValue=nan z#actual_range=-1.072200e+048.046000e+03 x#long_name=x x#actual_range=-180180 y#long_name=y y#actual_range=-9090 Corner Coordinates: Upper Left (-180.0166667, -90.0166667) Lower Left (-180.0166667, 90.0166667) Upper Right ( 180.0166667, -90.0166667) Lower Right ( 180.0166667, 90.0166667) Center ( -0.0000000, 0.0000000) Band 1 Block=10801x1 Type=Float32, ColorInterp=Undefined Min=-10722.000 Max=8046.000 Minimum=-10722.000, Maximum=8046.000, Mean=-1887.982, StdDev=2649.789 NoData Value=nan Metadata: NETCDF_VARNAME=z STATISTICS_MINIMUM=-10722 STATISTICS_MAXIMUM=8046 STATISTICS_MEAN=-1887.9822298507 STATISTICS_STDDEV=2649.7888386366 GDAL: GDALClose(ETOPO2v2g_f4.nc) GDAL: GDALDeregister_GTiff() called. ~/data/etopo2/ETOPO2v2-2006/ETOPO2v2g/netCDF $ otool -L ~/dev/gdal/_svn/trunk/gdal/apps/gdalinfo /Users/mloskot/dev/gdal/_svn/trunk/gdal/apps/gdalinfo: /Users/mloskot/dev/gdal/_svn/trunk/gdal/libgdal.dylib (compatibility version 0.0.0, current version 0.0.0) ...
- using branches/1.4 (upcoming 1.4.3)
~/data/etopo2/ETOPO2v2-2006/ETOPO2v2g/netCDF $ ~/dev/gdal/_svn/branches/1.4/gdal/apps/gdalinfo -stats ETOPO2v2g_f4.nc GDAL_netCDF: dim_count = 2 GDAL_netCDF: var_count = 3 GDAL: GDALOpen(ETOPO2v2g_f4.nc) succeeds as netCDF. Driver: netCDF/Network Common Data Format Size is 512, 512 Coordinate System is `' Metadata: NC_GLOBAL#Conventions=COARDS NC_GLOBAL#title=z NC_GLOBAL#history=grdreformat ETOPO2v2g_GMT_int.grd ETOPO2v2g_GMT_flt2.grd NC_GLOBAL#node_offset=0 Subdatasets: SUBDATASET_1_NAME=NETCDF:"ETOPO2v2g_f4.nc":z SUBDATASET_1_DESC=[5401x10801] z (32-bit floating-point) Corner Coordinates: Upper Left ( 0.0, 0.0) Lower Left ( 0.0, 512.0) Upper Right ( 512.0, 0.0) Lower Right ( 512.0, 512.0) Center ( 256.0, 256.0) GDAL: GDALClose(ETOPO2v2g_f4.nc) GDAL: GDALDeregister_GTiff() called. ~/data/etopo2/ETOPO2v2-2006/ETOPO2v2g/netCDF $ otool -L ~/dev/gdal/_svn/branches/1.4/gdal/apps/gdalinfo /Users/mloskot/dev/gdal/_svn/branches/1.4/gdal/apps/gdalinfo: /Users/mloskot/dev/gdal/_svn/branches/1.4/gdal/libgdal.dylib (compatibility version 0.0.0, current version 0.0.0) ...I have not noticed any crash.
According to my knowledge about status of the netCDF driver, these difference are explainable, because recently there have been a lot of changes and fixes applied to this driver.
Also, I've managed to import the ETOPO2v2g_f4_netCDF.zip file to OpenEV using File -> Import (using FWTools 1.3.9). See screenshot attached.
comment:10 by , 16 years ago
I see from the etopo2 readme.txt that they refer to the netcdf file as a: "floating pt grid in COARDS-compliant netCDF Format", not as a GMT netcdf format... I am not sure if this is supported by GDAL.
comment:11 by , 16 years ago
I tried to read the ARC grids, ETOPO2v2g_f4_LSB
The zip file contains a headerfile in ARC format: ETOPO2v2g_f4_LSB.hdr, and a raw binary file containing the data: ETOPO2v2g_f4_LSB.flt
GDALDataset* poDataset = (GDALDataset* ) GDALOpen(filename, GA_ReadOnly); //is ok int xsize = poDataset->GetRasterXSize(); //is ok: 10801 int ysize = poDataset->GetRasterYSize(); //is ok: 5401 int count = poDataset->GetRasterCount(); //is ok: 1 GDALDataType first_band_type = poDataset->GetRasterBand(1)->GetRasterDataType(); //wrong: first_band_type is now GDT_BYTE, file contains 4byte float values
the header file is listed here:
NCOLS 10801 NROWS 5401 XLLCENTER -180.00000 YLLCENTER -90.00000 CELLSIZE 0.03333333333 NODATA_VALUE 999999 BYTEORDER LSBFIRST NUMBERTYPE 4_BYTE_FLOAT UNITS METERS MIN_VALUE -10722.0 MAX_VALUE 8046.0
comment:12 by , 16 years ago
COARDS compliant netcdf files should be supported by the netcdf driver (as opposed to gmt driver which is a very specific netcdf product).
But if this bug report is really about the arc file, then it seems a poor bug report resulted in Mateusz wasting time on unrelated issues.
BTW, I've never seen a EHDR file with NUMBERTYPE 4_BYTE_FLOAT.
comment:13 by , 16 years ago
If NUMBERTYPE 4_BYTE_FLOAT is not standard, then that is probably the problem with the ARC grids.
The problem with the netcdf file can be seen in mloskot's short test above.
The values from GDAL trunk reports values which seem ok.
branches/1.4 (upcoming 1.4.3) return different values, probably caused by wrong output from :
poDataset->GetRasterXSize();
poDataset->GetRasterYSize();
and
poDataset->GetRasterCount();
getRasterZSize returns 512, should be 10801
getRasterYSize returns 512, should be 5401
getRasterCount returns 0, should be 1
He mentions that there has been a some changes and bugfixes on the netcdf driver. If this is the reason for the bug i experienced, then I guess this is OK now.
comment:14 by , 16 years ago
If you are interested in non standard headers, the integer version of the same ARC grid:ETOPO2v2g_i2_LSB has a header which looks like this
NCOLS 10801 NROWS 5401 XLLCENTER -180.00000 YLLCENTER -90.00000 CELLSIZE 0.03333333333 NODATA_VALUE -32768 BYTEORDER LSBFIRST NUMBERTYPE 2_BYTE_INTEGER UNITS METERS MIN_VALUE -10722 MAX_VALUE 8046
I guess NUMBERTYPE 2_BYTE_INTEGER is also not standard.
comment:15 by , 16 years ago
Resolution: | → fixed |
---|---|
Status: | assigned → closed |
Replying to jonmu:
(From 10/27/07 18:53:57 changed by mloskot )
Size is 10801, 5401 Upper Left (-180.0166667, -90.0166667) Lower Left (-180.0166667, 90.0166667) Upper Right ( 180.0166667, -90.0166667) Lower Right ( 180.0166667, 90.0166667) Center ( -0.0000000, 0.0000000)
This is correct!
Size is 512, 512 Upper Left ( 0.0, 0.0) Lower Left ( 0.0, 512.0) Upper Right ( 512.0, 0.0) Lower Right ( 512.0, 512.0) Center ( 256.0, 256.0)
This is wrong.
The latter incorrect result is produced by GDAL 1.4.x which I assume does not include recent fixes in NetCDF driver. The former result confirms the problem has been fixed in current development line, so the fix will be included in 1.5.0.
Regarding the non standard headers, it looks to me like a good candidate to enhancement request as a separate ticket.
I'm closing this ticket as fixed.
Mateusz,
Can you try and isolate the possibly one or more bugs here and do some preliminary analysis?