Ticket #1625 (closed defect: fixed)

Opened 1 year ago

Last modified 9 months ago

problems reading etopo2v2 (arc grid, / GMT (netcdf) /HDF)

Reported by: jonmu Assigned to: mloskot
Priority: normal Milestone: 1.4.4
Component: GDAL_Raster Version: 1.4.1
Severity: normal Keywords: hdf gmt etopo2 netcdf
Cc: warmerdam

Description

The Etopo2 data set is a 2 minute topography/bathymetry grid covering the entire world provided by NGDC.

http://www.ngdc.noaa.gov/mgg/fliers/01mgg04.html

Download: http://www.ngdc.noaa.gov/mgg/global/relief/ETOPO2/ETOPO2v2-2006/

Version 2 of this grid comes in several different formats: I have tried to import them using the GDAL library, and it does not seem to work.

I have also tried to import the grids using the GW tools (file->import) but i seems to crash the program.

This is what happens: GMT: poDataset->GetRasterXSize(); poDataset->GetRasterYSize(); poDataset->GetRasterCount?(); returns wrong values.

ARC, and HDF does not have the same problem. The values returned from the GDALDataset seem to be correct, but when i try to read the array using poDataset->RasterIO(...) it does not work, it returns errors.

Do anyone else have the same problem with these grids?

Attachments

openev-etopo2-imported.2.png (187.5 kB) - added by mloskot on 10/27/07 19:23:21.
ETOPO2 grid imported to OpenEV

Change History

05/10/07 10:14:30 changed by warmerdam

  • owner changed from warmerdam to mloskot.
  • cc set to warmerdam.
  • component changed from default to GDAL_Raster.
  • milestone set to 1.4.2.

Mateusz,

Can you try and isolate the possibly one or more bugs here and do some preliminary analysis?

06/25/07 12:34:58 changed by warmerdam

  • milestone changed from 1.4.2 to 1.4.3.

Put off till 1.4.3.

10/26/07 01:29:12 changed by mloskot

  • keywords set to hdf gmt etopo2.
  • status changed from new to assigned.

10/27/07 02:21:25 changed by mloskot

jonmu,

I have a couple of questions:

  • could you precise what dataset file exactly you used?
  • what you mean saying you were trying to import them?
  • could you provide me with exact command you used?
  • what do you mean as GW tools?
  • what you mean as GMT in This is what happens: GMT:? Is this GMT dataset or GMT tools (I'm quite sure it's the former, but it's always better to ask :-))?

Thanks in advance for your assistance!

(follow-up: ↓ 8 ) 10/27/07 18:22:09 changed by warmerdam

  • milestone changed from 1.4.3 to 1.4.4.

Pushing off to 1.4.4 for lack of information.

(follow-up: ↓ 9 ) 10/27/07 18:53:57 changed by mloskot

  • keywords changed from hdf gmt etopo2 to hdf gmt etopo2 netcdf.

Short tests of sample ETOPO2 grid ETOPO2v2g_f4_netCDF.zip using trunk and branches/1.4 give different results:

  • using GDAL trunk 
~/data/etopo2/ETOPO2v2-2006/ETOPO2v2g/netCDF $ ~/dev/gdal/_svn/trunk/gdal/apps/gdalinfo -stats ETOPO2v2g_f4.nc 
GDAL_netCDF: dim_count = 2

GDAL_netCDF: var_count = 3

GDAL_netCDF: NETCDFFilename = NETCDF:"ETOPO2v2g_f4.nc":z

GDAL: GDALOpen(ETOPO2v2g_f4.nc) succeeds as netCDF.
Driver: netCDF/Network Common Data Format
Files: ETOPO2v2g_f4.nc
       ETOPO2v2g_f4.nc.aux.xml
Size is 10801, 5401
Coordinate System is `'
Origin = (-180.016666666666680,-90.016666666666666)
Pixel Size = (0.033333333333333,0.033333333333333)
Metadata:
  NC_GLOBAL#Conventions=COARDS
  NC_GLOBAL#title=z
  NC_GLOBAL#history=grdreformat ETOPO2v2g_GMT_int.grd ETOPO2v2g_GMT_flt2.grd
  NC_GLOBAL#node_offset=0
  z#long_name=z
  z#_FillValue=nan
  z#actual_range=-1.072200e+048.046000e+03
  x#long_name=x
  x#actual_range=-180180
  y#long_name=y
  y#actual_range=-9090
Corner Coordinates:
Upper Left  (-180.0166667, -90.0166667) 
Lower Left  (-180.0166667,  90.0166667) 
Upper Right ( 180.0166667, -90.0166667) 
Lower Right ( 180.0166667,  90.0166667) 
Center      (  -0.0000000,   0.0000000) 
Band 1 Block=10801x1 Type=Float32, ColorInterp=Undefined
  Min=-10722.000 Max=8046.000 
  Minimum=-10722.000, Maximum=8046.000, Mean=-1887.982, StdDev=2649.789
  NoData Value=nan
  Metadata:
    NETCDF_VARNAME=z
    STATISTICS_MINIMUM=-10722
    STATISTICS_MAXIMUM=8046
    STATISTICS_MEAN=-1887.9822298507
    STATISTICS_STDDEV=2649.7888386366
GDAL: GDALClose(ETOPO2v2g_f4.nc)
GDAL: GDALDeregister_GTiff() called.


~/data/etopo2/ETOPO2v2-2006/ETOPO2v2g/netCDF $ otool -L ~/dev/gdal/_svn/trunk/gdal/apps/gdalinfo
/Users/mloskot/dev/gdal/_svn/trunk/gdal/apps/gdalinfo:
        /Users/mloskot/dev/gdal/_svn/trunk/gdal/libgdal.dylib (compatibility version 0.0.0, current version 0.0.0)
...
  • using branches/1.4 (upcoming 1.4.3) 
~/data/etopo2/ETOPO2v2-2006/ETOPO2v2g/netCDF $ ~/dev/gdal/_svn/branches/1.4/gdal/apps/gdalinfo -stats ETOPO2v2g_f4.nc
GDAL_netCDF: dim_count = 2

GDAL_netCDF: var_count = 3

GDAL: GDALOpen(ETOPO2v2g_f4.nc) succeeds as netCDF.
Driver: netCDF/Network Common Data Format
Size is 512, 512
Coordinate System is `'
Metadata:
  NC_GLOBAL#Conventions=COARDS
  NC_GLOBAL#title=z
  NC_GLOBAL#history=grdreformat ETOPO2v2g_GMT_int.grd ETOPO2v2g_GMT_flt2.grd
  NC_GLOBAL#node_offset=0
Subdatasets:
  SUBDATASET_1_NAME=NETCDF:"ETOPO2v2g_f4.nc":z
  SUBDATASET_1_DESC=[5401x10801] z (32-bit floating-point)
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  512.0)
Upper Right (  512.0,    0.0)
Lower Right (  512.0,  512.0)
Center      (  256.0,  256.0)
GDAL: GDALClose(ETOPO2v2g_f4.nc)
GDAL: GDALDeregister_GTiff() called.

     
~/data/etopo2/ETOPO2v2-2006/ETOPO2v2g/netCDF $ otool -L ~/dev/gdal/_svn/branches/1.4/gdal/apps/gdalinfo
/Users/mloskot/dev/gdal/_svn/branches/1.4/gdal/apps/gdalinfo:
        /Users/mloskot/dev/gdal/_svn/branches/1.4/gdal/libgdal.dylib (compatibility version 0.0.0, current version 0.0.0)
...

I have not noticed any crash.

According to my knowledge about status of the netCDF driver, these difference are explainable, because recently there have been a lot of changes and fixes applied to this driver.

Also, I've managed to import the ETOPO2v2g_f4_netCDF.zip file to OpenEV using File -> Import (using FWTools 1.3.9). See screenshot attached.

10/27/07 19:23:21 changed by mloskot

  • attachment openev-etopo2-imported.2.png added.

ETOPO2 grid imported to OpenEV

10/27/07 21:28:05 changed by warmerdam

Mateusz,

I concur, everything here looks fine. If we don't hear back in a week or two with details to reproduce the original problem we should just close this.

(in reply to: ↑ 5 ) 10/29/07 06:51:09 changed by jonmu

Replying to warmerdam:

Pushing off to 1.4.4 for lack of information.

Hi, thanks for looking at my problem. I will give you a more precise description of whats wrong. I will answer your questions first:

could you precise what dataset file exactly you used?

I tried many of them, none of them worked. I will come back to you with concrete examples.

what you mean saying you were trying to import them?

The different formats stopped working at different stages. I will describe each of them.

could you provide me with exact command you used?

Yes I can, I will get back to you on this.

what do you mean as GW tools?

It has been a while since I looked at this, but I think i downloaded some of the utilities from the GDAL page, which had problems with the same files.

what you mean as GMT in This is what happens: GMT:? Is this GMT dataset or GMT tools (I'm quite sure it's the former, but it's always better to ask :-))?

I meant a GMT netcdf dataset. I have not used the GMT Tools.

Thanks in advance for your assistance!

(in reply to: ↑ 6 ; follow-up: ↓ 15 ) 10/29/07 09:37:16 changed by jonmu

When trying to read this file (ETOPO2v2g_f4_netCDF mentioned above my mloskot) I get the same error as he mentions in branches/1.4 (upcoming 1.4.3)

GDALDataset* poDataset; poDataset = (GDALDataset* ) GDALOpen(filename, GA_ReadOnly); //ok so far... but then: int xsize = poDataset->GetRasterXSize(); //xsize == 512 ...which is wrong int ysize = poDataset->GetRasterYSize(); //ysize == 512 ...also wrong int count = poDataset->GetRasterCount?(); //count == 0 should be 1

gdal trunk seems to return a better result. This is a dataset covering the entire world with 2 minute grid, in lat lon coords, so the report from this code seems more reasonable:

(From 10/27/07 18:53:57 changed by mloskot )

Size is 10801, 5401 Upper Left (-180.0166667, -90.0166667) Lower Left (-180.0166667, 90.0166667) Upper Right ( 180.0166667, -90.0166667) Lower Right ( 180.0166667, 90.0166667) Center ( -0.0000000, 0.0000000)

This is correct!

Size is 512, 512 Upper Left ( 0.0, 0.0) Lower Left ( 0.0, 512.0) Upper Right ( 512.0, 0.0) Lower Right ( 512.0, 512.0) Center ( 256.0, 256.0)

This is wrong.

Replying to mloskot:

Short tests of sample ETOPO2 grid ETOPO2v2g_f4_netCDF.zip using trunk and branches/1.4 give different results: * using GDAL trunk {{{ ~/data/etopo2/ETOPO2v2-2006/ETOPO2v2g/netCDF $ ~/dev/gdal/_svn/trunk/gdal/apps/gdalinfo -stats ETOPO2v2g_f4.nc GDAL_netCDF: dim_count = 2 GDAL_netCDF: var_count = 3 GDAL_netCDF: NETCDFFilename = NETCDF:"ETOPO2v2g_f4.nc":z GDAL: GDALOpen(ETOPO2v2g_f4.nc) succeeds as netCDF. Driver: netCDF/Network Common Data Format Files: ETOPO2v2g_f4.nc ETOPO2v2g_f4.nc.aux.xml Size is 10801, 5401 Coordinate System is `' Origin = (-180.016666666666680,-90.016666666666666) Pixel Size = (0.033333333333333,0.033333333333333) Metadata: NC_GLOBAL#Conventions=COARDS NC_GLOBAL#title=z NC_GLOBAL#history=grdreformat ETOPO2v2g_GMT_int.grd ETOPO2v2g_GMT_flt2.grd NC_GLOBAL#node_offset=0 z#long_name=z z#_FillValue=nan z#actual_range=-1.072200e+048.046000e+03 x#long_name=x x#actual_range=-180180 y#long_name=y y#actual_range=-9090 Corner Coordinates: Upper Left (-180.0166667, -90.0166667) Lower Left (-180.0166667, 90.0166667) Upper Right ( 180.0166667, -90.0166667) Lower Right ( 180.0166667, 90.0166667) Center ( -0.0000000, 0.0000000) Band 1 Block=10801x1 Type=Float32, ColorInterp?=Undefined Min=-10722.000 Max=8046.000 Minimum=-10722.000, Maximum=8046.000, Mean=-1887.982, StdDev?=2649.789 NoData? Value=nan Metadata: NETCDF_VARNAME=z STATISTICS_MINIMUM=-10722 STATISTICS_MAXIMUM=8046 STATISTICS_MEAN=-1887.9822298507 STATISTICS_STDDEV=2649.7888386366 GDAL: GDALClose(ETOPO2v2g_f4.nc) GDAL: GDALDeregister_GTiff() called. ~/data/etopo2/ETOPO2v2-2006/ETOPO2v2g/netCDF $ otool -L ~/dev/gdal/_svn/trunk/gdal/apps/gdalinfo /Users/mloskot/dev/gdal/_svn/trunk/gdal/apps/gdalinfo: /Users/mloskot/dev/gdal/_svn/trunk/gdal/libgdal.dylib (compatibility version 0.0.0, current version 0.0.0) ... }}} * using branches/1.4 (upcoming 1.4.3) {{{ ~/data/etopo2/ETOPO2v2-2006/ETOPO2v2g/netCDF $ ~/dev/gdal/_svn/branches/1.4/gdal/apps/gdalinfo -stats ETOPO2v2g_f4.nc GDAL_netCDF: dim_count = 2 GDAL_netCDF: var_count = 3 GDAL: GDALOpen(ETOPO2v2g_f4.nc) succeeds as netCDF. Driver: netCDF/Network Common Data Format Size is 512, 512 Coordinate System is `' Metadata: NC_GLOBAL#Conventions=COARDS NC_GLOBAL#title=z NC_GLOBAL#history=grdreformat ETOPO2v2g_GMT_int.grd ETOPO2v2g_GMT_flt2.grd NC_GLOBAL#node_offset=0 Subdatasets: SUBDATASET_1_NAME=NETCDF:"ETOPO2v2g_f4.nc":z SUBDATASET_1_DESC=[5401x10801] z (32-bit floating-point) Corner Coordinates: Upper Left ( 0.0, 0.0) Lower Left ( 0.0, 512.0) Upper Right ( 512.0, 0.0) Lower Right ( 512.0, 512.0) Center ( 256.0, 256.0) GDAL: GDALClose(ETOPO2v2g_f4.nc) GDAL: GDALDeregister_GTiff() called. ~/data/etopo2/ETOPO2v2-2006/ETOPO2v2g/netCDF $ otool -L ~/dev/gdal/_svn/branches/1.4/gdal/apps/gdalinfo /Users/mloskot/dev/gdal/_svn/branches/1.4/gdal/apps/gdalinfo: /Users/mloskot/dev/gdal/_svn/branches/1.4/gdal/libgdal.dylib (compatibility version 0.0.0, current version 0.0.0) ... }}} I have not noticed any crash. According to my knowledge about status of the netCDF driver, these difference are explainable, because recently there have been a lot of changes and fixes applied to this driver. Also, I've managed to import the ETOPO2v2g_f4_netCDF.zip file to OpenEV using File -> Import (using FWTools 1.3.9). See screenshot attached.

10/29/07 11:03:16 changed by jonmu

I see from the etopo2 readme.txt that they refer to the netcdf file as a: "floating pt grid in COARDS-compliant netCDF Format", not as a GMT netcdf format... I am not sure if this is supported by GDAL.

10/29/07 11:19:21 changed by jonmu

I tried to read the ARC grids, ETOPO2v2g_f4_LSB
The zip file contains a headerfile in ARC format: ETOPO2v2g_f4_LSB.hdr, and a raw binary file containing the data: ETOPO2v2g_f4_LSB.flt 

GDALDataset* poDataset = (GDALDataset* ) GDALOpen(filename, GA_ReadOnly); //is ok
int xsize = poDataset->GetRasterXSize();  //is ok: 10801
int ysize = poDataset->GetRasterYSize();  //is ok:  5401
int count = poDataset->GetRasterCount();  //is ok:  1
GDALDataType first_band_type = poDataset->GetRasterBand(1)->GetRasterDataType();
//wrong: first_band_type is now GDT_BYTE, file contains 4byte float values

the header file is listed here: 

NCOLS 10801
NROWS 5401
XLLCENTER -180.00000
YLLCENTER -90.00000
CELLSIZE 0.03333333333
NODATA_VALUE 999999
BYTEORDER LSBFIRST
NUMBERTYPE 4_BYTE_FLOAT
UNITS METERS
MIN_VALUE -10722.0
MAX_VALUE 8046.0

10/29/07 11:28:20 changed by warmerdam

COARDS compliant netcdf files should be supported by the netcdf driver (as opposed to gmt driver which is a very specific netcdf product).

But if this bug report is really about the arc file, then it seems a poor bug report resulted in Mateusz wasting time on unrelated issues.

BTW, I've never seen a EHDR file with NUMBERTYPE 4_BYTE_FLOAT.

10/29/07 12:11:45 changed by jonmu

If NUMBERTYPE 4_BYTE_FLOAT is not standard, then that is probably the problem with the ARC grids.

The problem with the netcdf file can be seen in mloskot's short test above. The values from GDAL trunk reports values which seem ok.

branches/1.4 (upcoming 1.4.3) return different values, probably caused by wrong output from :
poDataset->GetRasterXSize();
poDataset->GetRasterYSize();
and
poDataset->GetRasterCount?();


getRasterZSize returns 512, should be 10801
getRasterYSize returns 512, should be 5401
getRasterCount returns 0, should be 1


He mentions that there has been a some changes and bugfixes on the netcdf driver. If this is the reason for the bug i experienced, then I guess this is OK now.

10/30/07 08:17:37 changed by jonmu

If you are interested in non standard headers, the integer version of the same ARC grid:ETOPO2v2g_i2_LSB has a header which looks like this 

NCOLS 10801
NROWS 5401
XLLCENTER -180.00000
YLLCENTER -90.00000
CELLSIZE 0.03333333333
NODATA_VALUE -32768
BYTEORDER LSBFIRST
NUMBERTYPE 2_BYTE_INTEGER
UNITS METERS
MIN_VALUE -10722
MAX_VALUE 8046

I guess NUMBERTYPE 2_BYTE_INTEGER is also not standard.

(in reply to: ↑ 9 ) 11/22/07 14:23:10 changed by mloskot

  • status changed from assigned to closed.
  • resolution set to fixed.

Replying to jonmu:

(From 10/27/07 18:53:57 changed by mloskot )

Size is 10801, 5401 Upper Left (-180.0166667, -90.0166667) Lower Left (-180.0166667, 90.0166667) Upper Right ( 180.0166667, -90.0166667) Lower Right ( 180.0166667, 90.0166667) Center ( -0.0000000, 0.0000000)

This is correct!

Size is 512, 512 Upper Left ( 0.0, 0.0) Lower Left ( 0.0, 512.0) Upper Right ( 512.0, 0.0) Lower Right ( 512.0, 512.0) Center ( 256.0, 256.0)

This is wrong.

The latter incorrect result is produced by GDAL 1.4.x which I assume does not include recent fixes in NetCDF driver. The former result confirms the problem has been fixed in current development line, so the fix will be included in 1.5.0.

Regarding the non standard headers, it looks to me like a good candidate to enhancement request as a separate ticket.

I'm closing this ticket as fixed.