can't make large/64-bit netCDF file

[rviger]

Howdy folks,

Apologies if this should be an enhancement request or if I'm posting to the wrong place altogether.

I'm trying to get my ArcInfo? grids (each is > 50,000 x 20,000 cells) into NetCDF, but am not having luck. I'm thinking the 32-bit encoding is the problem because I have succeeded in converting smaller grids to NetCDF with the same version of FWTools/GDAL (GDAL 1.7.0dev, FWTools 2.4.4, released 2009/09/23). I'm also confident that the grid I'm using to test is healthy because I have successfully used gdal_translate to turn the grid into HFA (which is about 7.5GB).

Is there an option on gdal_translate to force 64-bit encoding of the NetCDF file? I looked at -co "NAME=VALUE" but couldn't find doc on the name/value pairs; also thought it wasn't really the right track. Or is there another tool I should be looking at?

Thanks very much for any information.

Regards,

Roland Viger

[warmerdam]

Chaitanya,

As time permits could you look into whether the netcdf3 format supports files larger than 4GB, and if so, what would be needed to enable reading and writing large netcdf3 files with the netcdf driver?

I was vaguely under the impression that classic netcdf (aka netcdf 3) format does not support large files, and that was a feature added with use of hdf5 for "netcdf4".

[rviger]

Hi again Frank/Chaitanya?,

Didn't know if this page would help:  http://www.unidata.ucar.edu/software/netcdf/faq-lfs.html. You all probably know this stuff inside out.

My understanding of that page is that the unidata folks think it is possible to support large files w/netCDF version 3.6 (64-bit), even on a 32-bit platform, using things like ncgen -v or ncgen -x. I just didn't understand how these tools relate to the ever-handy FWTools environment, or how to get from an ArcInfo? grid or an HFA/IGE file set to netCDF with them.

Hadn't thought about tools relating to HDF so far.

Regards,

Roland Viger

[rsignell]

Replying to rviger:

Should just be able to change one line in frmts/netcdf/netcdfdataset.cpp

status = nc_create( pszFilename, NC_CLOBBER,  &fpImage );

to

status = nc_create( pszFilename, NC_64BIT_OFFSET,  &fpImage );

[rsignell]

Guys,

Uh oh. I see this on the NetCDF 3.6 page:

Q:Have all netCDF 3.6 size limits been eliminated? A:No, there are still some limits on sizes of netCDF objects, even with the new 64-bit offset format. Each fixed-size variable and the data for one record's worth of a record variable are limited in size to a little less that 4 GiB, which is twice the size limit in versions earlier than netCDF 3.6.

Unfortunately I think this is just what Roland wants to do: create a single variable larger than 4GB. So maybe for this case we have to link with NetCDF4 instead. That would be fine for Roland, I think, because he just wants that big file to serve via OpenDAP using THREDDS Data Server, which can handle NetCDF4 files just fine.

So I think we really want to recompile with

status = nc_create( pszFilename, NC_NETCDF4,  &fpImage );

using the netcdf4 libraries.

[rviger]

Replying to rsignell:

Hi Rich and others,

So this looks like I need to go back and recompile my GDAL from source, which I can do.

If I want this update to be in place from w/in the FWTools environment, does anyone know which file(s) I need to replace in the FWTools binary distro with what from the new compile?

I also wanted to make sure I understand what's happening here. As a result of this change will all netCDF files I make be 64 bit/netCDF4? I'm guessing that this will cause a little bit of bloat in the size of files that are not above the 2 or 4 (or whatever it is) GB limit for classic netCDF?

Thanks!

[rviger]

Just another update:

Scott McFarlane? and I just got through compiling GDAL on Linux. We added a flag and an environment variable to the configure with:

CPPFLAGS="-I/usr/include/netcdf-3" ./configure --enable-netcdf-4

The CPPFLAG reflects our (Mandrake) Linux RPM install of the latest netCDF/HDF binaries. We found the NC_FORMAT_NETCDF4 variable defined in the associated netcdf include file, but didn't find NC_NETCDF4. Based on this (and the fact that the NC_NETCDF4 substitution didn't compile), we used:

status = nc_create( pszFilename, NC_FORMAT_NETCDF4, &fpImage );

So far, we've made a 60 GB netCDF file.

I still have to look a little more at the health of that file. I did note that the od -c shows "0000000 C D F 001 " as the start of the first line. I don't know whether the "001" is a problem (I thought "002" was supposed to indicate that the file was 64-bit?). Ultimately, I don't care if the file is actually 64-bit, just as long as I can squash my data into the netCDF file.

[kyle]

I successfully created a 4.7 GB netcdf file from a 50000x50000 single band tif. I am using netcdf 3.6.3 and gdal trunk (r20109) on ubuntu

[etourigny]

Can you try latest svn trunk and see if either using one of "-co FORMAT=NC2|NC4|NC4C" works for you? Support for NC2(64-bit)/NC4/NC4 Classic has been added in r23272.

[etourigny]

Joaquim, can you please see if using the latest dev version you can create NC2 and NC4 files? Please test them afterwards with 'ncdump -k' to make sure file format is right.

[jluis]

Replying to etourigny:

Joaquim, can you please see if using the latest dev version you can create NC2 and NC4 files? Please test them afterwards with 'ncdump -k' to make sure file format is right.

Etienne,

I tested only with the 32 bits versions (netCDF4.1.3 & HDF5 1.8.8) as I have not yet built for 64 bits, and things seam to work well. I created a dumb netcdf file with GMT grdmath -Rg -I5 X = lixo.grd and converted with gdal_translate gdal_translate -of netcdf -co FORMAT=NC2 lixo.grd ofile2.nc gdal_translate -of netcdf -co FORMAT=NC4 lixo.grd ofile4.nc

chaecked with ncdump

ncdump -k ofile2.nc 64-bit offset

ncdump -k ofile4.nc netCDF-4

Joaquim

[etourigny]

Joaquim,

thanks for your input. I can mark this as closed, but I'd like to confirm that the driver can create nc2 and nc4 files on a 64-bit windows platform. If you manage to get it working please let me know here.

[jluis]

Etienne,

I'm afraid I'll have to set aside my attempt to build the 64-bits netcdf4.1.3 till I have access to a Win7 64 with SUA because with my current install this build has proven much more difficult than I first thought it would be. However, I rebuild netcdf4.1.2b with HDF51.8.8 and the above tests worked well on both 32 & 64-bits, so it seams that you have this case solved.

[rsignell]

Joachim,

You may have seen this on the NetCDF mailing list, but you can get a native Windows build of NetCDF 4.1.3 (64-bit and 32-bit) with OPeNDAP support (and Fortran support even) at  http://coast-enviro.er.usgs.gov/models/share/netcdf-4.1.3-win-dev.zip This was built by Christoph Gohlke at UCI.

-Rich

[etourigny]

Joaquim, I am going to close this ticket as you are able to create both formats in 32 and 64-bit windows. However, It would be great to have support for latest hdf. I believe I sent the above link to you some time ago and you had this problem, which I couldn't help you with, perhaps rsignell can help you?

I have now a HDF5 that seams coherent (though several test programs failed to build) and managed to compile netcdf4.1.3 but it depends on XDR, do you have any idea why?

[jluis]

Guys,

I did try the package that Rich pointed out but had lots of problems while trying to build it with my dependencies. I need to build with my dependencies because I want to control to names of the dlls. Also, I remember to have crossed with the XDR problem which I did not solve. I don't remember more of the details but I do remember that I put a reasonable effort in the attempt (which proves nothing).

[etourigny]

Luis, thanks for your help! Closing this ticket as resolved.