Opened 4 weeks ago

Closed 4 weeks ago

Last modified 4 weeks ago

#6870 closed defect (fixed)

Segmentation fault when trying to open a NetCDF subdataset with a 3rd (time) dimension

Reported by: atedstone Owned by: warmerdam
Priority: normal Milestone: 2.1.4
Component: default Version: unspecified
Severity: normal Keywords:
Cc:

Description

I am trying to create a NetCDF file which will be compliant with the NetCDF Conventions CF-1.4, including defining the projection of the geo-referenced subdatasets.

I have found that GDAL crashes with a segmentation fault when trying to access a subdataset in this file. I posted a question on GIS Exchange https://gis.stackexchange.com/questions/237951/gdal-crashes-with-segmentation-fault-when-trying-to-open-a-netcdf-subdataset-wit and received a reply suggesting that this is a problem with GDAL and that I should file a bug report.

The dimensions of the main subdataset are (TIME, Y, X) where Y and X are the polar stereographic grid.

The file that I generate contains three variables: runoff (TIME, Y, X), lat (Y, X) and lon (Y, X). With gdalinfo I can access the lat and lon grids, but not runoff:

$ gdalinfo NETCDF:"myfile.nc":runoff
Segmentation fault (core dumped)

In other words it looks like the TIME dimension is causing problems.

The file was created using xarray (http://xarray.pydata.org), with output set to NETCDF4. The ncdump of the file is as follows, and I've attached the file to the ticket.

N.b. this is a test-only file: runoff has dimensions TIME=2 and both matrices are filled with zeros. I can provide the creation script if useful.

netcdf myfile {
dimensions:
	X = 756 ;
	Y = 786 ;
	TIME = 2 ;
variables:
	int64 X(X) ;
		X:units = "meters" ;
		X:standard_name = "projection_x_coordinate" ;
		X:point_spacing = "even" ;
		X:axis = "X" ;
	int64 Y(Y) ;
		Y:units = "meters" ;
		Y:standard_name = "projection_y_coordinate" ;
		Y:point_spacing = "even" ;
		Y:axis = "Y" ;
	int64 TIME(TIME) ;
		TIME:standard_name = "time" ;
		TIME:axis = "TIME" ;
		TIME:units = "days since 1958-01-01 00:00:00" ;
		TIME:calendar = "proleptic_gregorian" ;
	byte polar_stereographic ;
		polar_stereographic:grid_mapping_name = "polar_stereographic" ;
		polar_stereographic:scale_factor_at_central_origin = 0. ;
		polar_stereographic:standard_parallel = 70. ;
		polar_stereographic:straight_vertical_longitude_from_pole = -45. ;
		polar_stereographic:latitude_of_projection_origin = 90. ;
		polar_stereographic:false_easting = 0. ;
		polar_stereographic:false_northing = 0. ;
	double lon(Y, X) ;
		lon:_FillValue = -9999. ;
		lon:grid_mapping = "polar_stereographic" ;
		lon:units = "degrees" ;
		lon:standard_name = "longitude" ;
	byte runoff(TIME, Y, X) ;
		runoff:_FillValue = -15b ;
		runoff:long_name = "runoff" ;
		runoff:units = "mmWE" ;
		runoff:standard_name = "runoff" ;
		runoff:grid_mapping = "polar_stereographic" ;
	double lat(Y, X) ;
		lat:_FillValue = -9999. ;
		lat:grid_mapping = "polar_stereographic" ;
		lat:units = "degrees" ;
		lat:standard_name = "latitude" ;

// global attributes:
		:Conventions = "CF-1.4" ;
		:history = "redacted" ;
		:proj4 = "+init=epsg:3413" ;
		:nx = 756. ;
		:ny = 786. ;
		:xmin = -1790000. ;
		:ymax = -65000. ;
		:spacing = 5000. ;
}

The file was created using the same conda environment that also provides GDAL, although I've tried to open the file on a different server with a different GDAL version and I also get the same problem.

Conda environment:

Running a conda environment:
basemap                   1.0.8.dev0          np112py35_3    conda-forge
basemap-data-hires        1.0.8.dev0                    2    conda-forge
blas                      1.1                    openblas    conda-forge
bleach                    1.5.0                    py35_0    conda-forge
bokeh                     0.12.4                   py35_0  
bottleneck                1.2.0               np112py35_0    conda-forge
ca-certificates           2017.1.23                     0    conda-forge
cairo                     1.14.6                        4    conda-forge
cartopy                   0.15.1              np112py35_1    conda-forge
certifi                   2017.1.23                py35_0    conda-forge
cloudpickle               0.2.2                    py35_2    conda-forge
curl                      7.52.1                        0    conda-forge
cycler                    0.10.0                   py35_0    conda-forge
dask                      0.13.0                   py35_0    conda-forge
decorator                 4.0.11                   py35_0    conda-forge
descartes                 1.1.0                    py35_0    conda-forge
dill                      0.2.6                    py35_0    conda-forge
entrypoints               0.2.2                    py35_1    conda-forge
expat                     2.1.0                         2    conda-forge
fastcache                 1.0.2                    py35_1  
fontconfig                2.12.1                        4    conda-forge
freetype                  2.7                           1    conda-forge
freexl                    1.0.2                         1    conda-forge
gdal                      2.1.3               np112py35_1    conda-forge
geos                      3.5.1                         1    conda-forge
gettext                   0.19.7                        1    conda-forge
giflib                    5.1.4                         0    conda-forge
glib                      2.51.4                        0    conda-forge
gmp                       6.1.2                         0    conda-forge
graphite2                 1.3.9                         0    conda-forge
h5netcdf                  0.3.1                    py35_0    conda-forge
h5py                      2.6.0               np112py35_7    conda-forge
harfbuzz                  1.4.3                         0    conda-forge
hdf4                      4.2.12                        0    conda-forge
hdf5                      1.8.17                        9    conda-forge
html5lib                  0.999                    py35_0    conda-forge
hyperspy                  1.2                      py35_0    conda-forge
icu                       58.1                          1    conda-forge
imageio                   2.1.2                    py35_0    conda-forge
ipykernel                 4.6.0                    py35_0    conda-forge
ipyparallel               6.0.1                    py35_0    conda-forge
ipython                   5.3.0                    py35_0    conda-forge
ipython_genutils          0.2.0                    py35_0    conda-forge
ipywidgets                6.0.0                    py35_0    conda-forge
jinja2                    2.9.5                    py35_0    conda-forge
jpeg                      9b                            0    conda-forge
json-c                    0.12                          0    conda-forge
jsonschema                2.5.1                    py35_0    conda-forge
jupyter                   1.0.0                    py35_0    conda-forge
jupyter_client            5.0.1                    py35_0    conda-forge
jupyter_console           5.1.0                    py35_0    conda-forge
jupyter_core              4.3.0                    py35_0    conda-forge
kealib                    1.4.6                         3    conda-forge
libdap4                   3.18.3                        2    conda-forge
libffi                    3.2.1                         3    conda-forge
libgfortran               3.0.0                         1  
libiconv                  1.14                          4    conda-forge
libnetcdf                 4.4.1.1                       2    conda-forge
libpng                    1.6.28                        0    conda-forge
libpq                     9.5.4                         3    conda-forge
libsodium                 1.0.10                        0    conda-forge
libspatialite             4.3.0a                       14    conda-forge
libtiff                   4.0.6                         7    conda-forge
libxml2                   2.9.4                         4    conda-forge
libxslt                   1.1.29                        3    conda-forge
locket                    0.2.0                    py35_1    conda-forge
lxml                      3.7.3                    py35_0    conda-forge
markupsafe                0.23                     py35_1    conda-forge
matplotlib                2.0.0               np112py35_2    conda-forge
mistune                   0.7.4                    py35_0    conda-forge
mpmath                    0.19                     py35_2    conda-forge
natsort                   5.0.2                    py35_0    conda-forge
nbconvert                 5.1.1                    py35_1    conda-forge
nbformat                  4.2.0                    py35_0    conda-forge
ncurses                   5.9                          10    conda-forge
netcdf4                   1.2.7               np112py35_0    conda-forge
networkx                  1.11                     py35_0    conda-forge
notebook                  5.0.0                    py35_0    conda-forge
numpy                     1.12.1          py35_blas_openblas_200  [blas_openblas]  conda-forge
olefile                   0.44                     py35_0    conda-forge
openblas                  0.2.19                        1    conda-forge
openjpeg                  2.1.2                         2    conda-forge
openssl                   1.0.2h                        3    conda-forge
owslib                    0.14.0                   py35_0    conda-forge
pandas                    0.19.2              np112py35_1    conda-forge
pandoc                    1.19.2                        0    conda-forge
pandocfilters             1.4.1                    py35_0    conda-forge
pango                     1.40.4                        0    conda-forge
partd                     0.3.7                    py35_0    conda-forge
patsy                     0.4.1                    py35_0    conda-forge
pcre                      8.39                          0    conda-forge
pexpect                   4.2.1                    py35_0    conda-forge
pickleshare               0.7.3                    py35_0    conda-forge
pillow                    4.0.0                    py35_2    conda-forge
pip                       9.0.1                    py35_0    conda-forge
pixman                    0.34.0                        0    conda-forge
proj4                     4.9.3                         2    conda-forge
prompt_toolkit            1.0.13                   py35_0    conda-forge
ptyprocess                0.5.1                    py35_0    conda-forge
pyepsg                    0.3.1                    py35_0    conda-forge
pyface                    5.1.0                    py35_1    conda-forge
pygments                  2.2.0                    py35_0    conda-forge
pyparsing                 2.2.0                    py35_0    conda-forge
pyproj                    1.9.5.1                  py35_0    conda-forge
pyqt                      4.11.4                   py35_2    conda-forge
pyshp                     1.2.10                   py35_0    conda-forge
python                    3.5.3                         1    conda-forge
python-dateutil           2.6.0                    py35_0    conda-forge
pytz                      2016.10                  py35_0    conda-forge
pywavelets                0.5.2               np112py35_0    conda-forge
pyyaml                    3.12                     py35_0    conda-forge
pyzmq                     16.0.2                   py35_1    conda-forge
qt                        4.8.7                         7    conda-forge
qtconsole                 4.3.0                    py35_0    conda-forge
readline                  6.2                           0    conda-forge
requests                  2.13.0                   py35_0    conda-forge
scikit-image              0.13.0              np112py35_0    conda-forge
scikit-learn              0.18.1          np112py35_blas_openblas_200  [blas_openblas]  conda-forge
scipy                     0.19.0          np112py35_blas_openblas_200  [blas_openblas]  conda-forge
seaborn                   0.7.1                    py35_0    conda-forge
setuptools                33.1.1                   py35_0    conda-forge
shapely                   1.5.17              np112py35_2    conda-forge
simplegeneric             0.8.1                    py35_0    conda-forge
sip                       4.18                     py35_1    conda-forge
six                       1.10.0                   py35_1    conda-forge
spectral                  0.18                      <pip>
sqlite                    3.13.0                        1    conda-forge
statsmodels               0.8.0               np112py35_0    conda-forge
sympy                     1.0                      py35_0    conda-forge
terminado                 0.6                      py35_0    conda-forge
testpath                  0.3                      py35_0    conda-forge
tk                        8.5.19                        1    conda-forge
toolz                     0.8.2                    py35_0    conda-forge
tornado                   4.4.2                    py35_0    conda-forge
tqdm                      4.11.2                   py35_0    conda-forge
traitlets                 4.3.2                    py35_0    conda-forge
traits                    4.6.0                    py35_1    conda-forge
traitsui                  5.1.0                    py35_0    conda-forge
util-linux                2.21                          0  
wcwidth                   0.1.7                    py35_0    conda-forge
webencodings              0.5                      py35_0    conda-forge
wheel                     0.29.0                   py35_0    conda-forge
widgetsnbextension        2.0.0                    py35_0    conda-forge
xarray                    0.9.1                    py35_0    conda-forge
xerces-c                  3.1.4                         3    conda-forge
xlrd                      1.0.0                    py35_1    conda-forge
xlwt                      1.2.0                    py35_0  
xz                        5.2.2                         0    conda-forge
yaml                      0.1.6                         0    conda-forge
zeromq                    4.2.1                         1    conda-forge
zlib                      1.2.11                        0    conda-forge

Change History (6)

comment:1 Changed 4 weeks ago by atedstone

I didn't appreciate that attached files had a < 1 mb limit. The file is 11 mb. I hope making it available via Dropbox is acceptable: https://dl.dropboxusercontent.com/u/93558837/myfile.nc.

comment:2 Changed 4 weeks ago by Kurt Schwehr

A quick stack (not at head, but I don't think it matters here)...

#0  0x00007ffff70f94da in strlen () from libc.so.6
#1  0x000000000051cc3b in NCDFSafeStrcat (ppszDest=0x7fffffffb530, pszSrc=0x13bfce9 <.L.str.51> "}", nDestSize=0x7fffffffb538) at frmts/netcdf/netcdfdataset.cpp:8101
#2  0x000000000050d0d1 in NCDFGet1DVar (nCdfId=65536, nVarId=2, pszValue=0x7fffffffd048) at frmts/netcdf/netcdfdataset.cpp:8758
#3  0x00000000005171b6 in netCDFDataset::Open (poOpenInfo=0x7fffffffd4f0) at frmts/netcdf/netcdfdataset.cpp:6542
#4  0x00000000005938eb in GDALOpenEx (pszFilename=0x7ffff6c68560 "NETCDF:/usr/local/google/home/schwehr/myfile.nc:runoff", nOpenFlags=66, papszAllowedDrivers=0x0, papszOpenOptions=0x0, 
    papszSiblingFiles=0x0) at gcore/gdaldataset.cpp:2781

A little debugging with cerr:

gdalinfo NETCDF:"myfile.nc":runoff
Warning 1: NetCDF driver detected file type=5, but libnetcdf detected type=3
nVarLen: 2
nVarType: 10
pszVarValue: Segmentation fault (core dumped)

This is going through the default case of switch (nVarType)

    default:
        CPLDebug("GDAL_netCDF", "NCDFGetVar1D unsupported type %d", nVarType);
        CPLFree(pszVarValue);
        pszVarValue = NULL;
        break;
    }

(gdb) n
8590	    switch (nVarType)
(gdb) n
8752	        CPLFree(pszVarValue);
(gdb) print nVarType
$5 = 10
#define NC_INT64        10

comment:3 Changed 4 weeks ago by Even Rouault

Resolution: fixed
Status: newclosed

In 38131:

netCDF: fix crash on int64/uint64 dimensions and variables, and add support for them (fixes #6870)

comment:4 Changed 4 weeks ago by Even Rouault

In 38132:

netCDF: fix crash on int64/uint64 dimensions and variables, and add support for them (fixes #6870)

comment:5 Changed 4 weeks ago by Even Rouault

In 38133:

netCDF: fix crash on int64/uint64 dimensions and variables, and add support for them (fixes #6870)

comment:6 Changed 4 weeks ago by Even Rouault

Milestone: 2.1.4
Note: See TracTickets for help on using tickets.