Opened 15 years ago

Closed 14 years ago

#2901 closed defect (fixed)

MapServer WCS can't locate NetCDF/HDF/ArcSDE data correctly

Reported by: yaxingwei Owned by: warmerdam
Priority: highest Milestone: 5.6 release
Component: WCS Server Version: svn-trunk (development)
Severity: blocker Keywords:
Cc: sdlime

Description

Hi,

I'm trying to serve some multi-variable NetCDF files through MapServer WCS (included in MS4W version 2.2.7/2.2.8/2.2.9/2.3.0/2.3.1). In the mapfile, I specified the DATA parameter for a NetCDF layer as

DATA 'NETCDF:"D:/DATA/wcstest.nc":NEE'

NetCDF files are handled by GDAL driver, so the above location format (NETCDF:"FILE_LOCATION":VARIABLE_NAME) is required by GDAL. But seems MapServer thinks this is a relative file path and always tries to concatenate SHAPEPATH and this DATA parameter together as the location of the NetCDF data file. Here is the error message returned by MapServer when I request this WCS layer:

msWCSGetCoverageMetadata(): Unable to access file. `D:/DATA/NETCDF:"D:/DATA/wcstest.nc":NEE' does not exist in the file system, and is not recognised as a supported dataset name.

The above described situation exists for HDF files and ArcSDE data sources also:

DATA 'HDF4_EOS:EOS_SWATH:"D:/DATA/MOD11_L2.A2005010.0010.004.2005011134147.hdf":MOD_Swath_LST:LST'
DATA "SDE:HOSTNAME,port:1234,sde,USERNAME,PASSWORD,SDE.SDE.US_FOREST_BIOMASS_ERROR,RASTER"

Someone else also found this issue. Details can be found here: http://www.mail-archive.com/mapserver-users@lists.osgeo.org/msg01627.html

Thanks.

Change History (5)

comment:1 by sdlime, 15 years ago

Cc: warmerdam added

More of a GDAL error, cc'ing Frank.

Steve

comment:2 by warmerdam, 15 years ago

Cc: sdlime added; warmerdam removed
Owner: changed from sdlime to warmerdam

It appears the problem is that not all logic uses consistent filename processing. The mapraster.c code, in particular, will only do the various path processing if the final result ends up being something that exists in the filesystem (fopen succeeds) otherwise it just reverts to the original unaltered filename. I think we need to change some other places (such as the coverage description fetcher) to use msTryBuildPath3() instead of msBuildPath3() which will give us a similar effect.

comment:3 by warmerdam, 15 years ago

Status: newassigned

Wei,

I have adjusted the raster get extents, and wcs get coverage code to *not* alter the filename by path prefixing if the result will not point to a valid object in the filesystem. I believe this should resolve the problem, but I have not tested it. Could you test against subversion trunk?

The fix is in trunk (r8880). I will propagate to the stable branch if it works smoothly.

comment:4 by ilucena, 15 years ago

Frank,

I would like to make a suggestion about that fix. It should be working fine for a caller that does not use the returning value (NULL), example:

  msTryBuildPath3(szPath, map->mappath, map->shapepath, layer->data);

  msAcquireLock( TLOCK_GDAL );
  hDS = GDALOpen(szPath, GA_ReadOnly );
  
  if( hDS != NULL )

http://trac.osgeo.org/mapserver/browser/trunk/mapserver/maprasterquery.c#L786

But I think you should remove that line of the code. I commented that out just for testing:

    fp = fopen( szReturnPath, "r" );
    if( fp == NULL )
    {
        strcpy( szReturnPath, path2 );
//      return NULL;
    }
    else
        fclose( fp );

REF: http://trac.osgeo.org/mapserver/browser/trunk/mapserver/mapstring.c#L668

So that callers like this one could still use the returning value:

    hDS = GDALOpen(msTryBuildPath3(szPath,  layer->map->mappath, layer->map->shapepath, layer->data) , GA_ReadOnly );
    if( hDS == NULL ) {
      msReleaseLock( TLOCK_GDAL );
      msSetError( MS_IOERR, "%s", "msWCSGetCoverageMetadata()", CPLGetLastErrorMsg() );
      return MS_FAILURE;
    }

http://trac.osgeo.org/mapserver/browser/trunk/mapserver/mapwcs.c#L2045

Or it will always result in this error:

[Mon Aug 31 00:09:02 2009].30083 mapserv request processing time (msLoadMap not incl.): 0.000s
[Mon Aug 31 00:09:02 2009].30104 msFreeMap(): freeing map at 0x8186540.
[Mon Aug 31 00:30:14 2009].649606 msWCSGetCoverageMetadata(): Unable to access file.-Pointer 'pszFilename' is NULL in 'GDALOpen'.

Ivan

comment:5 by warmerdam, 14 years ago

Milestone: 5.6 release
Resolution: fixed
Status: assignedclosed
Version: unspecifiedsvn-trunk (development)

Ivan,

I apologize for not dealing with this for so long. I have applied the first proposed approach. The docs for msTryBuildPath3() say it will return NULL if it does not find a physical file, and I don't want to change that assumption. But clearly the return value should not be used directly in GDALOpen().

The fix is applied in trunk (r9407) and should appear in beta3 tomorrow.

Thanks for the detailed analysis of my botch!

Note: See TracTickets for help on using tickets.