Opened 13 years ago

Closed 11 years ago

Last modified 8 years ago

#2654 closed defect (fixed)

[PATCH] Fix flipped latitude with NetCDF CF-1 files

Reported by: ekato Owned by: warmerdam
Priority: high Milestone:
Component: GDAL_Raster Version: 1.6.2
Severity: major Keywords: netcdf
Cc: Markus Neteler, dnadeau, mdsumner, Kyle Shannon

Description

Maybe related to JoseM's comment in #2584, I've experienced getting upside-down image with NetCDF driver. So I've created a simple patch to flip latitude for CF-1.0 files.

In netCDFDataset::SetProjection?(), I assume the files with following attributes as CF-1.0 files.

  • have valid nVarDimXID and nVarDimYID
  • poDS->papszDimName[poDS->nDimYid] starts with "lat"
  • pdfYCoord[0] < pdfYCoord[ydim-1]

Attachments (6)

netcdf-y-flip.patch (2.3 KB) - added by ekato 13 years ago.
Revised one which also works with files created by GMT's grdraster
netcdf-y-flip-rev.patch (2.7 KB) - added by ekato 13 years ago.
Real revised patch which also works with GMT's grdraster
netcdf-y-flip-rev2.patch (3.3 KB) - added by tbouffon 12 years ago.
Added support of more GMT-generated files
file.gmt (1.2 KB) - added by tbouffon 12 years ago.
file.xyz (37.5 KB) - added by tbouffon 12 years ago.
netcdf-y-flip-rev3.patch (3.4 KB) - added by tbouffon 12 years ago.

Download all attachments as: .zip

Change History (30)

Changed 13 years ago by ekato

Attachment: netcdf-y-flip.patch added

Revised one which also works with files created by GMT's grdraster

comment:1 Changed 13 years ago by Markus Neteler

Cc: Markus Neteler added

comment:2 Changed 13 years ago by Even Rouault

Component: defaultGDAL_Raster
Keywords: netcdf added
Summary: Fix flipped latitude with NetCDF CF-1 files[PATCH] Fix flipped latitude with NetCDF CF-1 files

comment:3 Changed 13 years ago by dylan

Any word when/if this can be merged into GDAL proper? It would save considerable time when working with NetCDF, by removing 2 steps involved in manually flipping the grid with a combination of gdal_translate and gdalwarp.

Thanks for the patch!

comment:4 Changed 13 years ago by warmerdam

Cc: warmerdam added
Owner: changed from warmerdam to dnadeau

This seems pretty similar to #2584 - are they related? Reassigning to Denis, our netcdf guru.

Changed 13 years ago by ekato

Attachment: netcdf-y-flip-rev.patch added

Real revised patch which also works with GMT's grdraster

comment:5 Changed 12 years ago by tbouffon

Milestone: 1.6.3
Priority: normalhigh
Severity: normalmajor
Version: 1.5.31.6.2

Hi, The patch did not work for me. Files created with xyz2grd (GMT 4.2.1) had y as poDS->papszDimName[poDS->nDimYid]. So I used GMT Metada to detect Latitude, and added a warning.

I haven't tested it a lot yet.

Changed 12 years ago by tbouffon

Attachment: netcdf-y-flip-rev2.patch added

Added support of more GMT-generated files

comment:6 Changed 12 years ago by mdsumner

Cc: mdsumner added

FWIW we've been using netcdf-y-flip-rev.patch successfully. The third one does not work at all, but I haven't explored why.

comment:7 Changed 12 years ago by warmerdam

Cc: dnadeau added; warmerdam removed
Owner: changed from dnadeau to warmerdam

I'll take a crack at reviewing the patches and applying one. I would like a test file - mdsumner its going to try and find one.

comment:8 Changed 12 years ago by warmerdam

Resolution: fixed
Status: newclosed

Based on a review of the code, and mdsumner's feedback I'm reasonable confident with netcdf-y-flip-rev.patch from ekato. There seem to be doubts still about the rev2 patches efforts with additional GMT files so I'm skipping it - it might be appropriate to file that in a new ticket which includes a reference to a file or files demonstrating the problem the patch solves.

I would add I'm interested in very small netcdf files demonstrating the issues in question that can be added to the autosuite. mdsumner provide me a test file, but it is large, and it isn't known if it is suitable for redistribution.

Patch applied in trunk (r18151,r18153), and 1.6 branch (r18152).

comment:9 Changed 12 years ago by tbouffon

Hi, indeed there is a problem with my rev2 patch. I thought I had posted the new version. I and a few of my colleagues have been working with the version I'm posting without any problem for one month now. I'm joining a set of files as an example : file.xyz is an ascii file containing gridded data file.gmt is generated with :

xyz2grd file.xyz -R0/10/0/10 -I1 -Gfile.gmt -fg

the version with rev.patch still flipped the raster, as the metada where not matching the criteria. But rev2.patch segfaults when the lat long name is empty. So I worked it around by setting a new string. I'm leaving the french comments in the patch to avoid any typo.

Is my answer fast enough or should I open a new ticket ?

Changed 12 years ago by tbouffon

Attachment: file.gmt added

Changed 12 years ago by tbouffon

Attachment: file.xyz added

Changed 12 years ago by tbouffon

Attachment: netcdf-y-flip-rev3.patch added

comment:10 Changed 12 years ago by Even Rouault

Resolution: fixed
Status: closedreopened

Re-opening as the netcdf-y-flip-rev3.patch was added after the commits

comment:11 Changed 12 years ago by Kyle Shannon

Cc: Kyle Shannon added

comment:12 Changed 11 years ago by Kyle Shannon

Has anyone tested the third revision of this patch?

comment:13 Changed 11 years ago by Kyle Shannon

Is there a reference for the bottom-first convention? I didn't see it in cf-1.4 docs. Is it always true?

comment:14 Changed 11 years ago by etiennesky

This problem is still found in the stable version 1.7.2. Can I test the first and 3rd patch in 1.7.2 stable or trunk?

I'd like to help fix this problem.

comment:15 Changed 11 years ago by etiennesky

Sorry looks like it's been fixed, it works correctly in stable 1.7.2.

comment:16 Changed 11 years ago by Kyle Shannon

I can't say that this is fixed completely in 1.7.2, as I still run into datasets that open upside down occasionally. I am still working on this one though...

comment:17 in reply to:  16 ; Changed 11 years ago by JeffBullard

Replying to kyle:

I can't say that this is fixed completely in 1.7.2, as I still run into datasets that open upside down occasionally. I am still working on this one though...

This problem is still not fixed (for me) in 1.7.2, although the problem could be with NOAA's degrib utility. This problem is reproducible: Download a grib file from NOAA's NOMAD server. Convert to NetCDF format using NOAA's 'degrib' utility Convert the NetCDF to GeoTiff? with gdalwarp. The resulting GeoTiff? is upside down, every time.

I have been dealing with this problem for the past four years and my 'workaround solution' is to gdal_translate -a_ullr the NetCDF corner coordinates (flip them upside down) before converting to Geotiff.

comment:18 in reply to:  17 ; Changed 11 years ago by tbouffon

Replying to JeffBullard:

Replying to kyle: This problem is still not fixed (for me) in 1.7.2, although the problem could be with NOAA's degrib utility. This problem is reproducible: Download a grib file from NOAA's NOMAD server. Convert to NetCDF format using NOAA's 'degrib' utility Convert the NetCDF to GeoTiff? with gdalwarp. The resulting GeoTiff? is upside down, every time.

I have been dealing with this problem for the past four years and my 'workaround solution' is to gdal_translate -a_ullr the NetCDF corner coordinates (flip them upside down) before converting to Geotiff.

Hi, did you try one of the patches above ? As long as the degrib utility writes latitude as y-coord name in the metadata, they should work.

comment:19 in reply to:  18 ; Changed 11 years ago by JeffBullard

Replying to tbouffon:

Replying to JeffBullard:

Replying to kyle: This problem is still not fixed (for me) in 1.7.2, although the problem could be with NOAA's degrib utility. This problem is reproducible: Download a grib file from NOAA's NOMAD server. Convert to NetCDF format using NOAA's 'degrib' utility Convert the NetCDF to GeoTiff? with gdalwarp. The resulting GeoTiff? is upside down, every time.

I have been dealing with this problem for the past four years and my 'workaround solution' is to gdal_translate -a_ullr the NetCDF corner coordinates (flip them upside down) before converting to Geotiff.

Hi, did you try one of the patches above ? As long as the degrib utility writes latitude as y-coord name in the metadata, they should work.

After more testing, I think the problem is with the degrib utility from NOAA. I downloaded software from NCAR to convert my GRIB data to NetCDF then converted to GeoTiff? with GDAL 1.7.2. The image was NOT upside-down. NCAR utility with GDAL gives me correct output. NOAA utility with GDAL gives me upside-down.

comment:20 in reply to:  19 Changed 11 years ago by Kyle Shannon

Replying to JeffBullard:

Replying to tbouffon:

Replying to JeffBullard:

Replying to kyle: This problem is still not fixed (for me) in 1.7.2, although the problem could be with NOAA's degrib utility. This problem is reproducible: Download a grib file from NOAA's NOMAD server. Convert to NetCDF format using NOAA's 'degrib' utility Convert the NetCDF to GeoTiff? with gdalwarp. The resulting GeoTiff? is upside down, every time.

I have been dealing with this problem for the past four years and my 'workaround solution' is to gdal_translate -a_ullr the NetCDF corner coordinates (flip them upside down) before converting to Geotiff.

Hi, did you try one of the patches above ? As long as the degrib utility writes latitude as y-coord name in the metadata, they should work.

After more testing, I think the problem is with the degrib utility from NOAA. I downloaded software from NCAR to convert my GRIB data to NetCDF then converted to GeoTiff? with GDAL 1.7.2. The image was NOT upside-down. NCAR utility with GDAL gives me correct output. NOAA utility with GDAL gives me upside-down.

Jeff, Can you test your file with Ivan's changes in r20006? Let me know if that does anything to help with your problem.

comment:21 Changed 11 years ago by Kyle Shannon

Ivan's correction fixed the files I was having issues with.

comment:22 Changed 11 years ago by tbouffon

Ivan's correction also fixes the problems with the file.gmt attached above.

comment:23 Changed 11 years ago by Kyle Shannon

Resolution: fixed
Status: reopenedclosed

I am closing this for now, if anyone finds another instance, please reopen. Thanks Ivan

comment:24 Changed 8 years ago by Even Rouault

Milestone: 1.6.4

Milestone 1.6.4 deleted

Note: See TracTickets for help on using tickets.