Opened 13 years ago

Last modified 13 years ago

#3099 new bug

Projection Failure

Reported by: badams Owned by: nobody
Priority: major: does not work as expected Milestone: Version 1.7.0
Component: Projection Support Version:
Keywords: Projection Failure Cc:
Must Fix for Release: No Platform: All
Platform Version: Awaiting user input: no

Description

QGIS does not operate with Australian Datums AGD66, AGD84 and GDA94. Although it will allow the selection of these datums, they are not applied to the data upon importing instead defaulting to WGS84. In the case of GDA94 V's AGD84 this results in an approx 200m offset error.

Similarly when exporting the data to either Shapefiles or Mapinfo TAB files, the projection is not sticking maintaining the default WGS84.

Roland Hill and Ken Norris(QGIS users) figured out the projection problem lies within the SRS.DB file and the Australia distortion grids (AGD66.gsb and AGD84.gsb), possibly indicating the problem lies with Proj4. I have attached Roland's edited files, he was able to make these work on the Linux platform but they didn't work for me on Windows 7. Not sure if he was able to get the projections to stick, I never did.

Attached CSV file contains data points in AGD66, AGD84, GDA94 and WSG84. This is the same grid in different datums. Note the AGD66 datum uses the "AGD66(5m)-Australia-GDA94" Local Datum Transformation. There are at least 10 other AGD66 - Local Datum Transformations however only AGD66(5m)-Australia-GDA94 covers all of Australia (or so I believe).

Attachments (4)

Brett.csv (29.7 KB ) - added by badams 13 years ago.
Data with different projections
project.zip (112.1 KB ) - added by lutra 13 years ago.
Screenshot.png (61.3 KB ) - added by lutra 13 years ago.
Screenshot-2.png (170.6 KB ) - added by lutra 13 years ago.

Download all attachments as: .zip

Change History (26)

by badams, 13 years ago

Attachment: Brett.csv added

Data with different projections

comment:1 by lutra, 13 years ago

Did you created the necessary custom CRSs in order to make the ntv2 grid transformation?

the source SRS must include both the "+nadgrids" paramater as well the "+wktext"

the tagert SRS must include (if is a projected CRS) "+towgs84=0,0,0"

Did you copied the necessary .gsb file in the right place in your windows installation?

otherwise the grid transformation will not be correctly done, and will "fail" silently.

I tested the system on windows Seven with data and grids for Portugal, and it works. You also must be sure that the geographic area of your vectors in actually covered by the grid you are using.

comment:2 by badams, 13 years ago

Sorry, Remaining files are too big to attach. They can be downloaded at;

http://dl.dropbox.com/u/9546967/srs.db http://dl.dropbox.com/u/9546967/agd66.gsb http://dl.dropbox.com/u/9546967/agd84.gsb

Additional information "Geocentric Datum of Australia Technical Manual" http://www.icsm.gov.au/gda/gdatm/gdav2.3.pdf.

comment:3 by lutra, 13 years ago

Can you attach data so I can make a test?

in reply to:  2 comment:4 by lutra, 13 years ago

Additional information "Geocentric Datum of Australia Technical Manual" http://www.icsm.gov.au/gda/gdatm/gdav2.3.pdf.

too big, don't have time to study such document, just tell ma what are the source/target srs, the appropriate grid and attach some data.

comment:5 by badams, 13 years ago

Additional Projection information

Universal Transverse Mercator AGD66 / AMG zone51 AGD84 / AMG zone51 GDA94 / MGA zone51 WGS84 / UTM zone51S Sorry, I'll get the information in eventually. Brett

comment:6 by lutra, 13 years ago

Component: Build/InstallProjection Support
Priority: critical: causes crash or data corruptionmajor: does not work as expected

a ntv2 grid is appropriate for just one kind of transformation, from one specific datum to another specific one. I'll need to know them (along with vectors), if you want me to test here.

comment:7 by badams, 13 years ago

Sorry Lutra, I actually have no idea what your talking about. I'm way out of my depth here.

The test is simple. The CSV file contains the same grid, listed in 4 different datums, UTM: AGD66, AGD84, GDA94 and WGS84(see projection information above). The projections already exist in QGIS, you don't need to change anything. Forget about the srs.db and gsb files for the moment.

If it worked properly, all 4 datums would plot in the same place.

in reply to:  7 comment:8 by lutra, 13 years ago

The projections already exist in QGIS, you don't need to change anything. Forget about the srs.db and gsb files for the moment.

If it worked properly, all 4 datums would plot in the same place.

The "re-projection" already exist in QGIS, right, but without transformation parameters such towgs84 or nadgrids, not all datum transformations will give good results. This is why in such situations you must use towgs84 or nadgrids parameters. Proj4 coordinate system definitions come *without* such parameters, so you need to add them manually.

Can you please specify, in terms of EPSG codes, what are the coordinate systems of the point grids in the csv file (beside obviously wgs84 that has the 4326 code)? thanks

comment:9 by badams, 13 years ago

Lutra, I think these are the codes your chasing

EPSG codes as listed in QGIS are as follows;

AGD66 / AMG zone51: 20251 AGD84 / AMG zone51: 20351 GDA94 / MGA zone51: 28351 WGS84 / UTM zone51S: 32651

I suspect ESPG: 4326 is a lat/long projection.

Brett

comment:10 by lutra, 13 years ago

Ok, this is a start. Now I just miss to know what kind of transformation are supposed to do the grids (the gsb files) you linked. Each grid is good for one kind of datum transformation.

comment:11 by badams, 13 years ago

Sorry Lutra, I don't know how the transformation process works. Can't help you with that.

comment:12 by lutra, 13 years ago

Supposing that the grids you linked are the right ones for the the geographic area of your interest (amg zone 51, but I don't think so, especially after reading this http://www.mail-archive.com/qgis-user@lists.osgeo.org/msg07255.html in particular http://www.mail-archive.com/qgis-user@lists.osgeo.org/msg07363.html and following answers), the transformation works like this

ogr2ogr -f "ESRI Shapefile" -s_srs "+proj=utm +zone=51 +south +ellps=aust_SA +units=m +no_defs +nadgrids=agd66.gsb +wktext" -t_srs "+proj=utm +zone=51 +south +ellps=GRS80 +towgs84=0,0,0 +units=m +no_defs" adg66_to_gda94.shp agd66_51.shp


where:

*) ogr2ogr is the program that underlies QGIS (and many other software) to handle vectors and make things like coordinate transformations

*) s_srs is the source coordinate system, here you need to add "+nadgrids=agd66.gsb" and "+wktext" and where "agd66.gsb" is the appropriate grid file you need to copy in the right folder (it depends on the OS and, under Windows, depends also in what kind of installation/program are you using)


*) t_srs is the target coordinate system, if is a projected one and you want the grid transformation to work you need to add always "+towgs84=0,0,0 "


Then you indicate the target shape and the source one. The source shape was obtained with the csv file you provided and saving the import result as shapefile with the appropriate CRS.


Obviously in the above example the operation is made by hand, with the CLI, very handy if you need to transform bunches of shapefiles, but it can be done obviously also with QGIS.


But let see before the results (see attachment).

after transforming the points grid from adg66 to gda94, using the ntv2 grid, the resulting grid (in gda94 crs) is just 1.3 meters away from the original grid in gda94, obtained by the coordinates in the csv file.

Usually grid transformations give better results, this is another reasons why I think that the grid you provided is not the right one for the geographic area of your sample data.

Back to QGIS:

the boring problem in QGIS is that in none of the GUIs that allow you to save a copy of a vector is available the field to let you choose the source SRS (I have already filed a enhancement ticket) and/or let the user add +nadgrids and +towgs84 parameters. So what you need to do:

create a custom CRS, to be used as source SRS

proj=utm +zone=51 +south +ellps=aust_SA +units=m +no_defs +nadgrids=agd66.gsb +wktext

save a copy of your source vector ("save as...") and give to it this custom coordinate system.

save again this "intermediate" vector and give to it the target coordinate system, in this case


"+proj=utm +zone=51 +south +ellps=GRS80 +towgs84=0,0,0 +units=m +no_defs"


and don't forget to check if the Proj definition in the QGIS database included "+towgs84=0,0,0" , if not you'll need to create a custom CRS also for the target coordinate system.


It seems complex, but it is not. The method works, and the results are the same under Linux and Windows, by hand or with QGIS.

Now one question and one problem. Is the precision of the results the one you were looking for? If not you than probably need to find a more appropriate ntv2 grid.

The problem: while doing the transformation with QGIS under windows Seven (osgeo4w/trunk) I'm getting some warnings that I don't get under Linux. So I'll need to investigate more.

Open the attached project and tell if it makes sense to you.

by lutra, 13 years ago

Attachment: project.zip added

by lutra, 13 years ago

Attachment: Screenshot.png added

by lutra, 13 years ago

Attachment: Screenshot-2.png added

comment:13 by badams, 13 years ago

Lutra, I'll need some time to digest this but I'll have a go.

The proposed projections are correct. I've been playing with them for the last 15 years. The link below will take to the Geoscience Australia site where they are explained fully.

http://www.ga.gov.au/geodesy/datums/cosys.jsp

Thanks Brett

comment:14 by badams, 13 years ago

Hi Lutra, Sorry for the slow response. Needed to run some trials.

The 1.3m variation between your conversion and mine is acceptable. Ran some trials with a professional Mapinfo user and we able to get a similar variation out of Mapinfo depending on the transforms selected so I'm happy the parameters you have are correct.

As I don't have the understanding, or the time, to implement these changes myself, I'm going to put some $$$ up for a bug fix. I'll run it past Paolo Cavallini first but I need a "plug and play" solution. Ideally the results will become part of QGIS so that anyone working on Australian data will benefit.

Given you understand the problem, can I ask you to help me put the bug fix proposal together?

Thanks Brett

comment:15 by lutra, 13 years ago

Hi Brett, I'm resuming what would be needed for both the vector and raster dialogs. see here

https://trac.osgeo.org/qgis/ticket/2913#comment:3

comment:16 by lutra, 13 years ago

see also #3118

comment:17 by jef, 13 years ago

r14399 add the possibility for user defined proj.4 paths and updates the srs.db to include references to the australian GSBs. The GSB themselves are not included. The user needs to either add them to the standard PROJ.4 directory or add a user defined path.

comment:18 by badams, 13 years ago

I loaded r14423. The 4 vector projections now import in the correct location so thats a good start.

Projections are not sticking for a shape or mapinfo export. They seem to reload in the right place, just with the wrong projection attached.

comment:19 by badams, 13 years ago

How is this bug fix progressing? I haven't seen any updates for a long time.

comment:20 by lutra, 13 years ago

no idea, meanwhile I compiled trunk and the indirect methods* do still work.

* copy the ntv2 grids in the proper system folder, create the proper custom CRS then use them for datum transformation (for vectors a middle passage is needed due to the lack of the "source srs" filed in the "save as..." dialogs).

comment:21 by custard, 13 years ago

Platform: WindowsAll

as at 1.7 r15530

From a quick test with the dev version isn't (r15530) and a set of single point shape files in AMGz51, MGAz51 and WGSz51s.

  • the AMGz51 point won't display in MGAz51 or WGSz51s.

QGIS puts up a window saying "Could not draw ShapeFile because:" and not giving a reason. The debug information says: "failed with error: failed to load datum shift file"

  • the MGAz51 and WGSz51s points won't display in AMGz51 and fail with the same set of errors.

More debug available if anyone wants it.

(and this bug exists in my unbuntu install too).

comment:22 by custard, 13 years ago

If the following files: ttp://www.icsm.gov.au/icsm/gda/gdatm/national66.zip http://www.icsm.gov.au/icsm/gda/gdatm/national84.zip are downloaded, and unzipped then renamed to agd66.gsb and agd84.gsb respectively and copied in the the projections folder[1] it appears to work.

If they could be included in the distribution I'd consider this fixed.

(any issues about these files not being recognised by other GIS programmes probably ought to be in a different bug)

[1] /usr/share/proj or C:/osge4w/share/proj or C:/Program Files/Quantum Version/share/proj and equivalent on a mac.

Note: See TracTickets for help on using tickets.