Opened 12 years ago
Closed 6 years ago
#4778 closed defect (fixed)
Some ARCM type geometries come in empty when converting SHP -> WKT in 1.9.1
| Reported by: | dtarc | Owned by: | warmerdam |
|---|---|---|---|
| Priority: | normal | Milestone: | |
| Component: | OGR_SRS | Version: | 1.9.1 |
| Severity: | normal | Keywords: | shapefile M value |
| Cc: | pka |
Description
This is my first time participating on http://trac.osgeo.org so please be patient with me. We have been getting great use out of ogr2ogr command line tool.
We are using ogr2ogr to get WKT from shapefiles. This has been working great.
We recently had a shapefile uploaded that could not properly be processed. Unfortunately I can not include the shapefile for privacy reasons but I will attempt to provide enough information to diagnose the problem.
We are able to process the shapefile successfully on our dev server, which is running GDAL 1.8.1. However, on our prod server where we are running GDAL 1.9.1, ogr2ogr outputs empty data.
Here is the command line call being made to ogr2ogr:
/usr/local/bin/ogr2ogr -f CSV /vsistdout/ "/path/to/shapefile.shp" -lco GEOMETRY=AS_WKT -t_srs EPSG:4326
Here is the contents of the shapefile's .prj file (new lines and indents added for readability):
PROJCS[
"NAD_1983_UTM_Zone_12N",
GEOGCS[
"GCS_North_American_1983",
DATUM[
"D_North_American_1983",
SPHEROID["GRS_1980",6378137.0,298.257222101]
],
PRIMEM["Greenwich",0.0],
UNIT["Degree",0.0174532925199433]
],
PROJECTION["Transverse_Mercator"],
PARAMETER["False_Easting",500000.0],
PARAMETER["False_Northing",0.0],
PARAMETER["Central_Meridian",-111.0],
PARAMETER["Scale_Factor",0.9996],
PARAMETER["Latitude_Of_Origin",0.0],
UNIT["Meter",1.0]
]
The shapefile consists of a whole bunch of LINESTRINGs and MULTILINESTRINGs.
The data/metadata looks like this (some metadata removed):
0 (String, 3 characters ) WKT
1 (String, 8 characters ) OBJECTID
...
5 (String, 10 characters ) SHAPE_Leng
When I run the ogr2ogr command from the dev server with GDAL 1.8.1 I get output like this:
0 (String, 271 characters ) MULTILINESTRING ((-111.420747368885614 57.114691178173281 0,-111.419128322207925 57.11207642429202 0,-111.409814661906921 57.097710767770103 0,-111.40118599935515 57.084430689254518 0),(-111.422865111792362 57.117815452150026 0,-111.421061936375139 57.115166987602194 0))
1 (String, 3 characters ) 110
...
5 (String, 16 ) 3885.36509873000
When I run the ogr2ogr command from prod with GDAL 1.9.1 I get output like this:
0 (String, 57 characters ) MULTILINESTRING ((0 0 0,0 0 0,0 0 0,0 0 0),(0 0 0,0 0 0))
1 (String, 3 characters ) 110
...
5 (String, 16 characters ) 3885.36509873000
As you can see, SHAPE_Leng still gets properly calculated, but the WKT is empty. This happens for both LINESTRINGs and MULTILINESTRINGs...all values in the WKT are turned to 0.
I can provide any more details as needed.
Any feedback appreciated as to why this is happening or what possible easy fixes or command line additions/workarounds could be made to fix this issue. Thanks
Change History (15)
comment:1 by , 12 years ago
comment:2 by , 12 years ago
ogrinfo returns different results with 1.8.1 and 1.9.1.
ogrinfo output from 1.8.1:
INFO: Open of `TEST_SHAPEFILE.shp'
using driver `ESRI Shapefile' successful.
1: TEST_SHAPEFILE (Line String)
ogrinfo output from 1.9.1:
INFO: Open of `TEST_SHAPEFILE.shp'
using driver `ESRI Shapefile' successful.
1: TEST_SHAPEFILE (3D Line String)
I'm guessing the problem is that there's z/altitude data in the shapefile then? Can I get ogr2ogr to ignore it somehow?
comment:3 by , 12 years ago
comment:4 by , 12 years ago
1.8.1:
$ ogrinfo -al TEST_SHAPEFILE.shp
INFO: Open of `TEST_SHAPEFILE.shp'
using driver `ESRI Shapefile' successful.
Layer name: TEST_SHAPEFILE
Geometry: Line String
Feature Count: 199
Extent: (462020.353400, 6319711.121700) - (497746.525500, 6355939.085900)
Layer SRS WKT:
PROJCS["NAD_1983_UTM_Zone_12N",
GEOGCS["GCS_North_American_1983",
DATUM["North_American_Datum_1983",
SPHEROID["GRS_1980",6378137.0,298.257222101]],
PRIMEM["Greenwich",0.0],
UNIT["Degree",0.0174532925199433]],
PROJECTION["Transverse_Mercator"],
PARAMETER["False_Easting",500000.0],
PARAMETER["False_Northing",0.0],
PARAMETER["Central_Meridian",-111.0],
PARAMETER["Scale_Factor",0.9996],
PARAMETER["Latitude_Of_Origin",0.0],
UNIT["Meter",1.0]]
OBJECTID: Integer (9.0)
…more metadata column definitions...
SHAPE_Leng: Real (19.11)
OGRFeature(TEST_SHAPEFILE):0
OBJECTID (Integer) = 110
…metadata values...
SHAPE_Leng (Real) = 3885.36509873000
MULTILINESTRING ((474519.26410000026226 6330231.4915 0,474615.525899999774992 6329939.8237 0,475169.998800000175834 6328337.2691 0,475684.101700000464916 6326855.8794 0),(474393.168700000271201 6330580.0666 0,474500.540699999779463 6330284.5743 0))
OGRFeature(TEST_SHAPEFILE):1
OBJECTID (Integer) = 111
…metadata values...
SHAPE_Leng (Real) = 7121.01688903000
MULTILINESTRING ((480817.859299999661744 6329223.7378 0,480817.869900000281632 6329212.9459 0,480808.256699999794364 6328348.7112 0,477337.3936 6328335.3215 0,477335.501400000415742 6328821.8777 0,477328.583399999886751 6329830.8071 0,477327.476099999621511 6329902.9382 0,477328.99170000012964 6329938.4392 0,477328.4836 6330290.0975 0,477325.033800000324845 6330363.4286 0,477325.014999999664724 6330363.8291 0,477242.442200000397861 6330360.4279 0,476909.1711 6330345.4597 0),(476908.326999999582767 6330345.4977 0,476906.995000000111759 6330536.4251 0),(476907.396099999547005 6330536.4453 0,476907.9627 6330671.0791 0,476907.881 6330675.6516 0))
...more features
1.9.1:
$ ogrinfo -al TEST_SHAPEFILE.shp
INFO: Open of `TEST_SHAPEFILE.shp'
using driver `ESRI Shapefile' successful.
Layer name: TEST_SHAPEFILE
Geometry: 3D Line String
Feature Count: 199
Extent: (462020.353400, 6319711.121700) - (497746.525500, 6355939.085900)
Layer SRS WKT:
PROJCS["NAD_1983_UTM_Zone_12N",
GEOGCS["GCS_North_American_1983",
DATUM["North_American_Datum_1983",
SPHEROID["GRS_1980",6378137.0,298.257222101]],
PRIMEM["Greenwich",0.0],
UNIT["Degree",0.0174532925199433]],
PROJECTION["Transverse_Mercator"],
PARAMETER["False_Easting",500000.0],
PARAMETER["False_Northing",0.0],
PARAMETER["Central_Meridian",-111.0],
PARAMETER["Scale_Factor",0.9996],
PARAMETER["Latitude_Of_Origin",0.0],
UNIT["Meter",1.0]]
OBJECTID: Integer (9.0)
…more metadata column definitions...
SHAPE_Leng: Real (19.11)
OGRFeature(TEST_SHAPEFILE):0
OBJECTID (Integer) = 110
…metadata values...
SHAPE_Leng (Real) = 3885.36509873000
MULTILINESTRING ((0 0 0,0 0 0,0 0 0,0 0 0),(0 0 0,0 0 0))
OGRFeature(TEST_SHAPEFILE):1
OBJECTID (Integer) = 111
…metadata values...
SHAPE_Leng (Real) = 7121.01688903000
MULTILINESTRING ((0 0 0,0 0 0,0 0 0,0 0 0,0 0 0,0 0 0,0 0 0,0 0 0,0 0 0,0 0 0,0 0 0,0 0 0,0 0 0),(0 0 0,0 0 0),(0 0 0,0 0 0,0 0 0))
...more features
Since I notice it and it seems strange, I'll mention that there is one feature that does seem to come through properly with 1.9.1. It is one of the last features in the shapefile.
1.8.1:
OGRFeature(TEST_SHAPEFILE):192
OBJECTID (Integer) = 309
…metadata values...
SHAPE_Leng (Real) = 2316.41761688000
MULTILINESTRING ((485401.37540000025183 6345410.755 0,485344.158800000324845 6345525.5271 0,485274.742600000463426 6345669.7123 0,485196.5778 6345827.9636 0,485184.018899999558926 6345831.0564 0),(484734.383100000210106 6346804.2683 0,484685.5855 6346804.4499 0,484583.8964 6346806.5004 0),(484583.910000000149012 6346806.0619 0,484574.235500000417233 6346803.9708 0,484531.474399999715388 6346804.1315 0,484396.6036 6346804.6415 0,484126.941200000233948 6346809.1269 0,484006.860000000335276 6346812.5972 0,484005.056400000117719 6346812.9401 0),(484999.2105 6346234.0846 0,484934.7908 6346369.1507 0,484856.713200000114739 6346527.4085 0,484805.1075 6346634.577 0,484803.251299999654293 6346637.1404 0,484798.700199999846518 6346646.5068 0,484754.389399999752641 6346734.9499 0,484727.897199999541044 6346796.2772 0,484728.3854 6346797.0499 0,484733.211500000208616 6346799.0444 0),(485196.285199999809265 6345828.1513 0,485186.1185 6345855.6019 0,485125.744300000369549 6345977.2792 0,485054.9567 6346121.1373 0,484999.68680000025779 6346232.9912 0,484977.417999999597669 6346250.466 0))
1.9.1:
OGRFeature(TEST_SHAPEFILE):192
OBJECTID (Integer) = 309
…metadata values...
SHAPE_Leng (Real) = 2316.41761688000
MULTILINESTRING ((485401.37540000025183 6345410.755 0,485344.158800000324845 6345525.5271 0,485274.742600000463426 6345669.7123 0,485196.5778 6345827.9636 0,485184.018899999558926 6345831.056399999186 0),(484734.383100000210106 6346804.2683 0,484685.5855 6346804.449899999425 0,484583.896399999969 6346806.500399999321 0),(484583.910000000149012 6346806.061899999157 0,484574.235500000417233 6346803.970799999312 0,484531.474399999715388 6346804.1315 0,484396.6036 6346804.641499999911 0,484126.941200000233948 6346809.1269 0,484006.860000000335276 6346812.5972 0,484005.056400000117719 6346812.940099999309 0),(484999.2105 6346234.084599999711 0,484934.79079999961 6346369.150699999183 0,484856.713200000114739 6346527.4085 0,484805.107499999925 6346634.57699999958 0,484803.251299999654293 6346637.140399999917 0,484798.700199999846518 6346646.506799999624 0,484754.389399999752641 6346734.949899999425 0,484727.897199999541044 6346796.2772 0,484728.3854 6346797.0499 0,484733.211500000208616 6346799.044399999082 0),(485196.285199999809265 6345828.1513 0,485186.118499999866 6345855.6019 0,485125.744300000369549 6345977.2792 0,485054.956699999981 6346121.137299999595 0,484999.68680000025779 6346232.9912 0,484977.417999999597669 6346250.466 0))
Thanks for any advice.
comment:5 by , 12 years ago
Strange indeed. I'm not sure how we can debug that further in a remote way. Perhaps you could just send me privately (even.rouault at mines-paris.org) the .shp and .shx files (this will just contain the geometries, no need for the .dbf which contains the attributes) so I can check ? Otherwise I'm afraid the issue will be left unresolved.
comment:7 by , 12 years ago
| Cc: | added |
|---|---|
| Keywords: | shapefile M value added |
Frank, "nbo",
The support for measures added per #2374 causes a regression on the test case provided by Dave. His shapefile is of SHPT_ARCM type. With r23599, we now display the M value. In that instance, the M value is set to the binary sequence "FF FF FF FF FF FF EF FF FF", which is -1.79769313486232E+308 (-infinity I guess). OGRMakeWktCoordinate() didn't have enough space to display the M value (there's 75 bytes available for each component, which is OK for reasonable coordinates), so it ended up displaying (0,0,0).
I've committed something in r24820 that make OGRFormatDouble() return "too_big", so that the error is more obvious. But I'm wondering if we should really use the M measure when it is set to -infinity if that's value is known/documented as meaning "unset M value", which I don't know.
Dave, as a workaround you can do : "ogr2ogr shape_without_M.shp your.shp -nlt MULTILINESTRING". This will drop the M value that causes the issue.
comment:8 by , 12 years ago
| Cc: | added; removed |
|---|
Actually, I didn't read #2374 history correctly and should haved CC'ed Pirmin rather
comment:9 by , 12 years ago
Thanks for sorting this out. I will try adding that command line option. We are actually processing shapefiles that have POINT/MULTIPOINT and POLYGON/MULTIPOLYGON geometries as well. It just happens that this shapefile causing the problems only had LINESTRINGs and MULTILINESTRINGs.
The code is actually for a Drupal module that processes shapefiles that we're about to release on drupal.org. So I'm not sure that we can make assumptions about the geometry types.
Can you comment on whether "-nlt MULTILINESTRING" will work for geometries across the board? Or do I need to add other geometry types in order to remove the M values causing the issue?
comment:10 by , 12 years ago
For POINT/MULTIPOINT with M value, you should of course use -nlt POINT (or MULTIPOINT). For POLYGON/MULTIPOLYGON, -nlt POLYGON (or MULTIPOLYGON). There's no option that discard the 3rd component for all cases (although that it should be doable easily in the absolute, just not existing right now)
comment:11 by , 12 years ago
The shapefile could contain any type though and we won't know beforehand...looking at --long-usage it doesn't look like I can use -nlt to limit the types, only force it to one in particular.
comment:12 by , 12 years ago
Shapefiles cannot have a mix of points, (multi)linestrings or (multi)polygons. You can find the right value by using the OGR API (C, C++, Python, Java, Perl, C# available) to query the geometry type of your shapefile. See the OGR_L_GetGeomType() API ( http://www.gdal.org/ogr/ogr__api_8h.html#a0adea8ce1ca795ce0a6a76505f90f078 ).
comment:13 by , 9 years ago
When it comes to the value of M, it seems to be used in a way that ESRI specifies. See http://webhelp.esri.com/arcgisdesktop/9.3/index.cfm?TopicName=Geoprocessing_considerations_for_shapefile_output
Number—When tool requires a NULL, infinity, or NaN (Not a Number) to be output -1.7976931348623158e+308 (IEEE standard for the maximum negative value)
I do not know if this means that GDAL is not totally shapefile compliant because it can't interpret small number as NULL or not. Anyway, on page 2 of the shapefile specification reads:
Floating point numbers must be numeric values. Positive infinity, negative infinity, and Not-a-Number (NaN) values are not allowed in shapefiles. Nevertheless, shapefiles support the concept of "no data" values, but they are currently used only for measures. Any floating point number smaller than –10^38 is considered by a shapefile reader to represent a "no data" value.
Further, for example on page 10 which deals with PointM type:
Shapes of this type have an additional coordinate M. Note that "no data" value can be specified as a value for M (see Numeric Types on page 2).
This should give a definitive answer to what Even was wondering "But I'm wondering if we should really use the M measure when it is set to -infinity if that's value is known/documented as meaning "unset M value", which I don't know."
comment:14 by , 8 years ago
| Summary: | Some geometries come in empty when converting SHP -> WKT in 1.9.1 → Some ARCM type geometries come in empty when converting SHP -> WKT in 1.9.1 |
|---|
comment:15 by , 6 years ago
| Resolution: | → fixed |
|---|---|
| Status: | new → closed |
This issue is likely no longer current since GDAL supports M dimension
Weird. Does ogrinfo on the shapefile return correct geometries ?
(SHAPE_Leng has nothing to do with that, and it is not "calculated". It is an attribute from the DBF file, so it is not computed by OGR, just read...)