OGR GPX driver loses time field in shapefile

[hamish]

Hi,

thanks for looking at the two GPX issues I reported yesterday. Here's another one for you:

If using ogr2ogr to convert a GPX file containing a track into a shapefile the 'time' Field gets broken and you end up with no time data in the shapefile's DBF.

for example:

$ ogr2ogr  -f "ESRI Shapefile" test_trk_shp test_trk.gpx
Warning 6: Field time create as date field, though DateTime requested.

Warning 6: Field time create as date field, though DateTime requested.

Warning 6: Field time create as date field, though DateTime requested.

$ dbview -e test_trk_shp/track_points.dbf | head
Field Name      Type    Length  Decimal Pos
track fid         N        11       0
track seg         N        11       0
track seg         N        11       0
ele               N        24      15
time              D         8       0
magvar            N        24      15
geoidheigh        N        24      15
name              C        80       0
cmt               C        80       0
...

# DBF contains records 4000 data points, half with no timestamp (doh!)
$ dbview track_points.dbf | grep '^time' | uniq
time       : 00000000
time       : 

i.e. when the <time> GPX element is missing from the <trkpt> "00000000" is written. when a <time> value is in the <trkpt> record it is written empty.

good times look like:

<time>2009-04-28T06:31:53Z</time>

and for many needs recording time to the nearest second is needed. e.g. we are adding spatial data to recording equipment logs which record timestamps but are not directly connected to the GPS.

if DateTime? is not available could it write to (e.g.) varchar(255) instead?

thanks, Hamish

[rouault]

I'm a bit confused of which <time> element you are speaking about. Could you attach a small GPX file demonstrating the issue ?

[hamish]

example gpx file with partial missing timestamps

[hamish]

Replying to rouault:

I'm a bit confused of which <time> element you are speaking about. Could you attach a small GPX file demonstrating the issue ?

ok, attached. the resulting track_points.shp's .dbf is losing the <time> field in this:

<trkpt lat="-46.035737442" lon="170.171801448">
  <ele>9.750610</ele>
  <time>2009-04-27T02:38:41Z</time>
  <extensions>
    <gpxx:Depth>8.128929</gpxx:Depth>
    <gpxx:Temperature>12.739347</gpxx:Temperature>
  </extensions>
</trkpt>

thanks, Hamish

[rouault]

OK, yes, this was an issue in GDAL 1.5.2

• From GDAL 1.5.4, field of type DateTime? are written in shapefiles as fields of type Date (See http://trac.osgeo.org/gdal/ticket/2474). But of course, this keep only the day part and loses the hour:minute:second part of a datetime field

• From GDAL 1.6.0, you can use the OGR SQL CAST operator to convert any field to another type. For example :

  ogr2ogr out source.gpx -sql 'select ele, cast (time as character(32)) from track_points'
  

• In trunk (GDAL 1.7.0dev r16894), I've introduced a new option '-fieldTypeToString' for ogr2ogr that is a convenient way of casting all field of given type(s) to fields of type string in the destination layer. For example :

  ogr2ogr out source.gpx -fieldTypeToString DateTime
  

• In branches/1.6 (r16895), I've just backported the core fix done in OGRFeature::SetFrom?(), so that datetime fields can be written as strings

[rouault]

Added (in r16897) doc for the issue discussed in that ticket

[hamish]

Replying to rouault:

* In branches/1.6 (r16895), I've just backported the core fix done in OGRFeature::SetFrom?(), so that datetime fields can be written as strings

thanks. For 1.7dev it would seem to me that writing the time as a string should be the default for shapefiles (lacking DateTime? support), rather than to be intentionally 50% lossy due to the partial name overlap of the types. the loss of precision is severe. or another way to think about it, the nearest neighbor type-name is not always the best choice.

cheers, Hamish

[rouault]

It depends on the use cases. You may want that the column is of type Date. Anyway, now you have full control on how to output it in the shapefile. And I'd note that if you try to write a DateTime? field, the shapefile driver issues a warning : "Warning 6: Field time create as date field, though DateTime? requested."