Avoid loading logically deleted records from shapefiles

[strk]

Some shapefiles contain records that are "logically deleted" according to some convention (or standard?) used by ESRI tools and at some point some version of QGIS too.

The convention states that a logically-deleted DBF record starts with an asterisk on the first byte. Such records should not be loaded, unless explicitly asked for by user.

[strk]

As QGIS struggled with this already, you can find more references there: ​http://hub.qgis.org/issues/11007

It would probably be best to let users decide what to do about these records, and maybe also mark them somehow on load (a "deleted" attribute?)

[strk]

The DBF specification: ​http://www.dbase.com/KnowledgeBase/int/db7_file_fmt.htm

[pramsey]

You have an example of one of these files and we don't do it? I vaguely recall investigation this ages ago and finding that shapelib actually already supported this case. Could be wrong though!

[robe]

I thought so too. I vaguely remember coming across that code when working on shp2pgsql.

[strk]

shapefile with 3 records, 2 of which logically deleted

[strk]

Attached test file, courtesy of Andrea who prepared it with qgis-2.12 (bogus)

It contains 3 records, 2 of which are "logically deleted". A recent QGIS would show just 1 shape in the map, while still reporting 3 when right-click→Show_Feature_Count is done on the lagend entry (still bogus).

Our loader loads all 3 of them, with no indication of which one would have been deleted.

[pramsey]

Committed a fix to trunk at r15271

[robe]

Can we mark this closed?

[strk]

Don't we want to keep the possibility to "resurrect" those records with a command-line switch ?

[robe]

why? I'd rather not have more switches and can't think of why anyone would want to resurrect deleted records.

[strk]

You might have deleted them on accident, for example

There could be a:

-L <policy> Logically-deleted records handling policy (skip*,keep,abort)

Where "keep" would create a system boolean column "deleted" on the output table.

This would be similar to the existing:

-N <policy> NULL geometries handling policy (insert*,skip,abort).

[robe]

I still say not worth the hassle of another switch. I could accidentally delete all my email too. It's not the job of a reader to try to help you sort this out. Use whatever tool you used to delete to undelete. I think the need is so rare, it causes more confusion than benefit to the 99.9% of people who don't need it.

[strk]

Ok then, let's close this. Won't insist anymore