Opened 6 years ago

Closed 6 years ago

#3645 closed defect (fixed)

Avoid loading logically deleted records from shapefiles

Reported by: strk Owned by: pramsey
Priority: medium Milestone: PostGIS 2.4.0
Component: postgis Version: master
Keywords: shapefile Cc: aperi2007, esseffe

Description

Some shapefiles contain records that are "logically deleted" according to some convention (or standard?) used by ESRI tools and at some point some version of QGIS too.

The convention states that a logically-deleted DBF record starts with an asterisk on the first byte. Such records should not be loaded, unless explicitly asked for by user.

Attachments (1)

logical_delete.zip (29.0 KB ) - added by strk 6 years ago.
shapefile with 3 records, 2 of which logically deleted

Download all attachments as: .zip

Change History (15)

comment:1 by strk, 6 years ago

As QGIS struggled with this already, you can find more references there: http://hub.qgis.org/issues/11007

It would probably be best to let users decide what to do about these records, and maybe also mark them somehow on load (a "deleted" attribute?)

comment:3 by pramsey, 6 years ago

You have an example of one of these files and we don't do it? I vaguely recall investigation this ages ago and finding that shapelib actually already supported this case. Could be wrong though!

comment:4 by robe, 6 years ago

I thought so too. I vaguely remember coming across that code when working on shp2pgsql.

by strk, 6 years ago

Attachment: logical_delete.zip added

shapefile with 3 records, 2 of which logically deleted

comment:5 by strk, 6 years ago

Attached test file, courtesy of Andrea who prepared it with qgis-2.12 (bogus)

It contains 3 records, 2 of which are "logically deleted". A recent QGIS would show just 1 shape in the map, while still reporting 3 when right-click→Show_Feature_Count is done on the lagend entry (still bogus).

Our loader loads all 3 of them, with no indication of which one would have been deleted.

comment:6 by pramsey, 6 years ago

Committed a fix to trunk at r15271

comment:7 by robe, 6 years ago

Milestone: PostGIS FuturePostGIS 2.4.0

comment:8 by robe, 6 years ago

Can we mark this closed?

comment:9 by strk, 6 years ago

Don't we want to keep the possibility to "resurrect" those records with a command-line switch ?

comment:10 by robe, 6 years ago

why? I'd rather not have more switches and can't think of why anyone would want to resurrect deleted records.

comment:11 by strk, 6 years ago

You might have deleted them on accident, for example

There could be a:

-L <policy> Logically-deleted records handling policy (skip*,keep,abort)

Where "keep" would create a system boolean column "deleted" on the output table.

This would be similar to the existing:

-N <policy> NULL geometries handling policy (insert*,skip,abort).

comment:12 by robe, 6 years ago

I still say not worth the hassle of another switch. I could accidentally delete all my email too. It's not the job of a reader to try to help you sort this out. Use whatever tool you used to delete to undelete. I think the need is so rare, it causes more confusion than benefit to the 99.9% of people who don't need it.

comment:13 by strk, 6 years ago

Ok then, let's close this. Won't insist anymore :)

comment:14 by robe, 6 years ago

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.