Opened 3 years ago

Closed 2 years ago

#3645 closed defect (fixed)

Avoid loading logically deleted records from shapefiles

Reported by: strk Owned by: pramsey
Priority: medium Milestone: PostGIS 2.4.0
Component: postgis Version: trunk
Keywords: shapefile Cc: aperi2007, esseffe

Description

Some shapefiles contain records that are "logically deleted" according to some convention (or standard?) used by ESRI tools and at some point some version of QGIS too.

The convention states that a logically-deleted DBF record starts with an asterisk on the first byte. Such records should not be loaded, unless explicitly asked for by user.

Attachments (1)

logical_delete.zip (29.0 KB) - added by strk 3 years ago.
shapefile with 3 records, 2 of which logically deleted

Download all attachments as: .zip

Change History (15)

comment:1 Changed 3 years ago by strk

As QGIS struggled with this already, you can find more references there: http://hub.qgis.org/issues/11007

It would probably be best to let users decide what to do about these records, and maybe also mark them somehow on load (a "deleted" attribute?)

comment:3 Changed 3 years ago by pramsey

You have an example of one of these files and we don't do it? I vaguely recall investigation this ages ago and finding that shapelib actually already supported this case. Could be wrong though!

comment:4 Changed 3 years ago by robe

I thought so too. I vaguely remember coming across that code when working on shp2pgsql.

Changed 3 years ago by strk

Attachment: logical_delete.zip added

shapefile with 3 records, 2 of which logically deleted

comment:5 Changed 3 years ago by strk

Attached test file, courtesy of Andrea who prepared it with qgis-2.12 (bogus)

It contains 3 records, 2 of which are "logically deleted". A recent QGIS would show just 1 shape in the map, while still reporting 3 when right-click->Show_Feature_Count is done on the lagend entry (still bogus).

Our loader loads all 3 of them, with no indication of which one would have been deleted.

comment:6 Changed 2 years ago by pramsey

Committed a fix to trunk at r15271

comment:7 Changed 2 years ago by robe

Milestone: PostGIS FuturePostGIS 2.4.0

comment:8 Changed 2 years ago by robe

Can we mark this closed?

comment:9 Changed 2 years ago by strk

Don't we want to keep the possibility to "resurrect" those records with a command-line switch ?

comment:10 Changed 2 years ago by robe

why? I'd rather not have more switches and can't think of why anyone would want to resurrect deleted records.

comment:11 Changed 2 years ago by strk

You might have deleted them on accident, for example

There could be a:

-L <policy> Logically-deleted records handling policy (skip*,keep,abort)

Where "keep" would create a system boolean column "deleted" on the output table.

This would be similar to the existing:

-N <policy> NULL geometries handling policy (insert*,skip,abort).

comment:12 Changed 2 years ago by robe

I still say not worth the hassle of another switch. I could accidentally delete all my email too. It's not the job of a reader to try to help you sort this out. Use whatever tool you used to delete to undelete. I think the need is so rare, it causes more confusion than benefit to the 99.9% of people who don't need it.

comment:13 Changed 2 years ago by strk

Ok then, let's close this. Won't insist anymore :)

comment:14 Changed 2 years ago by robe

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.