Opened 6 years ago

Closed 4 months ago

#5146 closed defect (wontfix)

[Shapelib patch] EOF-weirdness when deleting columns in .dbf

Reported by: berendsen Owned by: warmerdam
Priority: normal Milestone: closed_because_of_github_migration
Component: default Version: 1.9.2
Severity: normal Keywords: dbf, columns, delete
Cc:

Description

Symptom: When deleting a column in a shapefiles dbf table, it leads to strangeness at the end of the dbf-file. Strangeness is always visible when looking at the file in Excel, but only sometimes in Libreoffice / OOo Calc (seems to be only when having deleted a LOT of columns).

(noticed this when a colleague had problems loading such a file into an propietary ArcGIS 9.x extension)

Steps to reproduce:

  1. Create shapefile
  2. Create some random points, lines or polygons
  3. Create and fill some random columns in the table
  4. Delete a column
  5. Check .dbf in Excel

(The attached .zip contains two shapefiles. 'dbf-test' is before the deletion of table column, 'dbf-test-afterdelete' afterwards)

Happens: Always

Observed in version: GDAL/OGR 1.9.2 \ QGIS 1.9.0-326 / Windows (OSGeo4W-installer) GDAL/OGR 1.9.1 \ QGIS 1.8.0-5 / Windows (OSGeo4W-installer) Other systems not tested

Might be related to: RFC35 http://trac.osgeo.org/gdal/ticket/2671, especially the statement: "One downside is that after deletion of a field, the .dbf file is not truncated as there's no hook for truncation. However this doesn't affect overall functionality. "

Workarounds: Use the 'Save layer as' feature in QGIS or ArcGIS 10.

Attachments (3)

dbf-deletecolumn.zip (3.2 KB) - added by berendsen 6 years ago.
Two shapes, 'dbf-test-afterdelete.dbf' shows the problem. Can be created from 'dbf-test' by deleting a column in the table via GDAL/OGR.
ogrshape_remove_truncatedbf.patch (2.1 KB) - added by Even Rouault 6 years ago.
Add a FTruncate callback to SAHooks and use it in DBFDeleteField() and DBFAlterFieldDefn()
ogrshape_remove_truncatedbf.2.patch (2.1 KB) - added by Even Rouault 6 years ago.
Remove hack (to be applied after shapelib_truncate.patch)

Download all attachments as: .zip

Change History (6)

Changed 6 years ago by berendsen

Attachment: dbf-deletecolumn.zip added

Two shapes, 'dbf-test-afterdelete.dbf' shows the problem. Can be created from 'dbf-test' by deleting a column in the table via GDAL/OGR.

comment:1 Changed 6 years ago by Even Rouault

Summary: EOF-weirdness when deleting columns in .dbf[Shapelib patch] EOF-weirdness when deleting columns in .dbf

This issue has been fixed in GDAL 1.10. I've just done a backport in branch/1.9 in case a 1.9.3 is release some day...

branch/1.9 r26134 "Shape: truncate DBF file when removing a column (partial backport of r23972, #5146)"

Frank, I'm attaching a patch to shapelib to implement the truncation in shapelib itself, which is the proper solution (instead of doing it in OGR as currently). Note that I've only implemented the new FTruncate() SAHooks callback in the context of shp_vsi.c. I've not looked at the SASetupDefaultHooks() implementation of shapelib, but it should be extended to implement FTruncate() (inspiration can certainly be taken from GDAL port POSIX and Windows implementations) and to define the ABI version so that SAHooks can be extended someday. After that, the ogrshape_remove_truncatedbf.patch can be applied to remove the OGR hack.

Changed 6 years ago by Even Rouault

Add a FTruncate callback to SAHooks and use it in DBFDeleteField() and DBFAlterFieldDefn()

Changed 6 years ago by Even Rouault

Remove hack (to be applied after shapelib_truncate.patch)

comment:2 Changed 6 years ago by berendsen

Thanks very much for the quick fix. I don't know whether I'll have time to create a build at work today, so I might have to wait until OSGeo4W/QGIS carries the patch.

comment:3 Changed 4 months ago by Even Rouault

Milestone: closed_because_of_github_migration
Resolution: wontfix
Status: newclosed

This ticket has been automatically closed because Trac is no longer used for GDAL bug tracking, since the project has migrated to GitHub?. If you believe this ticket is still valid, you may file it to https://github.com/OSGeo/gdal/issues if it is not already reported there.

Note: See TracTickets for help on using tickets.