Opened 14 years ago

Last modified 14 years ago

#3236 closed defect

Ogr2ogr stops appending to .shp file at 2 GB — at Version 1

Reported by: Jukka Rahkonen Owned by: chaitanya
Priority: normal Milestone:
Component: OGR_SF Version: unspecified
Severity: normal Keywords: Shape
Cc: warmerdam

Description (last modified by warmerdam)

I was combining shapefiles together with the following command: ogr2ogr -f "ESRI Shapefile" -append -update combined_shapefile.shp shapefile_number_n.shp -nln combined_shapefile.

Process is progressing well to the point where the .shp file is reaching 2 GB size. Exact file size then was 2 147 484 604 bytes. What happens is that ogr2ogr starts to pring these error messages: ERROR 1: Error in psSHP->sHooks.FSeek() or fwrite() writing object to .shp file.

Shapefiles to be combined are still processed by ogr2ogr. The .dbf file is growing in size and it can obviously grow however big. I reached 32 GB .dbf file size in my run. Also the .shx file is growing.

Frank Warmerdam has written: "The .shp format explicitly uses 32bit offsets and so cannot practically go over 8GB (it actually uses 32bit offsets to 16bit words) and I would not trust compatability over 4GB." In real life the .shp file size seems to be limited to 2 GB. In addition, it is seldom reasonable to continue appending data into shape file if the .shp file is not updating any more, even if the attributes are being added to .dbf file. Perhaps ogr2ogr could just stop with error, or alternatively send some more informative error message than now.

I used ogr2ogr from FWTools 2.4.2. and I could repeat the error with two datasets and with two computers, one running Windows XP Pro and another Windows Server 2003.

I made also another trial by appending the input shapefiles to MapInfo output format. In this trial it was the attribute file .DAT which stopped growing at 2 GB file size. Process continues without any error messages. I thing it would be better to get some error message because the result is not what the user is assuming.

Change History (2)

comment:1 by warmerdam, 14 years ago

Cc: warmerdam added
Component: UtilitiesOGR_SF
Description: modified (diff)
Keywords: Shape added
Owner: changed from warmerdam to chaitanya

Chaitanya,

Could you review this problem and propose improvements to error out properly at the 2GB mark? Possibly some of the changes will need to be upstreamed by me.

Keep in mind it is important for us to preserve the existing ability to read and write .dbf files much larger than 2GB.

by chaitanya, 14 years ago

Attachment: 3236.4.diff added

Patch to fix some error reporting for file operations and detect file size overflow. Also adds a test.

Note: See TracTickets for help on using tickets.