Opened 11 years ago

Closed 10 years ago

#3236 closed defect (fixed)

Ogr2ogr stops appending to .shp file at 2 GB

Reported by: Jukka Rahkonen Owned by: chaitanya
Priority: normal Milestone:
Component: OGR_SF Version: unspecified
Severity: normal Keywords: Shape
Cc: warmerdam

Description (last modified by warmerdam)

I was combining shapefiles together with the following command: ogr2ogr -f "ESRI Shapefile" -append -update combined_shapefile.shp shapefile_number_n.shp -nln combined_shapefile.

Process is progressing well to the point where the .shp file is reaching 2 GB size. Exact file size then was 2 147 484 604 bytes. What happens is that ogr2ogr starts to pring these error messages: ERROR 1: Error in psSHP->sHooks.FSeek() or fwrite() writing object to .shp file.

Shapefiles to be combined are still processed by ogr2ogr. The .dbf file is growing in size and it can obviously grow however big. I reached 32 GB .dbf file size in my run. Also the .shx file is growing.

Frank Warmerdam has written: "The .shp format explicitly uses 32bit offsets and so cannot practically go over 8GB (it actually uses 32bit offsets to 16bit words) and I would not trust compatability over 4GB." In real life the .shp file size seems to be limited to 2 GB. In addition, it is seldom reasonable to continue appending data into shape file if the .shp file is not updating any more, even if the attributes are being added to .dbf file. Perhaps ogr2ogr could just stop with error, or alternatively send some more informative error message than now.

I used ogr2ogr from FWTools 2.4.2. and I could repeat the error with two datasets and with two computers, one running Windows XP Pro and another Windows Server 2003.

I made also another trial by appending the input shapefiles to MapInfo? output format. In this trial it was the attribute file .DAT which stopped growing at 2 GB file size. Process continues without any error messages. I thing it would be better to get some error message because the result is not what the user is assuming.

Attachments (1)

3236.4.diff (14.3 KB) - added by chaitanya 11 years ago.
Patch to fix some error reporting for file operations and detect file size overflow. Also adds a test.

Download all attachments as: .zip

Change History (8)

comment:1 Changed 11 years ago by warmerdam

Cc: warmerdam added
Component: UtilitiesOGR_SF
Description: modified (diff)
Keywords: Shape added
Owner: changed from warmerdam to chaitanya

Chaitanya,

Could you review this problem and propose improvements to error out properly at the 2GB mark? Possibly some of the changes will need to be upstreamed by me.

Keep in mind it is important for us to preserve the existing ability to read and write .dbf files much larger than 2GB.

Changed 11 years ago by chaitanya

Attachment: 3236.4.diff added

Patch to fix some error reporting for file operations and detect file size overflow. Also adds a test.

comment:2 Changed 11 years ago by chaitanya

Status: newassigned

Frank,

This patch moves the .shp file size limit to 4 GB by using unsigned int for filesize variable in SHPInfo object. Also added code to detect integer overflow for that variable while appending to the file.

comment:3 Changed 11 years ago by warmerdam

Chaitanya,

Will you need me to test such a file with ArcGIS or have you already done compatability testing?

comment:4 Changed 11 years ago by chaitanya

Frank,

I didn't do compatibility testing with ArcGIS Explorer. The application kept crashing before I open any file.

It worked with ogr2ogr.

You can generate a file of that size by running the test added in the patch. You need to change the directory it is created so it won't be deleted by the cleanup.

comment:5 Changed 11 years ago by chaitanya

Applied the patch r18413 in trunk to handle the file size limits.

With this patch OGR can read/write .shp files upto 4GB in size. However, files more than 2GB in size need to be tested with other tools for compatibility.

comment:6 Changed 10 years ago by warmerdam

I have upstreamed these changes into Shapelib to avoid further reversions as happened in (r18678 and r18679) and corrected in (r18880 and r18681).

comment:7 Changed 10 years ago by chaitanya

Resolution: fixed
Status: assignedclosed
Note: See TracTickets for help on using tickets.