Opened 14 years ago
Last modified 14 years ago
#3236 closed defect
Ogr2ogr stops appending to .shp file at 2 GB — at Version 1
Reported by: | Jukka Rahkonen | Owned by: | chaitanya |
---|---|---|---|
Priority: | normal | Milestone: | |
Component: | OGR_SF | Version: | unspecified |
Severity: | normal | Keywords: | Shape |
Cc: | warmerdam |
Description (last modified by )
I was combining shapefiles together with the following command: ogr2ogr -f "ESRI Shapefile" -append -update combined_shapefile.shp shapefile_number_n.shp -nln combined_shapefile.
Process is progressing well to the point where the .shp file is reaching 2 GB size. Exact file size then was 2 147 484 604 bytes. What happens is that ogr2ogr starts to pring these error messages: ERROR 1: Error in psSHP->sHooks.FSeek() or fwrite() writing object to .shp file.
Shapefiles to be combined are still processed by ogr2ogr. The .dbf file is growing in size and it can obviously grow however big. I reached 32 GB .dbf file size in my run. Also the .shx file is growing.
Frank Warmerdam has written: "The .shp format explicitly uses 32bit offsets and so cannot practically go over 8GB (it actually uses 32bit offsets to 16bit words) and I would not trust compatability over 4GB." In real life the .shp file size seems to be limited to 2 GB. In addition, it is seldom reasonable to continue appending data into shape file if the .shp file is not updating any more, even if the attributes are being added to .dbf file. Perhaps ogr2ogr could just stop with error, or alternatively send some more informative error message than now.
I used ogr2ogr from FWTools 2.4.2. and I could repeat the error with two datasets and with two computers, one running Windows XP Pro and another Windows Server 2003.
I made also another trial by appending the input shapefiles to MapInfo output format. In this trial it was the attribute file .DAT which stopped growing at 2 GB file size. Process continues without any error messages. I thing it would be better to get some error message because the result is not what the user is assuming.
Change History (2)
comment:1 by , 14 years ago
Cc: | added |
---|---|
Component: | Utilities → OGR_SF |
Description: | modified (diff) |
Keywords: | Shape added |
Owner: | changed from | to
by , 14 years ago
Attachment: | 3236.4.diff added |
---|
Patch to fix some error reporting for file operations and detect file size overflow. Also adds a test.
Chaitanya,
Could you review this problem and propose improvements to error out properly at the 2GB mark? Possibly some of the changes will need to be upstreamed by me.
Keep in mind it is important for us to preserve the existing ability to read and write .dbf files much larger than 2GB.