Ticket #3236 (closed defect: fixed)
Ogr2ogr stops appending to .shp file at 2 GB
|Reported by:||jratike80||Owned by:||chaitanya|
Description (last modified by warmerdam) (diff)
I was combining shapefiles together with the following command: ogr2ogr -f "ESRI Shapefile" -append -update combined_shapefile.shp shapefile_number_n.shp -nln combined_shapefile.
Process is progressing well to the point where the .shp file is reaching 2 GB size. Exact file size then was 2 147 484 604 bytes. What happens is that ogr2ogr starts to pring these error messages: ERROR 1: Error in psSHP->sHooks.FSeek() or fwrite() writing object to .shp file.
Shapefiles to be combined are still processed by ogr2ogr. The .dbf file is growing in size and it can obviously grow however big. I reached 32 GB .dbf file size in my run. Also the .shx file is growing.
Frank Warmerdam has written: "The .shp format explicitly uses 32bit offsets and so cannot practically go over 8GB (it actually uses 32bit offsets to 16bit words) and I would not trust compatability over 4GB." In real life the .shp file size seems to be limited to 2 GB. In addition, it is seldom reasonable to continue appending data into shape file if the .shp file is not updating any more, even if the attributes are being added to .dbf file. Perhaps ogr2ogr could just stop with error, or alternatively send some more informative error message than now.
I used ogr2ogr from FWTools 2.4.2. and I could repeat the error with two datasets and with two computers, one running Windows XP Pro and another Windows Server 2003.
I made also another trial by appending the input shapefiles to MapInfo? output format. In this trial it was the attribute file .DAT which stopped growing at 2 GB file size. Process continues without any error messages. I thing it would be better to get some error message because the result is not what the user is assuming.