ogr2ogr stops writing shp file

[tbsprs]

I am trying to convert a digital elevation model (geo-coordinates and height information) [1] stored in a text file to ESRI Shapefile. The source file is 770MB and contains 35,618,622 rows. I created the following VRT file to prepare the conversion.

<OGRVRTDataSource>
    <OGRVRTLayer name="DGM5_BE">
        <SrcDataSource>DGM5_BE.csv</SrcDataSource>
        <SrcLayer>DGM5_BE</SrcLayer>
        <LayerSRS>EPSG:25833</LayerSRS>
        <GeometryType>wkbPoint</GeometryType>
        <GeometryField encoding="PointFromColumns" x="x" y="y" z="d"/>
    </OGRVRTLayer>
</OGRVRTDataSource>

Here is the command I use:

ogr2ogr -f "ESRI Shapefile" DGM5_BE.shp DGM5_BE.vrt

After a short while the conversion stops with the following error output:

ERROR 1: Failure writing DBF record 31213762.
ERROR 1: Failure writing DBF record 31213779.
ERROR 1: Failure writing DBF record 31213796.
ERROR 1: Failure writing DBF record 31213813.
ERROR 1: Failure writing DBF record 31213830.
ERROR 1: Failure writing DBF record 31213847.
ERROR 1: Error in psSHP->sHooks.Fwrite() while writing object to .shp file.
ERROR 1: Terminating translation prematurely after failed
translation of layer DGM5_BE (use -skipfailures to skip errors)

Another try aborts similar:

ERROR 1: Failure writing DBF record 31212216.
ERROR 1: Failure writing DBF record 31212233.
ERROR 1: Error in psSHP->sHooks.Fwrite() while writing object to .shp file.
ERROR 1: Terminating translation prematurely after failed
translation of layer DGM5_BE (use -skipfailures to skip errors)

I am running Ubuntu 14.04. with GDAL 1.10.1, released 2013/08/26.

[1] ​http://fbarc.stadt-berlin.de/FIS_Broker_Atom/DGM/DGM5_BE.zip

[Even Rouault]

You have reached the 2GB limit for a DBF file. This is because the x,y and d columns are copied as string columns in the DBF. You can avoid that by adding the reportSrcColumn="false" attribute in the GeometryField element. See ​http://www.gdal.org/drv_vrt.html

[tbsprs]

But then the values kept in the d-field are not written into the Shapefile. How can I accompish that?

[Even Rouault]

Likely by manually adding <Field name="d" />

[tbsprs]

I tried the same with the latest build from QGIS. First I import the CSV as delimted data. Then I save the layer as Shapefile. It worked without problem. It seems they are using another method. Here are the file sizes:

   5 DGM5_BE.cpg
1,5G DGM5_BE.dbf
 389 DGM5_BE.prj
952M DGM5_BE.shp
272M DGM5_BE.shx

[Even Rouault]

I'm not sure why you reopened the ticket and what could be fixed. If you properly set up the VRT as indicated this should result in a similar shape as qgis.

[Jukka Rahkonen]

Note: I do not totally believe that the dbf part has a 2 GB size limit. At least I have created much bigger ones with GDAL and also the GDAL shapefile driver page informs us: "Attributes: The dbf format does not have any offsets in it, so it can be arbitrarily large".

[Even Rouault]

Replying to jratike80:

Note: I do not totally believe that the dbf part has a 2 GB size limit. At least I have created much bigger ones with GDAL and also the GDAL shapefile driver page informs us: "Attributes: The dbf format does not have any offsets in it, so it can be arbitrarily large".

Yes, that's true. But now GDAL >= 1.11 should warn when .shp or .dbf exceed 2 GB, to maximize compatibility with other software implementations. As far as the errors reported by the reporter, I'm not sure why he got them. I didn't replicate them. Perhaps disk full ? Might be that since there are both errors while writing the .dbf and .shp, but the .shp remains < 1 GB, so I suspect that the huge DBF cause the disk to be full.

[tbsprs]

Really, it seems to have been a full disk. Meanwhile I could produce the Shapefiles as suggested by rouault. Thanks you for your great analysis and patience.