shape driver no longer recognizes INT columns in DBF file?

[neteler]

Frank,

I have updated the dbf driver in GRASS 6.1-CVS to the 1.2.6
version (dbfopen.c, shpopen.c shapefil.h).

Now I face the problem that the 'cat' (GRASS ID) column is no
longer detected as INT but as FLOAT. Further testing showed that
it seems to be a new bug (?) in OGR:

ogrinfo -al -summary ~/grassdata/pat/neteler/dbf/mygrafo_bclosed.dbf
ERROR 4: Unable to open
/home/neteler/grassdata/pat/neteler/dbf/mygrafo_bclosed.shp or
/home/neteler/grassdata/pat/neteler/dbf/mygrafo_bclosed.SHP.
INFO: Open of `/home/neteler/grassdata/pat/neteler/dbf/mygrafo_bclosed.dbf'
using driver `ESRI Shapefile' successful.

Layer name: mygrafo_bclosed
Geometry: None
Feature Count: 3
Layer SRS WKT:
(unknown)
cat: Real (11.0)
cat2: Real (11.0)
label: String (80.0)
forward: Real (20.6)
backward: Real (20.6)
length: Real (20.6)

I have an old statically compiled dbfinfo here which reports correctly
dbfinfo ~/grassdata/pat/neteler/dbf/mygrafo_bclosed.dbf
Info for /home/neteler/grassdata/pat/neteler/dbf/mygrafo_bclosed.dbf
6 Columns,  3 Records in file
            cat         integer  (11,0)
           cat2         integer  (11,0)
          label          string  (80,0)
        forward           float  (20,6)
       backward           float  (20,6)
         length           float  (20,6)

I don't know currently how to fix that (go back to GDAL 1.1.x is not
what I want to do).

Hopefully you have a suggestion,

thanks

 Markus

[warmerdam]

Markus, 

The quick patch is to change dbfopen.c (DBFGetFieldInfo) from this:

    else if( psDBF->pachFieldType[iField] == 'N' 
             || psDBF->pachFieldType[iField] == 'F'
             || psDBF->pachFieldType[iField] == 'D' )
    {
	if( psDBF->panFieldDecimals[iField] > 0 
            || psDBF->panFieldSize[iField] > 10 )
	    return( FTDouble );
	else
	    return( FTInteger );
    }

To this:
   else if( psDBF->pachFieldType[iField] == 'N' 
             || psDBF->pachFieldType[iField] == 'F'
             || psDBF->pachFieldType[iField] == 'D' )
    {
	if( psDBF->panFieldDecimals[iField] > 0 )
	    return( FTDouble );
	else
	    return( FTInteger );
    }

The change was put in so that fields to large to represent as an 
integer (such as 64bit integer values from ArcGIS) would get treated
as doubles instead.  But I can see it may cause unexpected problems. 

I will review what to do more permanently. 

[werchowyna@…]

(In reply to comment #1)
> Markus, 
> 
> The quick patch is to change dbfopen.c (DBFGetFieldInfo) from this:
> 
>     else if( psDBF->pachFieldType[iField] == 'N' 
>              || psDBF->pachFieldType[iField] == 'F'
>              || psDBF->pachFieldType[iField] == 'D' )
>     {
> 	if( psDBF->panFieldDecimals[iField] > 0 
>             || psDBF->panFieldSize[iField] > 10 )
> 	    return( FTDouble );
> 	else
> 	    return( FTInteger );
>     }
> 
> To this:
>    else if( psDBF->pachFieldType[iField] == 'N' 
>              || psDBF->pachFieldType[iField] == 'F'
>              || psDBF->pachFieldType[iField] == 'D' )
>     {
> 	if( psDBF->panFieldDecimals[iField] > 0 )
> 	    return( FTDouble );
> 	else
> 	    return( FTInteger );
>     }
> 
> The change was put in so that fields to large to represent as an 
> integer (such as 64bit integer values from ArcGIS) would get treated
> as doubles instead.  But I can see it may cause unexpected problems. 
> 
> I will review what to do more permanently. 

Frank,

If you find time to take a look at this issue, please do.

It is still present. Any INTEGER coulumns in dbf files v.out.ogr creates are
always reported as REAL by ogrinfo. What's worse, when import such data back
into Grass I get not the same what was my original data. Smells like (not
dangerous, propably, but anyway) data corruption.

Using Grass 6.1 and GDAL 1.3.1 both fresh from CVS.

Cheers,
Maciek

[msieczka]

From PostGIS ML via QGIS ML  http://www.nabble.com/forum/ViewPost.jtp?post=20704915&framed=y :

Tell the QGIS developers to expand their notion of an int to 11 digits. The whole problem is the mismatch between formal typing in C/C++ (integer, float, double) and the "typing" used in DBF headers (number(n,m)). QGIS interprets anything over 10 digits as float so that it doesn't accidentally bust an integer register trying to load 10B into an int. The reason PostGIS uses 11 digits is because valid *negative* integers, like -1B require 11 digits.

[msieczka]

Somehow I screwed formatting. Pasting the message here again hoping it'll look better this time:

Tell the QGIS developers to expand their notion of an int to 11 digits. The whole problem is the mismatch between formal typing in C/C++ (integer, float, double) and the "typing" used in DBF headers (number(n,m)). QGIS interprets anything over 10 digits as float so that it doesn't accidentally bust an integer register trying to load 10B into an int. The reason PostGIS uses 11 digits is because valid *negative* integers, like -1B require 11 digits.

[warmerdam]

I'm making #3615 the official ticket to address wide integer fields in dbf.

[rouault]

Milestone 1.6.4 deleted