Opened 15 years ago

Closed 9 years ago

Last modified 7 years ago

#809 closed defect (duplicate)

shape driver no longer recognizes INT columns in DBF file?

Reported by: Markus Neteler Owned by: warmerdam
Priority: high Milestone:
Component: OGR_SF Version: svn-trunk
Severity: major Keywords:
Cc:

Description (last modified by warmerdam)

Frank,

I have updated the dbf driver in GRASS 6.1-CVS to the 1.2.6
version (dbfopen.c, shpopen.c shapefil.h).

Now I face the problem that the 'cat' (GRASS ID) column is no
longer detected as INT but as FLOAT. Further testing showed that
it seems to be a new bug (?) in OGR:

ogrinfo -al -summary ~/grassdata/pat/neteler/dbf/mygrafo_bclosed.dbf
ERROR 4: Unable to open
/home/neteler/grassdata/pat/neteler/dbf/mygrafo_bclosed.shp or
/home/neteler/grassdata/pat/neteler/dbf/mygrafo_bclosed.SHP.
INFO: Open of `/home/neteler/grassdata/pat/neteler/dbf/mygrafo_bclosed.dbf'
using driver `ESRI Shapefile' successful.

Layer name: mygrafo_bclosed
Geometry: None
Feature Count: 3
Layer SRS WKT:
(unknown)
cat: Real (11.0)
cat2: Real (11.0)
label: String (80.0)
forward: Real (20.6)
backward: Real (20.6)
length: Real (20.6)

I have an old statically compiled dbfinfo here which reports correctly
dbfinfo ~/grassdata/pat/neteler/dbf/mygrafo_bclosed.dbf
Info for /home/neteler/grassdata/pat/neteler/dbf/mygrafo_bclosed.dbf
6 Columns,  3 Records in file
            cat         integer  (11,0)
           cat2         integer  (11,0)
          label          string  (80,0)
        forward           float  (20,6)
       backward           float  (20,6)
         length           float  (20,6)

I don't know currently how to fix that (go back to GDAL 1.1.x is not
what I want to do).

Hopefully you have a suggestion,

thanks

 Markus

Change History (6)

comment:1 Changed 15 years ago by warmerdam

Markus, 

The quick patch is to change dbfopen.c (DBFGetFieldInfo) from this:

    else if( psDBF->pachFieldType[iField] == 'N' 
             || psDBF->pachFieldType[iField] == 'F'
             || psDBF->pachFieldType[iField] == 'D' )
    {
	if( psDBF->panFieldDecimals[iField] > 0 
            || psDBF->panFieldSize[iField] > 10 )
	    return( FTDouble );
	else
	    return( FTInteger );
    }

To this:
   else if( psDBF->pachFieldType[iField] == 'N' 
             || psDBF->pachFieldType[iField] == 'F'
             || psDBF->pachFieldType[iField] == 'D' )
    {
	if( psDBF->panFieldDecimals[iField] > 0 )
	    return( FTDouble );
	else
	    return( FTInteger );
    }

The change was put in so that fields to large to represent as an 
integer (such as 64bit integer values from ArcGIS) would get treated
as doubles instead.  But I can see it may cause unexpected problems. 

I will review what to do more permanently. 

comment:2 Changed 14 years ago by werchowyna@…

(In reply to comment #1)
> Markus, 
> 
> The quick patch is to change dbfopen.c (DBFGetFieldInfo) from this:
> 
>     else if( psDBF->pachFieldType[iField] == 'N' 
>              || psDBF->pachFieldType[iField] == 'F'
>              || psDBF->pachFieldType[iField] == 'D' )
>     {
> 	if( psDBF->panFieldDecimals[iField] > 0 
>             || psDBF->panFieldSize[iField] > 10 )
> 	    return( FTDouble );
> 	else
> 	    return( FTInteger );
>     }
> 
> To this:
>    else if( psDBF->pachFieldType[iField] == 'N' 
>              || psDBF->pachFieldType[iField] == 'F'
>              || psDBF->pachFieldType[iField] == 'D' )
>     {
> 	if( psDBF->panFieldDecimals[iField] > 0 )
> 	    return( FTDouble );
> 	else
> 	    return( FTInteger );
>     }
> 
> The change was put in so that fields to large to represent as an 
> integer (such as 64bit integer values from ArcGIS) would get treated
> as doubles instead.  But I can see it may cause unexpected problems. 
> 
> I will review what to do more permanently. 

Frank,

If you find time to take a look at this issue, please do.

It is still present. Any INTEGER coulumns in dbf files v.out.ogr creates are
always reported as REAL by ogrinfo. What's worse, when import such data back
into Grass I get not the same what was my original data. Smells like (not
dangerous, propably, but anyway) data corruption.

Using Grass 6.1 and GDAL 1.3.1 both fresh from CVS.

Cheers,
Maciek

comment:3 Changed 11 years ago by msieczka

Milestone: 1.6.0
Severity: normalmajor
Version: unspecifiedsvn-trunk

From PostGIS ML via QGIS ML http://www.nabble.com/forum/ViewPost.jtp?post=20704915&framed=y :

Tell the QGIS developers to expand their notion of an int to 11 digits. The whole problem is the mismatch between formal typing in C/C++ (integer, float, double) and the "typing" used in DBF headers (number(n,m)). QGIS interprets anything over 10 digits as float so that it doesn't accidentally bust an integer register trying to load 10B into an int. The reason PostGIS uses 11 digits is because valid *negative* integers, like -1B require 11 digits.

comment:4 Changed 11 years ago by msieczka

Somehow I screwed formatting. Pasting the message here again hoping it'll look better this time:

Tell the QGIS developers to expand their notion of an int to 11 digits. The whole problem is the mismatch between formal typing in C/C++ (integer, float, double) and the "typing" used in DBF headers (number(n,m)). QGIS interprets anything over 10 digits as float so that it doesn't accidentally bust an integer register trying to load 10B into an int. The reason PostGIS uses 11 digits is because valid *negative* integers, like -1B require 11 digits.

comment:5 Changed 9 years ago by warmerdam

Description: modified (diff)
Resolution: duplicate
Status: assignedclosed

I'm making #3615 the official ticket to address wide integer fields in dbf.

comment:6 Changed 7 years ago by Even Rouault

Milestone: 1.6.4

Milestone 1.6.4 deleted

Note: See TracTickets for help on using tickets.