Changes between Version 9 and Version 10 of rfc31_ogr_64
- Timestamp:
- Jan 28, 2015, 4:15:41 AM (9 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
rfc31_ogr_64
v9 v10 12 12 13 13 Feature id's will be handled as type "GIntBig" instead of "long" internally. 14 This will include the nFID field of the OGRFeature. The existing GetFID() and SetFID() methods on the OGRFeature use type long. It is difficult to change GetFID() without significant disruption to existing application code, but SetFID() can be changed to accept GIntBig instead of long. Existing GetFID() will be deprecated in favor of GetFID64(), and will throw a warning if it appears that long cannot hold the FID. So the changes in the OGRFeature class are: 15 16 {{{ 17 long GetFID() CPL_WARN_DEPRECATED("Use GetFID64()"); 18 GIntBig GetFID64(); 14 This will include the nFID field of the OGRFeature. The existing GetFID() and SetFID() methods on the OGRFeature use type long and are changed to return (respectively accept) GIntBig instead. The change of return type for GetFID() will require application code to carefully adapt to avoid potential issues (for example if GetFID() is used in printf-like expression). SetFID() change should be mostly transparent. So the changes in the OGRFeature class are: 15 16 {{{ 17 GIntBig GetFID(); 19 18 OGRErr SetFID(GIntBig nFID ); 20 19 }}} … … 23 22 24 23 {{{ 25 long CPL_DLL OGR_F_GetFID( OGRFeatureH ) CPL_WARN_DEPRECATED("Use OGR_F_GetFID64() instead"); 26 GIntBig CPL_DLL OGR_F_GetFID64( OGRFeatureH ); 24 GIntBig CPL_DLL OGR_F_GetFID( OGRFeatureH ); 27 25 OGRErr CPL_DLL OGR_F_SetFID( OGRFeatureH, GIntBig ); 28 26 }}} … … 32 30 A layer that can discover in a relatively cheap way that it holds features with 64bit FID should advertize the OLMD_FID64 metadata item to "YES", so ogr2ogr can pass the FID64 creation option to drivers that support it. 33 31 34 The OGRLayer class allows several operations based on the FID. The signature of these will be *altered* to accept GIntBig instead of long. In theory this should not require any changes to application code since long can be converted to GIntBig losslessly. However, all existing OGR drivers require changes, including private drivers. This will also result in a backwards incompatible change in the C ABI. While we are at it, we want GetFeatureCount() to be able to return more than 2 billion record (currently returning 32 bit integer) . For the same reason as it is dangerous modifying the return type of GetFeatureCount(), we introduce GetFeatureCount64() and deprecate GetFeatureCount(), removing its virtual attribute.32 The OGRLayer class allows several operations based on the FID. The signature of these will be *altered* to accept GIntBig instead of long. In theory this should not require any changes to application code since long can be converted to GIntBig losslessly. However, all existing OGR drivers require changes, including private drivers. This will also result in a backwards incompatible change in the C ABI. While we are at it, we want GetFeatureCount() to be able to return more than 2 billion record (currently returning 32 bit integer), and thus it will return GIntBig. Similarly to GetFID(), this change of return type will require caution in application code. 35 33 36 34 So at the OGRLayer C++ class level: … … 39 37 virtual OGRErr DeleteFeature( GIntBig nFID ); 40 38 virtual OGRErr SetNextByIndex( GIntBig nIndex ); 41 int GetFeatureCount( int bForce = TRUE ) CPL_WARN_DEPRECATED("Use GetFeatureCount64() instead"); 42 virtual GIntBig GetFeatureCount64( int bForce = TRUE ); 39 virtual GIntBig GetFeatureCount( int bForce = TRUE ); 43 40 }}} 44 41 … … 48 45 OGRErr CPL_DLL OGR_L_DeleteFeature( OGRLayerH, GIntBig ); 49 46 OGRErr CPL_DLL OGR_L_SetNextByIndex( OGRLayerH, GIntBig ); 50 int CPL_DLL OGR_L_GetFeatureCount( OGRLayerH, int ) CPL_WARN_DEPRECATED("Use OGR_L_GetFeatureCount64() instead"); 51 GIntBig CPL_DLL OGR_L_GetFeatureCount64( OGRLayerH, int ); 47 GIntBig CPL_DLL OGR_L_GetFeatureCount( OGRLayerH, int ); 52 48 }}} 53 49 … … 134 130 Apart from the mechanical changes due to interface changes, the detailed list of changes is : 135 131 * Shapefile: OFTInteger fields are created by default with a width of 9 characters, so to be unambiguously read as OFTInteger (and if specifying integer that require 10 or 11 characters. the field is dynamically extended like managed since a few versions). OFTInteger64 fields are created by default with a width of 18 digits, so to be unambiguously read as OFTInteger64, and extented to 19 or 20 if needed. Integer fields of width between 10 and 18 will be read as OFTInteger64. Above they will be treated as OFTReal. In previous GDAL versions, Integer fields were created with a default with of 10, and thus will be now read as OFTInteger64. An open option, DETECT_TYPE=YES, can be specified so as OGR does a full scan of the DBF file to see if integer fields of size 10 or 11 hold 32 bit or 64 bit values and adjust the type accordingly (and same for integer fields of size 19 or 20, in case of overflow of 64 bit integer, OFTReal is chosen) 136 * PG: updated to read and create OFTInteger64 as INT8 and OFTInteger64List as bigint[]. 64 bit FIDs are supported. By default, on layer creation, the FID field is created as a SERIAL (32 bit integer) to avoid compatibility issues. The FID64=YES creation option can be passed to create it as a BIGSERIAL instead. If needed, the drivers will dynamically alter the schema to extend a 32 bit integer FID field to 64 bit. GetFeatureCount 64() modified to return 64 bit values. OLMD_FID64 = "YES" advertized as soon as the FID column is 64 bit.132 * PG: updated to read and create OFTInteger64 as INT8 and OFTInteger64List as bigint[]. 64 bit FIDs are supported. By default, on layer creation, the FID field is created as a SERIAL (32 bit integer) to avoid compatibility issues. The FID64=YES creation option can be passed to create it as a BIGSERIAL instead. If needed, the drivers will dynamically alter the schema to extend a 32 bit integer FID field to 64 bit. GetFeatureCount() modified to return 64 bit values. OLMD_FID64 = "YES" advertized as soon as the FID column is 64 bit. 137 133 * PGDump: Integer64, Integer64List and 64 bit FID supported in read/write. FID64=YES creation option available. 138 134 * GeoJSON: Integer64, Integer64List and 64 bit FID supported in read/write. The 64 bit variants are reported only if needed, otherwise OFTInteger/OFTIntegerList is used. OLMD_FID64 = "YES" advertized if needed 139 135 * CSV: Integer64 supported in read/write, including the autodetection feature of field types. 140 * GPKG: Integer64 and 64 bit FID supported in read/write. Conforming with the GeoPackage spec, "INT" or "INTEGER" columns are considered 64 bits, whereas "MEDIUMINT" is considered 32 bit. OLMD_FID64 = "YES" advertized as soon as MAX(fid_column) is 64 bit. GetFeatureCount 64() modified to return 64 bit values.141 * SQLite: Integer64 and 64 bit FID supported in read/write. On write, Integer64 are createad as "BIGINT" and on read BIGINT or INT8 are considered as Integer64. However it might be possible that databases produced by other tools are created with "INTEGER" and hold 64 bit values, in which case OGR will not be able to detect it. The OGR_PROMOTE_TO_INTEGER64=YES configuration option can then be passed to workaround that issue. OLMD_FID64 = "YES" advertized as soon as MAX(fid_column) is 64 bit. GetFeatureCount 64() modified to return 64 bit values.142 * MySQL: Integer64 and 64 bit FID supported in read/write. Similarly to PG, FID column is created as 32 bit by default, unless FID64=YES creation option is specified. OLMD_FID64 = "YES" advertized as soon as the FID column is 64 bit. GetFeatureCount 64() modified to return 64 bit values.136 * GPKG: Integer64 and 64 bit FID supported in read/write. Conforming with the GeoPackage spec, "INT" or "INTEGER" columns are considered 64 bits, whereas "MEDIUMINT" is considered 32 bit. OLMD_FID64 = "YES" advertized as soon as MAX(fid_column) is 64 bit. GetFeatureCount() modified to return 64 bit values. 137 * SQLite: Integer64 and 64 bit FID supported in read/write. On write, Integer64 are createad as "BIGINT" and on read BIGINT or INT8 are considered as Integer64. However it might be possible that databases produced by other tools are created with "INTEGER" and hold 64 bit values, in which case OGR will not be able to detect it. The OGR_PROMOTE_TO_INTEGER64=YES configuration option can then be passed to workaround that issue. OLMD_FID64 = "YES" advertized as soon as MAX(fid_column) is 64 bit. GetFeatureCount() modified to return 64 bit values. 138 * MySQL: Integer64 and 64 bit FID supported in read/write. Similarly to PG, FID column is created as 32 bit by default, unless FID64=YES creation option is specified. OLMD_FID64 = "YES" advertized as soon as the FID column is 64 bit. GetFeatureCount() modified to return 64 bit values. 143 139 * OCI: Integer64 and 64 bit FID supported in read/write. Detecting Integer/Integer64 on read is tricky since there's only a NUMBER SQL type with a field width. It is assumed that if the width is <= 9 or if it is the unspecified value (38), then it is a Integer. On creation, OGR will set a width of 20 for OFTInteger64, so a NUMBER without decimal part and with a width of 20 will be considered as a Integer64. 144 * MEM: Integer64 and 64 bit FID supported in read/write. GetFeatureCount 64() modified to return 64 bit values.145 * VRT: Integer64, Integer64List and 64 bit FID supported in read/write. GetFeatureCount 64() modified to return 64 bit values.140 * MEM: Integer64 and 64 bit FID supported in read/write. GetFeatureCount() modified to return 64 bit values. 141 * VRT: Integer64, Integer64List and 64 bit FID supported in read/write. GetFeatureCount() modified to return 64 bit values. 146 142 * JML: Integer64 supported on creation (created as "OBJECT"). On read, returned as String 147 * GML: Integer64, Integer64List and 64 bit FID supported in read/write. GetFeatureCount 64() modified to return 64 bit values.148 * WFS: Integer64, Integer64List and 64 bit FID supported in read/write. GetFeatureCount 64() modified to return 64 bit values.149 * CartoDB: Integer64 supported on creation. On read returned as Real (CartoDB only advertizes a 'Number' type). GetFeatureCount 64() modified to return 64 bit values.143 * GML: Integer64, Integer64List and 64 bit FID supported in read/write. GetFeatureCount() modified to return 64 bit values. 144 * WFS: Integer64, Integer64List and 64 bit FID supported in read/write. GetFeatureCount() modified to return 64 bit values. 145 * CartoDB: Integer64 supported on creation. On read returned as Real (CartoDB only advertizes a 'Number' type). GetFeatureCount() modified to return 64 bit values. 150 146 * XLSX: Integer64 supported in read/write. 151 147 * ODS: Integer64 supported in read/write. 152 * MSSQLSpatial: GetFeatureCount 64() modified to return 64 bit values. No Integer64 support implemented although could likely be done.148 * MSSQLSpatial: GetFeatureCount() modified to return 64 bit values. No Integer64 support implemented although could likely be done. 153 149 * OSM: FID is now always set even when sizeof(long) != 8 154 150 * LIBKML: KML 'uint' advertized as Integer64. 151 * MITAB: Change the way FID of Seamless tables are generated to make it more robust and accept arbitrary number of index tables made of an arbitrary number of features, by using full 64bit width of IDs 155 152 156 153 == Test Suite == … … 158 155 The test suite is extended to test the new capabilities: 159 156 * core SetField/GetField methods 160 * updated drivers: Shapefile, PG, GeoJSON, CSV, GPKG, SQLite, MySQL, VRT, GML, XLSX, ODS 157 * updated drivers: Shapefile, PG, GeoJSON, CSV, GPKG, SQLite, MySQL, VRT, GML, XLSX, ODS, MITAB 161 158 * OGR SQL 162 159 * option -mapFieldType of ogr2ogr … … 166 163 === Driver Code Changes === 167 164 168 * All drivers implementing SetNextByIndex(), DeleteFeature() or GetFeature() will need modest changes. 169 170 * All drivers implementing GetFeatureCount() should be modified to implement GetFeatureCount64() instead 165 * All drivers implementing SetNextByIndex(), DeleteFeature(), GetFeature(), GetFeatureCount() will need to change their prototype and do modest changes. 171 166 172 167 * Drivers supporting CreateField() likely ought to be extended to support OFTInteger64 as an integer/real/string field if nothing else is available (and if bApproxOK is TRUE). ogr2ogr will convert Integer64 to Real if Integer64 support is not advertized 173 168 174 * Drivers reporting FIDs via Debug statements, printf's or using sprintfs like statements to format them for output have been updated to use CPL_FRMT_GIB to format the FID. Failure to make these changes may result in code crashing. Due to the use of GCC annotation to advertize printf()-like formatting syntax in CPL functions, we are reasonably confident to have done the required changes in in-tree drivers (except in some proprietary drivers, like SDE, IDB, INGRES, ArcObjects, where this couldn't be compiled-checked) 169 * Drivers reporting FIDs via Debug statements, printf's or using sprintfs like statements to format them for output have been updated to use CPL_FRMT_GIB to format the FID. Failure to make these changes may result in code crashing. Due to the use of GCC annotation to advertize printf()-like formatting syntax in CPL functions, we are reasonably confident to have done the required changes in in-tree drivers (except in some proprietary drivers, like SDE, IDB, INGRES, ArcObjects, where this couldn't be compiled-checked). The same holds true for GetFeatureCount() 175 170 176 171 === Application Code ===