| | 1 | = RFC 46: GDAL/OGR unification = |
| | 2 | |
| | 3 | Author: Even Rouault[[BR]] |
| | 4 | Contact: even dot rouault at mines dash paris dot org[[BR]] |
| | 5 | Status: Development |
| | 6 | |
| | 7 | == Summary == |
| | 8 | |
| | 9 | In the 1.X series of GDAL/OGR, the GDAL/raster and OGR/vector sides are |
| | 10 | quite different on some aspects even where there is no strong reason for them to be |
| | 11 | different, particularly in the structure of drivers. This RFC aims at unifying |
| | 12 | the OGR driver structure with the GDAL driver structure. The main advantages of |
| | 13 | using the GDAL driver structure are : |
| | 14 | * metadata capabilities : description of driver, extensions, creation options, |
| | 15 | virtual IO capability ... |
| | 16 | * efficient driver identification and opening. |
| | 17 | |
| | 18 | Similarly, OGR datasource and layer classes lack the metadata mechanisms offered |
| | 19 | by the corresponding GDAL dataset and raster band classes. |
| | 20 | |
| | 21 | Another aspect is that the separation between GDAL "datasets" and OGR |
| | 22 | "datasources" is sometimes artificial. Various data containers can accept both |
| | 23 | data types. The list of drivers that have a GDAL side and OGR side |
| | 24 | is : SDTS, PDS, GRASS, KML, Spatialite/Rasterlite, GeoPackage (raster side not |
| | 25 | yet implemented), PostGIS/PostGIS Raster, PDF, PCIDSK, FileGDB (raster side not |
| | 26 | yet implemented). For applications that are interested in both, this currently |
| | 27 | means to open the file twice with different API. And for update mode, for |
| | 28 | file-based drivers, the updates must be done sequentially to avoid opening a file |
| | 29 | twice simultaneously in update mode and making conflicting changes. |
| | 30 | |
| | 31 | == Related RFCs == |
| | 32 | |
| | 33 | There are a few related past RFCs that have never been adopted but strongly |
| | 34 | relate to RFC 46 : |
| | 35 | * [wiki:rfc10_ogropen RFC 10: OGR Open Parameters]. All the functionality |
| | 36 | described in RFC 10 is included in RFC 46, mainly the GDALOpenEx() new API |
| | 37 | * [wiki:rfc25_fast_open RFC 25: Fast Open]. RFC 25 mentions avoiding to |
| | 38 | systematically listing the sibling files to the file being opened. This can |
| | 39 | now achieved in RFC 46 by lazy loading with GDALOpenInfo::GetSiblingFiles(). |
| | 40 | At least Identify() should not trigger GetSiblingFiles(). |
| | 41 | * [wiki:rfc38_ogr_faster_open RFC 38: OGR Faster Open] is completely included |
| | 42 | in RFC 46 through the possibility of using Open(GDALOpenInfo*) in OGR drivers |
| | 43 | |
| | 44 | == Self-assigned development constraints == |
| | 45 | |
| | 46 | The changes should have moderate impact on the existing GDAL/OGR code base, and |
| | 47 | particularly on most of its code, that lies in drivers. |
| | 48 | Existing users of the GDAL/OGR API should also be moderately impacted by the |
| | 49 | changes, if they do not need to use the new offered capabilities. |
| | 50 | |
| | 51 | == Core changes: summary == |
| | 52 | |
| | 53 | * OGRSFDriver extends GDALDriver. |
| | 54 | * Vector drivers can be implemented as GDALDriver. |
| | 55 | * OGRSFDriverRegistrar is a compatibility wrapper around GDALDriverManager |
| | 56 | for legacy OGRSFDriver. |
| | 57 | * OGRDataSource extends GDALDataSource. |
| | 58 | * GDALOpenEx() API is added to be able to open "mixed" datasets. |
| | 59 | * OGRLayer extends GDALMajorObject, thus adding metadata capability. |
| | 60 | * The methods of OGRDataSource related to layers are moved to GDALDataset, |
| | 61 | making it both a raster and vector capable container. |
| | 62 | * Performance improvements in GDALOpenInfo() mechanism. |
| | 63 | * New driver metadata item to describe open options (i.e. deprecate the use |
| | 64 | of configuration option). |
| | 65 | * New driver metadata item to describe layer creation options. |
| | 66 | |
| | 67 | == Core changes: details == |
| | 68 | |
| | 69 | === Drivers and driver registration === |
| | 70 | |
| | 71 | * The OGRSFDriver now extends GDALDriver and is meant as being a legacy way |
| | 72 | of implementing a vector driver. |
| | 73 | It is kept mainbly because, in the current implementation, not all drivers |
| | 74 | have been migrated to being "pure" GDALDriver. |
| | 75 | The CopyDataSource() virtual method has been removed since no in-tree drivers |
| | 76 | implement it. |
| | 77 | The inheritance to GDALDriver make it possible to manage vector drivers by |
| | 78 | the GDALDriverManager, and to be able to attach metadata to them, to document |
| | 79 | driver long name, link to documentation, file extension, datasource creation |
| | 80 | options with the existing GDAL_DMD_* metadata items. |
| | 81 | |
| | 82 | * Drivers directly inheriting from GDALDriver (to be opposed to those inheriting |
| | 83 | from OGRSFDriver) should : |
| | 84 | - declare SetMetadataItem( GDAL_DCAP_VECTOR, "YES" ). |
| | 85 | - implement pfnOpen() for dataset opening |
| | 86 | - optionaly, implement pfnCreate() for dataset creation. For vector drivers, |
| | 87 | the nBands parameter of Create() is supposed to be passed to 0. |
| | 88 | - optionaly, implement pfnDelete() for dataset deletion |
| | 89 | |
| | 90 | * The *C* OGR Driver API will still work with drivers that have been converted as |
| | 91 | "pure" GDALDrivers (this is not true of the C++ OGR Driver API). For example |
| | 92 | OGR_Dr_GetName() calls GDALDriver::GetDescription(), OGR_Dr_CreateDatasource() |
| | 93 | calls GDALDriver::Create(), etc... |
| | 94 | |
| | 95 | * The C++ definition of GDALDriver is extended with the following function |
| | 96 | pointers so that it can work with legacy OGRSFDriver. |
| | 97 | {{{ |
| | 98 | /* For legacy OGR drivers */ |
| | 99 | GDALDataset *(*pfnOpenWithDriverArg)( GDALDriver*, GDALOpenInfo * ); |
| | 100 | GDALDataset *(*pfnCreateVectorOnly)( GDALDriver*, |
| | 101 | const char * pszName, |
| | 102 | char ** papszOptions ); |
| | 103 | CPLErr (*pfnDeleteDataSource)( GDALDriver*, |
| | 104 | const char * pszName ); |
| | 105 | }}} |
| | 106 | They are used by GDALOpenEx(), GDALDriver::Create() and GDALDriver::Delete() |
| | 107 | if the pfnOpen, pfnCreate or pfnDelete pointers are NULL. The OGRSFDriverRegistrar |
| | 108 | class has an implementation of those function pointers that calls the |
| | 109 | legacy C++ OGRSFDriver::Open(), OGRSFDriver::CreateDataSource() and |
| | 110 | OGRSFDriver::DeleteDataSource() virtual methods. |
| | 111 | |
| | 112 | * GDALDriver::Create() can accept nBands == 0 for a vector capable driver. |
| | 113 | |
| | 114 | * GDALDriver::DefaultCreateCopy() can accept a dataset with 0 bands for a |
| | 115 | vector capable driver, and if the output dataset has layer creation capability |
| | 116 | and the source dataset has layers, it copies the layers from the source |
| | 117 | dataset into the target dataset. |
| | 118 | |
| | 119 | * GDALDriver::Identify() now iterates over all kinds of drivers. It has been modified |
| | 120 | to do a first pass on drivers that have an implementation of Identify(). |
| | 121 | If no match is found, it does a second pass on all drivers and use the |
| | 122 | potentially slower Open() as the identification method. |
| | 123 | |
| | 124 | * Related to the above point, the implementations of GDALDriver::pfnIdentify |
| | 125 | function pointer used to return a boolean value to indicate if the passed |
| | 126 | GDALOpenInfo was a match for the driver. For some drivers, this was too |
| | 127 | restrictive so that they were able to implement Identify(). For example |
| | 128 | where the detection logic can return "yes, I definitely recognize that file", |
| | 129 | "no, it is not for me" or "I have not enough elements in GDALOpenInfo to be |
| | 130 | able to tell". That last state can now be advertized with a negative return value. |
| | 131 | |
| | 132 | * The OGRSFDriverRegistrar is trimmed down to be mostly a wrapper around |
| | 133 | GDALDriverManager. In particular, it does not contain any longer a list of drivers. |
| | 134 | The Open(), OpenShared(), ReleaseDataSource(), |
| | 135 | DeregisterDriver() and AutoLoadDrivers() methods are removed from the class. |
| | 136 | This change can have impact on C++ code. A few adaptations in OGR utilities have |
| | 137 | been done to accomodate for those changes. |
| | 138 | The RegisterDriver() API has been kept for legacy OGR drivers and it automatically |
| | 139 | sets SetMetadataItem( GDAL_DCAP_VECTOR, "YES" ). |
| | 140 | The GetDriverCount(), GetDriver() and GetDriverByName() methods delegate to |
| | 141 | GDALDriverManager and make sure to only take into account drivers that have the |
| | 142 | GDAL_DCAP_VECTOR capability. In the case a driver has the same name as GDAL |
| | 143 | and OGR driver, the OGR variant is internally prefixed with OGR_, and GetDriverByName() |
| | 144 | will first try the OGR_ variant. |
| | 145 | The GetOpenDSCount() and GetOpenDS() have now a dummy implementation returning |
| | 146 | 0/NULL. For reference, neither MapServer nor QGIS use those functions. |
| | 147 | |
| | 148 | * OGRRegisterAll() is now an alias of GDALAllRegister(). The past OGRRegisterAll() |
| | 149 | is now renamed OGRRegisterAllInternal() and called by GDALAllRegister(). So, |
| | 150 | GDALAllRegister() and OGRRegisterAll() are now equivalent and register all |
| | 151 | drivers. |
| | 152 | |
| | 153 | * GDALDriverManager has received a few changes : |
| | 154 | - use of a map from driver name to driver object to speed-up GetDriverByName() |
| | 155 | - accept OGR_SKIP and OGR_DRIVER_PATH configuration options for backward |
| | 156 | compatibility. |
| | 157 | - The recommanded separator for driver names in GDAL_SKIP is now comma |
| | 158 | instead of space (similarly to what OGR_SKIP does). This is to make it |
| | 159 | possible to define OGR driver names in |
| | 160 | GDAL_SKIP that have spaces in their names like "ESRI Shapefile" or "MapInfo File". |
| | 161 | If there is no comma in the GDAL_SKIP value, then space separator is |
| | 162 | assumed (backward compatibility). |
| | 163 | - removal of GetHome()/SetHome() methods whose purpose seemed to define an |
| | 164 | alternate path for the search directory of plugins. Those methods only |
| | 165 | existed at the C++ level, and are redundant with GDAL_DRIVER_PATH configuration |
| | 166 | option |
| | 167 | |
| | 168 | * Raster-capable drivers should declare SetMetadataItem( GDAL_DCAP_RASTER, "YES" ). |
| | 169 | All in-tree GDAL drivers have been patched to declare it. But the registration |
| | 170 | code detects if a driver does not declare any of GDAL_DCAP_RASTER nor |
| | 171 | GDAL_DCAP_VECTOR, in which case it declares GDAL_DCAP_RASTER on behalf of the |
| | 172 | un-patched driver, with a debug message inviting to explicitely set it. |
| | 173 | |
| | 174 | * New metadata items : |
| | 175 | - GDAL_DCAP_RASTER=YES / GDAL_DCAP_VECTOR=YES at driver level. |
| | 176 | To declare that a driver has raster/vector capabilities. A driver can declare |
| | 177 | both. |
| | 178 | - GDAL_DMD_EXTENSIONS (with a final S) at driver level. |
| | 179 | This is a small evolution of GDAL_DMD_EXTENSION |
| | 180 | where one can specify several extensions in the value string. The extensions |
| | 181 | are space-separated. For example "shp dbf", "tab mif mid", etc... |
| | 182 | For ease of use, GDALDriver::SetMetadataItem(GDAL_DMD_EXTENSION) also sets |
| | 183 | the passed value as GDAL_DMD_EXTENSIONS, if it is not already set. |
| | 184 | So new code can always use GDAL_DMD_EXTENSIONS. |
| | 185 | - GDAL_DMD_OPENOPTIONLIST at driver level. |
| | 186 | The value of this item is an XML snippet with a format similar to creation |
| | 187 | options. GDALOpenEx(), once it has identified with Identify() that a driver |
| | 188 | accepts the file, will validate the passed open option list with the |
| | 189 | authorized open option list. Below an example of such an authorized open option list |
| | 190 | in the S57 driver |
| | 191 | {{{ |
| | 192 | <OpenOptionList> |
| | 193 | <Option name="UPDATES" type="string-select" |
| | 194 | description="Should update files be incorporated into the base data on the fly" default="APPLY"> |
| | 195 | <Value>APPLY</Value> |
| | 196 | <Value>IGNORE</Value> |
| | 197 | </Option> |
| | 198 | <Option name="SPLIT_MULTIPOINT" type="boolean" |
| | 199 | description="Should multipoint soundings be split into many single point " |
| | 200 | "sounding features" default="NO" /> |
| | 201 | <Option name="ADD_SOUNDG_DEPTH" type="boolean" |
| | 202 | description="Should a DEPTH attribute be added on SOUNDG features and " |
| | 203 | "assign the depth of the sounding" default="NO" /> |
| | 204 | <Option name="RETURN_PRIMITIVES" type="boolean" |
| | 205 | description="Should all the low level geometry primitives be returned as " |
| | 206 | "special IsolatedNode, ConnectedNode, Edge and Face layers" default="NO" /> |
| | 207 | <Option name="PRESERVE_EMPTY_NUMBERS" type="boolean" |
| | 208 | description="If enabled, numeric attributes assigned an empty string as a " |
| | 209 | "value will be preserved as a special numeric value" default="NO" /> |
| | 210 | <Option name="LNAM_REFS" type="boolean" |
| | 211 | description="Should LNAM and LNAM_REFS fields be attached to features " |
| | 212 | "capturing the feature to feature relationships in the FFPT " |
| | 213 | "group of the S-57 file" default="YES" /> |
| | 214 | <Option name="RETURN_LINKAGES" type="boolean" |
| | 215 | description="Should additional attributes relating features to their underlying " |
| | 216 | "geometric primtives be attached" default="NO" /> |
| | 217 | <Option name="RECODE_BY_DSSI" type="boolean" |
| | 218 | description="Should attribute values be recoded to UTF-8 from the character " |
| | 219 | "encoding specified in the S57 DSSI record." default="NO" /> |
| | 220 | </OpenOptionList> |
| | 221 | }}} |
| | 222 | - GDAL_DS_LAYER_CREATIONOPTIONLIST at dataset level. But can also be set at |
| | 223 | driver level because, in practice, layer creation options do not depend on the |
| | 224 | dataset instance. |
| | 225 | The value of this item is an XML snippet with a format similar to dataset creation |
| | 226 | options. |
| | 227 | If specified, the passed creation options to CreateLayer() are validated |
| | 228 | against that authorized creation option list. |
| | 229 | Below an example of such an authorized open option list in the Shapefile driver. |
| | 230 | {{{ |
| | 231 | <LayerCreationOptionList> |
| | 232 | <Option name="SHPT" type="string-select" description="type of shape" default="automatically detected"> |
| | 233 | <Value>POINT</Value> |
| | 234 | <Value>ARC</Value> |
| | 235 | <Value>POLYGON</Value> |
| | 236 | <Value>MULTIPOINT</Value> |
| | 237 | <Value>POINTZ</Value> |
| | 238 | <Value>ARCZ</Value> |
| | 239 | <Value>POLYGONZ</Value> |
| | 240 | <Value>MULTIPOINTZ</Value> |
| | 241 | <Value>NONE</Value> |
| | 242 | <Value>NULL</Value> |
| | 243 | </Option> |
| | 244 | <Option name="2GB_LIMIT" type="boolean" description="Restrict .shp and .dbf to 2GB" default="NO" /> |
| | 245 | <Option name="ENCODING" type="string" description="DBF encoding" default="LDID/87" /> |
| | 246 | <Option name="RESIZE" type="boolean" description="To resize fields to their optimal size." default="NO" /> |
| | 247 | </LayerCreationOptionList> |
| | 248 | }}} |
| | 249 | |
| | 250 | === Datasets / Datasources === |
| | 251 | |
| | 252 | * The main methods from OGRDataSource have been moved to GDALDataset : |
| | 253 | {{{ |
| | 254 | virtual int GetLayerCount() { return 0; } |
| | 255 | virtual OGRLayer *GetLayer(int) { return NULL; } |
| | 256 | virtual OGRLayer *GetLayerByName(const char *); |
| | 257 | virtual OGRErr DeleteLayer(int); |
| | 258 | |
| | 259 | virtual int TestCapability( const char * ) { return FALSE; } |
| | 260 | |
| | 261 | virtual OGRLayer *CreateLayer( const char *pszName, |
| | 262 | OGRSpatialReference *poSpatialRef = NULL, |
| | 263 | OGRwkbGeometryType eGType = wkbUnknown, |
| | 264 | char ** papszOptions = NULL ); |
| | 265 | virtual OGRLayer *CopyLayer( OGRLayer *poSrcLayer, |
| | 266 | const char *pszNewName, |
| | 267 | char **papszOptions = NULL ); |
| | 268 | |
| | 269 | virtual OGRStyleTable *GetStyleTable(); |
| | 270 | virtual void SetStyleTableDirectly( OGRStyleTable *poStyleTable ); |
| | 271 | |
| | 272 | virtual void SetStyleTable(OGRStyleTable *poStyleTable); |
| | 273 | |
| | 274 | virtual OGRLayer * ExecuteSQL( const char *pszStatement, |
| | 275 | OGRGeometry *poSpatialFilter, |
| | 276 | const char *pszDialect ); |
| | 277 | virtual void ReleaseResultSet( OGRLayer * poResultsSet ); |
| | 278 | |
| | 279 | int GetRefCount() const; |
| | 280 | int GetSummaryRefCount() const; |
| | 281 | OGRErr Release(); |
| | 282 | }}} |
| | 283 | The following matching C API is available : |
| | 284 | {{{ |
| | 285 | int CPL_DLL GDALDatasetGetLayerCount( GDALDatasetH ); |
| | 286 | OGRLayerH CPL_DLL GDALDatasetGetLayer( GDALDatasetH, int ); |
| | 287 | OGRLayerH CPL_DLL GDALDatasetGetLayerByName( GDALDatasetH, const char * ); |
| | 288 | OGRErr CPL_DLL GDALDatasetDeleteLayer( GDALDatasetH, int ); |
| | 289 | OGRLayerH CPL_DLL GDALDatasetCreateLayer( GDALDatasetH, const char *, |
| | 290 | OGRSpatialReferenceH, OGRwkbGeometryType, |
| | 291 | char ** ); |
| | 292 | OGRLayerH CPL_DLL GDALDatasetCopyLayer( GDALDatasetH, OGRLayerH, const char *, |
| | 293 | char ** ); |
| | 294 | int CPL_DLL GDALDatasetTestCapability( GDALDatasetH, const char * ); |
| | 295 | OGRLayerH CPL_DLL GDALDatasetExecuteSQL( GDALDatasetH, const char *, |
| | 296 | OGRGeometryH, const char * ); |
| | 297 | void CPL_DLL GDALDatasetReleaseResultSet( GDALDatasetH, OGRLayerH ); |
| | 298 | OGRStyleTableH CPL_DLL GDALDatasetGetStyleTable( GDALDatasetH ); |
| | 299 | void CPL_DLL GDALDatasetSetStyleTableDirectly( GDALDatasetH, OGRStyleTableH ); |
| | 300 | void CPL_DLL GDALDatasetSetStyleTable( GDALDatasetH, OGRStyleTableH ); |
| | 301 | }}} |
| | 302 | OGRDataSource definition is now reduced to : |
| | 303 | {{{ |
| | 304 | class CPL_DLL OGRDataSource : public GDALDataset |
| | 305 | { |
| | 306 | public: |
| | 307 | OGRDataSource(); |
| | 308 | |
| | 309 | virtual const char *GetName() = 0; |
| | 310 | |
| | 311 | static void DestroyDataSource( OGRDataSource * ); |
| | 312 | }; |
| | 313 | }}} |
| | 314 | The existing OGR_DS_* API is preserved. The implementation of those functions |
| | 315 | casts the OGRDataSourceH opaque pointer to GDALDataset*, so it is possible to |
| | 316 | consider GDALDatasetH and OGRDataSourceH as equivalent from the C API point of |
| | 317 | view. Note that it is not true at the C++ level ! |
| | 318 | |
| | 319 | * OGRDataSource::SyncToDisk() has been removed. The equivalent functionnality |
| | 320 | should be implemented in existing FlushCache(). GDALDataset::FlushCache() nows |
| | 321 | does the job of the previous generic implementation of OGRDataSource::SyncToDisk(), |
| | 322 | i.e. iterate over all layers and call SyncToDisk() on them. |
| | 323 | |
| | 324 | * GDALDataset has now a protected ICreateLayer() method. |
| | 325 | {{{ |
| | 326 | virtual OGRLayer *ICreateLayer( const char *pszName, |
| | 327 | OGRSpatialReference *poSpatialRef = NULL, |
| | 328 | OGRwkbGeometryType eGType = wkbUnknown, |
| | 329 | char ** papszOptions = NULL ); |
| | 330 | |
| | 331 | }}} |
| | 332 | This method is what used to be CreateLayer(), i.e. that drivers should |
| | 333 | rename their specialized CreateLayer() implementations as ICreateLayer(). |
| | 334 | CreateLayer() is kept at GDALDataset level, but its implementation does a |
| | 335 | prior validation of passed creation options against an optional authorized |
| | 336 | creation option list (GDAL_DS_LAYER_CREATIONOPTIONLIST), before calling |
| | 337 | ICreateLayer() (this is similar to RasterIO() / IRasterIO() ) |
| | 338 | A global pass on all in-tree OGR drivers has been made to rename CreateLayer() |
| | 339 | as ICreateLayer(). |
| | 340 | |
| | 341 | * GDALOpenEx() is added to be able to open raster-only, vector-only, or |
| | 342 | raster-vector datasets. It accepts read-only/update mode, shared/non-shared mode. |
| | 343 | A list of potential candidate drivers can be passed. If NULL, all drivers are |
| | 344 | probed. |
| | 345 | A list of open options (NAME=VALUE syntax) can be passed. |
| | 346 | If the list of sibling files has already been established, it can also be passed. |
| | 347 | Otherwise GDALOpenInfo will establish it. |
| | 348 | {{{ |
| | 349 | GDALDatasetH CPL_STDCALL GDALOpenEx( const char* pszFilename, |
| | 350 | unsigned int nOpenFlags, |
| | 351 | const char* const* papszAllowedDrivers, |
| | 352 | const char* const* papszOpenOptions, |
| | 353 | const char* const* papszSiblingFiles ); |
| | 354 | }}} |
| | 355 | The nOpenFlags argument is a 'or-able' combination of the following values : |
| | 356 | {{{ |
| | 357 | /* Note: we define GDAL_OF_READONLY and GDAL_OF_UPDATE to be on purpose */ |
| | 358 | /* equals to GA_ReadOnly and GA_Update */ |
| | 359 | |
| | 360 | /** Open in read-only mode. */ |
| | 361 | #define GDAL_OF_READONLY 0x00 |
| | 362 | /** Open in update mode. */ |
| | 363 | #define GDAL_OF_UPDATE 0x01 |
| | 364 | |
| | 365 | /** Allow raster and vector drivers. */ |
| | 366 | #define GDAL_OF_ALL 0x00 |
| | 367 | |
| | 368 | /** Allow raster drivers. */ |
| | 369 | #define GDAL_OF_RASTER 0x02 |
| | 370 | /** Allow vector drivers. */ |
| | 371 | #define GDAL_OF_VECTOR 0x04 |
| | 372 | /* Some space for GDAL 3.0 new types ;-) */ |
| | 373 | /*#define GDAL_OF_OTHER_KIND1 0x08 */ |
| | 374 | /*#define GDAL_OF_OTHER_KIND2 0x10 */ |
| | 375 | |
| | 376 | /** Open in shared mode. */ |
| | 377 | #define GDAL_OF_SHARED 0x20 |
| | 378 | |
| | 379 | /** Emit error message in case of failed open. */ |
| | 380 | #define GDAL_OF_VERBOSE_ERROR 0x40 |
| | 381 | }}} |
| | 382 | The existing GDALOpen(), GDALOpenShared(), OGROpen(), OGROpenShared(), |
| | 383 | OGR_Dr_Open() are just wrappers of GDALOpenEx() with appropriate open flags. |
| | 384 | From the user point of view, their behaviour is identical to the existing one, |
| | 385 | i.e. GDALOpen() family will only returns datasets of drivers with declared raster |
| | 386 | capabilities, and similarly with OGROpen() family with vector. |
| | 387 | |
| | 388 | * GDALOpenInfo class. The following changes are done : |
| | 389 | - the second argument of the constructor is now nOpenFlags instead of |
| | 390 | GDALAccess, with same semantics as GDALOpenEx(). |
| | 391 | GDALOpenInfo uses the read-only/update bit to "compute" the eAccess flag |
| | 392 | that is heavily used in existing drivers. |
| | 393 | Drivers with both raster and vector capabilities can |
| | 394 | use the GDAL_OF_VECTOR/GDAL_OF_RASTER bits to determine the intent of the |
| | 395 | caller. For example if a caller opens with GDAL_OF_RASTER only and the |
| | 396 | dataset only contains vector data, the driver might decide to not open the |
| | 397 | dataset (if it is a read-only driver. If it is a driver with update |
| | 398 | capability, it should do that only if the opening is done in read-only mode). |
| | 399 | - the open options passed to GDALOpenEx() are stored into a papszOpenOptions |
| | 400 | member of GDALOpenInfo, so that drivers can use them. |
| | 401 | - the "FILE* fp" member is transformed into "VSILFILE* fpL". This change is |
| | 402 | motivated by the fact that most popular drivers now use the VSI Virtual File |
| | 403 | API, so they can now directly use the fpL member instead of re-opening again |
| | 404 | the file. A global pass on all in-tree GDAL drivers that used fp has been |
| | 405 | made. |
| | 406 | - A VSIStatExL() was done previously to determine the nature of the file passed. |
| | 407 | Now, we optimistically begin with a VSIFOpenL(), assuming that in most use |
| | 408 | cases the passed filename is a file. If the opening fails, VSIStatExL() is |
| | 409 | done to determine the nature of the filename. |
| | 410 | - If the requested access mode is update, the opening of the file with |
| | 411 | VSIFOpenL() is done with "rb+" permissions to be directly usable. |
| | 412 | - The papszSiblingFiles member is now private. It is accessed by a |
| | 413 | GetSiblingFiles() method that does the ReadDir() on demand. This can speed |
| | 414 | up the Identify() method that generally does not require to know sibling |
| | 415 | files. |
| | 416 | - A new method, TryToIngest(), is added to read more than the first 1024 |
| | 417 | bytes of a file. This is useful for a few vector drivers, like GML or NAS, |
| | 418 | that must fetch a bit more bytes to be able to identify the file. |
| | 419 | |
| | 420 | === Layer === |
| | 421 | |
| | 422 | * OGRLayer extends GDALMajorObject. Drivers can now define layer metadata items |
| | 423 | that can be retrieved with the usual GetMetadata()/GetMetadateItem() API. |
| | 424 | |
| | 425 | * The GetInfo() method has been removed. It has never been implemented in any |
| | 426 | in-tree drivers and has been deprecated for a long time. |
| | 427 | |
| | 428 | === Other === |
| | 429 | |
| | 430 | * The deprecated and unused GDALProjDefH and GDALOptionDefinition types have |
| | 431 | been removed from gdal.h |
| | 432 | |
| | 433 | * GDALGeneralCmdLineProcessor() now interprets the nOptions (combination of |
| | 434 | GDAL_OF_RASTER and GDAL_OF_RASTER) argument as the |
| | 435 | type of drivers that should be displayed with the --formats option. If set to 0, |
| | 436 | GDAL_OF_RASTER is assumed. |
| | 437 | |
| | 438 | * the --formats option of GDAL utilities outputs whether drivers have |
| | 439 | raster and/or vector capabilities |
| | 440 | |
| | 441 | * the --format option of GDAL utilities outputs GDAL_DMD_EXTENSIONS, |
| | 442 | GDAL_DMD_OPENOPTIONLIST, GDAL_DS_LAYER_CREATIONOPTIONLIST. |
| | 443 | |
| | 444 | * OGRGeneralCmdLineProcessor() use GDALGeneralCmdLineProcessor() implementation, |
| | 445 | restricting --formats to vector capable drivers. |
| | 446 | |
| | 447 | == Changes in drivers == |
| | 448 | |
| | 449 | * OGR PCIDSK driver has been merged into GDAL PCIDSK driver. |
| | 450 | |
| | 451 | * OGR PDF driver has been merged into GDAL PDF driver. |
| | 452 | |
| | 453 | * A global pass has been made to in-tree OGR drivers that have to open a file to |
| | 454 | determine if they recognize it. They have been converted to GDALDriver to |
| | 455 | accept a GDALOpenInfo argument and they now use its pabyHeader field to examine |
| | 456 | the first bytes of files. |
| | 457 | The number of system calls realated to file access (open/stat), in order to |
| | 458 | determine that a file is not recognized by any OGR driver, has now dropped from |
| | 459 | 46 in GDAL 1.11 to 1. |
| | 460 | The converted drivers are : AeronavFAA, ArcGEN, AVCBin, AVCE00, BNA, CSV, |
| | 461 | DGN, EDIGEO, ESRI Shapefile, GeoJSON, GeoRSS, GML, GPKG, GPSBabel, GPX, GTM, HTF, |
| | 462 | ILI1, ILI2, KML, LIBKML, MapInfo File, MySQL, NAS, NTF, OpenAIR, OSM, PDS, |
| | 463 | REC, S57, SDTS, SEGUKOOA, SEGY, SOSI, SQLite, SUA, SVG, TIGER, VFK, VRT, WFS |
| | 464 | |
| | 465 | * Long driver descrption is set to most OGR drivers. |
| | 466 | |
| | 467 | * All classes deriving from OGRLayer have been modified to call SetDescription() |
| | 468 | whith the value of GetName()/poFeatureDefn->GetName(). test_ogrsf tests that |
| | 469 | it is properly set. |
| | 470 | |
| | 471 | * Following drivers are kept as OGRSFDriver, but their Open() method does |
| | 472 | early extension/prefix testing to avoid datasource object to be instanciated : |
| | 473 | CartoDB, CouchDB, DXF, EDIGEO, GeoConcept, GFT, GME, IDRISI, OGDI, PCIDSK, |
| | 474 | PG, XPlane. |
| | 475 | |
| | 476 | * Identify() has been implemented for CSV, DGN, DXF, EDIGEO, GeoJSON, GML, KML, LIBKML, |
| | 477 | MapInfo File, NAS, OpenFileGDB, OSM, S57, Shape, SQLite, VFK, VRT. |
| | 478 | |
| | 479 | * GDAL_DMD_EXTENSION/GDAL_DMD_EXTENSIONS set for following drivers: AVCE00, |
| | 480 | BNA, CSV, DGN, DWG, DXF, EDIGEO, FileGDB, Geoconcept, GeoJSON, Geomedia, |
| | 481 | GML, GMT, GPKG, GPX, GPSTrackMaker, IDRISI Vector, Interlis 1, Interlis 2, |
| | 482 | KML, LIBKML, MDB, MapInfo File, NAS, ODS, OpenFileGDB, OSM, PGDump, PGeo, |
| | 483 | REC, S57, ESRI Shapefile, SQLite, SVG, WaSP, XLS, XLSX, XPlane. |
| | 484 | |
| | 485 | * Document dataset and layer creation options of BNA, DGN, FileGDB, GeoConccept, |
| | 486 | GeoJSON, GeoRSS, GML, GPKG, KML, LIBKML, PG, PGDump and ESRI Shapefile |
| | 487 | drivers as GDAL_DMD_CREATIONOPTIONLIST / GDAL_DS_LAYER_CREATIONOPTIONLIST. |
| | 488 | |
| | 489 | * Add open options AAIGRID, PDF, S57 and ESRI Shapefile drivers. |
| | 490 | |
| | 491 | * GetFileList() implemented in OpenFileGDB, Shapefile and OGR VRT drivers. |
| | 492 | |
| | 493 | * Rename datasource SyncToDisk() as FlushCache() for LIBKML, OCI, ODS, XLSX |
| | 494 | drivers. |
| | 495 | |
| | 496 | * Use poOpenInfo->fpL to avoid useless file re-opening in GTiff, PNG, JPEG, |
| | 497 | GIF, VRT, NITF, DTED. |
| | 498 | |
| | 499 | * HTTP driver: declared as GDAL_DCAP_RASTER and GDAL_DCAP_VECTOR driver. |
| | 500 | |
| | 501 | * RIK: implement Identify() |
| | 502 | |
| | 503 | * Note: the compilation and good working of the following OGR drivers (mostly |
| | 504 | proprietary) could not be tested: ArcObjects, DWG, DODS, SDE, FME, GRASS, IDB, |
| | 505 | OCI, MSSQLSpatial(compilation OK, but not runtime tested) |
| | 506 | |
| | 507 | == Changes in utilities == |
| | 508 | |
| | 509 | * gdalinfo accepts a -oo option to define open options |
| | 510 | * ogrinfo accepts a -oo option to define open options |
| | 511 | * ogr2ogr accepts a -oo option to define input dataset open options, and |
| | 512 | -doo to define destination dataset open options |
| | 513 | |
| | 514 | == Changes in SWIG bindings == |
| | 515 | |
| | 516 | * Python and Java bindings: |
| | 517 | - add new GDALDataset methods taken from OGRDataSource : CreateLayer(), |
| | 518 | CopyLayer(), DeleteLayer(), GetLayerCount(), GetLayerByIndex(), |
| | 519 | GetLayerByName(), TestCapability(), ExecuteSQL(), ReleaseResultSet(), |
| | 520 | GetStyleTable() and SetStyleTable(). |
| | 521 | - make OGR Driver, DataSource and Layer objects derive from MajorObject |
| | 522 | * Perl and CSharp: make sure that it still compiles but some work would have |
| | 523 | to be done by their mainteners to be able to use the new capabilities |
| | 524 | |
| | 525 | == Potential changes that are *NOT* included in this RFC == |
| | 526 | |
| | 527 | "Natural" evolutions of current RFC : |
| | 528 | * Unifying the GDAL MEM and OGR Memory drivers. |
| | 529 | * Unifying the GDAL VRT and OGR VRT drivers. |
| | 530 | |
| | 531 | Further unification steps : |
| | 532 | * Source tree changes to move OGR drivers from ogr/ogrsf_frmts/ to frmts/ , |
| | 533 | to move ogr/ogrsf_frmts/generic/* to gcore/*, etc... |
| | 534 | * Documentation unification (pages with list of drivers, etc...) |
| | 535 | * Renaming to remove traces of OGR namespace : OGRLayer -> GDALLayer, etc... |
| | 536 | * Kill --without-ogr compilation option ? It has been preserved in a working |
| | 537 | state even if it embeds now ogr/ogrsf_frmts/generic and ogr/ogrsf_frmts/mitab |
| | 538 | for conveniency. |
| | 539 | * Unification of some utilities : "gdal info XXX", "gdal convert XXX" that |
| | 540 | would work on all kind of datasets. |
| | 541 | |
| | 542 | == Backward compatibility == |
| | 543 | |
| | 544 | GDALDriverManager::GetDriverCount(), GetDriver() now returns OGR drivers. |
| | 545 | |
| | 546 | The reference counting in GDAL datasets and GDAL 1.X OGR datasources was a bit |
| | 547 | different. It starts at 1 for GDAL datasets, and started at 0 for OGR datasources. |
| | 548 | Now that OGRDataSource is basically a GDALDataset, it starts at 1 for both cases. |
| | 549 | Hopefully there are very few users of the OGR_DS_GetRefCount() API. If it was |
| | 550 | deemed necessary we could restore the previous behaviour at the C API, but that |
| | 551 | would not be possible at the C++ level. For reference, neither MapServer nor QGIS |
| | 552 | use OGR_DS_GetRefCount(). |
| | 553 | |
| | 554 | == Documentation == |
| | 555 | |
| | 556 | A pass should be made on the documentation to check that all new methods are |
| | 557 | properly documented. The OGR general documentation (especially C++ API Read/Write |
| | 558 | tutorial, Driver implementation tutorial and OGR architecture) should be updated |
| | 559 | to reflect the changes. |
| | 560 | |
| | 561 | == Testing == |
| | 562 | |
| | 563 | Very few changes have been made so that the existing autotest suite still |
| | 564 | passes. |
| | 565 | Additions have been made to test the GDALOpenEx() API and the methods "imported" |
| | 566 | from OGRDataSource into GDALDataset. |
| | 567 | |
| | 568 | == Version numbering == |
| | 569 | |
| | 570 | Although the above describes changes should have very few impact on existing |
| | 571 | applications of the C API, some behaviour changes, C++ level changes and the |
| | 572 | conceptual changes are thought to deserve a 2.0 version number. |
| | 573 | |
| | 574 | == Implementation == |
| | 575 | |
| | 576 | Implementation will be done by Even Rouault. |
| | 577 | |
| | 578 | The proposed implementation lies in the "unification" branch of the |
| | 579 | https://github.com/rouault/gdal2/tree/unification repository. |
| | 580 | |
| | 581 | The list of changes : https://github.com/rouault/gdal2/compare/unification |
| | 582 | |
| | 583 | == Voting history == |
| | 584 | |
| | 585 | TBD |
