Add IBM DB2 raster support to OGR

[dadler]

We have had a customer requirement for raster support in DB2, along with the vector support which was just integrated.

My intent is to add this support based on the GPKG implementation which Even had suggested looking at.

The table structure will generally follow the GeoPackage specification except that table names like gpkg_contents will be represented as gpkg.contents.

Unfortunately, there is a lot of duplication of code between GPKG and DB2 - it would be nice if this could be refactored as much of the code is quite complex.

Added functionality to the DB2 driver in ogr/ogrsf_frmts/db2.

*Added gdaldb2rasterband.cpp which has most of raster functionality.

*Added ogrdb2datasourcemd.cpp which has metadata handling

*Added ogrdb2cli.cpp which has DB2 CLI functionality

*Added drv_db2_raster.html

*Modified most source for cleanup - line lengths exceed 79 in code copied from ogrgeopackagerasterband.cpp

*Added autotest/gdrivers/db2.py for minimal raster test

Code committed with revision 32763. Test committed with revision 32764.

[Even Rouault]

Nice idea to based raster support on GPKG

To be sure to understand: do you intend moving the common code of the GPKG and DB2 code somewhere ? I'm not too sure where that could lie. Probably a bit too specific to be put in gcore/. I'd suggest perhaps isolating common stuff of the GPKG driver in separate files/classes in ogr/ogrsf_frmts/gpkg and reusing that from the DB2 driver. One drawback is that ogr/ogrsf_frmts/gpkg is only compiled if SQLite is avaialble. So perhaps have a ogr/ogrsf_frmts/gpkg_common dir that doesn't depend on sqlite stuff ?

[dadler]

Your suggestion about ogr/ogrsf_frmts/gpkg_common is along the lines of what I was thinking. It will take considerable investigation before I can propose something concrete.

Is the GPKG raster code modeled after logic that is in some of the gdal/frmts drivers? The details of the logic are pretty much incomprehensible to me, however, I am able to convert the sqlite-specific logic to DB2-specific logic with some confidence.

I am currently subclassing GDALPamDataset for OGRDB2DataSource - maybe there needs to be another common class in between which could also be used for OGRSQLiteBaseDataSource.

None of this will happen very soon. I need to finish testing the DB2 raster support and cleaning up the source copied from MS SQL to match current coding conventions. Then I need to build/test on Linux and add support for prepared statements as some operations are abysmally slow.

Replying to rouault:

Nice idea to based raster support on GPKG

To be sure to understand: do you intend moving the common code of the GPKG and DB2 code somewhere ? I'm not too sure where that could lie. Probably a bit too specific to be put in gcore/. I'd suggest perhaps isolating common stuff of the GPKG driver in separate files/classes in ogr/ogrsf_frmts/gpkg and reusing that from the DB2 driver. One drawback is that ogr/ogrsf_frmts/gpkg is only compiled if SQLite is avaialble. So perhaps have a ogr/ogrsf_frmts/gpkg_common dir that doesn't depend on sqlite stuff ?

[Even Rouault]

Is the GPKG raster code modeled after logic that is in some of the gdal/frmts drivers? The details of the logic are pretty much incomprehensible to me,

No, basically one of the tricky aspect of GPKG raster is that, despite they are tiled, the area of interest that will define the GDAL raster, isn't necessarily aligned on the tiles. So there might be some shift between the GPKG tile origin and the corresponding GDAL block. For read support, this isn't *that* hard to handle. For write support, it is much trickier as you don't want to do partial block compression/decompression when using lossy compressions.

[Even Rouault]

pyflakes issues break the build :

autotest/gdrivers/db2.py:42: 'osr' imported but unused
autotest/gdrivers/db2.py:192: local variable 'clamped_expected_cs' is assigned to but never used

[dadler]

Replying to rouault:

pyflakes issues break the build :

autotest/gdrivers/db2.py:42: 'osr' imported but unused
autotest/gdrivers/db2.py:192: local variable 'clamped_expected_cs' is assigned to but never used

Corrected errors with revision 32766. Is there a way I can test for errors like this? Is this a Python 3 issue?

[Even Rouault]

You can install pyflakes with "pip install pyflakes" and then run it on any python script with "pyflakes /path/to/the.py" or even on a whole directory. No it is not a Python 3 error. Just some basic checks of Python use.

[Even Rouault]

MSVC 64bit throws following errors : ​https://ci.appveyor.com/project/rouault/gdal-coverage/build/1.0.2289/job/1wdw7xx2ht78nxuo

gdaldb2rasterband.cpp
c:\projects\gdal-coverage\gdal\ogr\ogrsf_frmts\db2\gdaldb2rasterband.cpp(151) : error C2664: 'SQLRETURN SQLGetData(SQLHSTMT,SQLUSMALLINT,SQLSMALLINT,SQLPOINTER,SQLLEN,SQLLEN *)' : cannot convert argument 6 from 'SQLINTEGER *' to 'SQLLEN *' 
Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast 
c:\projects\gdal-coverage\gdal\ogr\ogrsf_frmts\db2\gdaldb2rasterband.cpp(615) : error C2664: 'SQLRETURN SQLGetData(SQLHSTMT,SQLUSMALLINT,SQLSMALLINT,SQLPOINTER,SQLLEN,SQLLEN *)' : cannot convert argument 6 from 'SQLINTEGER *' to 'SQLLEN *' 
Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast 
c:\projects\gdal-coverage\gdal\ogr\ogrsf_frmts\db2\gdaldb2rasterband.cpp(696) : error C2664: 'SQLRETURN SQLGetData(SQLHSTMT,SQLUSMALLINT,SQLSMALLINT,SQLPOINTER,SQLLEN,SQLLEN *)' : cannot convert argument 6 from 'SQLINTEGER *' to 'SQLLEN *' 
Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast 
c:\projects\gdal-coverage\gdal\ogr\ogrsf_frmts\db2\gdaldb2rasterband.cpp(1395) : error C2664: 'SQLRETURN SQLBindParameter(SQLHSTMT,SQLUSMALLINT,SQLSMALLINT,SQLSMALLINT,SQLSMALLINT,SQLULEN,SQLSMALLINT,SQLPOINTER,SQLLEN,SQLLEN *)' : cannot convert argument 10 from 'SQLINTEGER *' to 'SQLLEN *' 
Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast 
ogrdb2datasourcemd.cpp
ogrdb2driver.cpp
ogrdb2datasource.cpp
ogrdb2layer.cpp
ogrdb2tablelayer.cpp
c:\projects\gdal-coverage\gdal\ogr\ogrsf_frmts\db2\ogrdb2tablelayer.cpp(927) : error C2220: warning treated as error - no 'object' file generated 
c:\projects\gdal-coverage\gdal\ogr\ogrsf_frmts\db2\ogrdb2tablelayer.cpp(927) : warning C4267: 'initializing' : conversion from 'size_t' to 'int', possible loss of data 
c:\projects\gdal-coverage\gdal\ogr\ogrsf_frmts\db2\ogrdb2tablelayer.cpp(1082) : warning C4267: 'initializing' : conversion from 'size_t' to 'int', possible loss of data 
c:\projects\gdal-coverage\gdal\ogr\ogrsf_frmts\db2\ogrdb2tablelayer.cpp(1182) : warning C4267: 'initializing' : conversion from 'size_t' to 'int', possible loss of data 
c:\projects\gdal-coverage\gdal\ogr\ogrsf_frmts\db2\ogrdb2tablelayer.cpp(1323) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data

[dadler]

Corrected these errors.

Replying to rouault:

MSVC 64bit throws following errors : ​https://ci.appveyor.com/project/rouault/gdal-coverage/build/1.0.2289/job/1wdw7xx2ht78nxuo

gdaldb2rasterband.cpp
c:\projects\gdal-coverage\gdal\ogr\ogrsf_frmts\db2\gdaldb2rasterband.cpp(151) : error C2664: 'SQLRETURN SQLGetData(SQLHSTMT,SQLUSMALLINT,SQLSMALLINT,SQLPOINTER,SQLLEN,SQLLEN *)' : cannot convert argument 6 from 'SQLINTEGER *' to 'SQLLEN *' 
Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast 
c:\projects\gdal-coverage\gdal\ogr\ogrsf_frmts\db2\gdaldb2rasterband.cpp(615) : error C2664: 'SQLRETURN SQLGetData(SQLHSTMT,SQLUSMALLINT,SQLSMALLINT,SQLPOINTER,SQLLEN,SQLLEN *)' : cannot convert argument 6 from 'SQLINTEGER *' to 'SQLLEN *' 
Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast 
c:\projects\gdal-coverage\gdal\ogr\ogrsf_frmts\db2\gdaldb2rasterband.cpp(696) : error C2664: 'SQLRETURN SQLGetData(SQLHSTMT,SQLUSMALLINT,SQLSMALLINT,SQLPOINTER,SQLLEN,SQLLEN *)' : cannot convert argument 6 from 'SQLINTEGER *' to 'SQLLEN *' 
Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast 
c:\projects\gdal-coverage\gdal\ogr\ogrsf_frmts\db2\gdaldb2rasterband.cpp(1395) : error C2664: 'SQLRETURN SQLBindParameter(SQLHSTMT,SQLUSMALLINT,SQLSMALLINT,SQLSMALLINT,SQLSMALLINT,SQLULEN,SQLSMALLINT,SQLPOINTER,SQLLEN,SQLLEN *)' : cannot convert argument 10 from 'SQLINTEGER *' to 'SQLLEN *' 
Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast 
ogrdb2datasourcemd.cpp
ogrdb2driver.cpp
ogrdb2datasource.cpp
ogrdb2layer.cpp
ogrdb2tablelayer.cpp
c:\projects\gdal-coverage\gdal\ogr\ogrsf_frmts\db2\ogrdb2tablelayer.cpp(927) : error C2220: warning treated as error - no 'object' file generated 
c:\projects\gdal-coverage\gdal\ogr\ogrsf_frmts\db2\ogrdb2tablelayer.cpp(927) : warning C4267: 'initializing' : conversion from 'size_t' to 'int', possible loss of data 
c:\projects\gdal-coverage\gdal\ogr\ogrsf_frmts\db2\ogrdb2tablelayer.cpp(1082) : warning C4267: 'initializing' : conversion from 'size_t' to 'int', possible loss of data 
c:\projects\gdal-coverage\gdal\ogr\ogrsf_frmts\db2\ogrdb2tablelayer.cpp(1182) : warning C4267: 'initializing' : conversion from 'size_t' to 'int', possible loss of data 
c:\projects\gdal-coverage\gdal\ogr\ogrsf_frmts\db2\ogrdb2tablelayer.cpp(1323) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data

[Even Rouault]

I think this ticket can be closed. New one more focused to be opened if needed