= RFC 59.1 : GDAL/OGR utilities as a library = Authors: Faza Mahamood, Even Rouault[[BR]] Contact: fazamhd at gmail dot com, even.rouault at spatialys.com[[BR]] Status: Adopted, implemented[[BR]] Implementation version: 2.1 == Summary == This RFC defines how to expose GDAL/OGR C/C++ utilities as C callable functions. The utility code is modified to call the new function. This RFC gives a general frame and principles, demonstrated with gdalinfo and gdal_translate, but aimed at being extended with other utilities. == Rationale == There is a need for calling GDAL utilities from code without involving system calls, to be able to work on in-memory datasets and use progress/cancellation callback functions. == Changes == A public header file gdal_utils.h is created which contains the public declarations of GDAL utilities. The current header(still in progress) can be found [https://github.com/rouault/gdal2/blob/rfc59.1/gdal/apps/gdal_utils.h here]. Each utility has a function (XXXXOptionsNew) to create an option structure from arguments specified as an array of strings. This function also accepts as argument an extra semi-private structure used to cooperate with the code of the command line utility itself. For GDALInfo(): {{{ /*! Options for GDALInfo(). Opaque type */ typedef struct GDALInfoOptions GDALInfoOptions; typedef struct GDALInfoOptionsForBinary GDALInfoOptionsForBinary; /** * Allocates a GDALInfoOptions struct. * * @param papszArgv NULL terminated list of options (potentially including filename and open options too) * The accepted options are the one of the gdalinfo utility. * @param psOptionsForBinary (output) may be NULL (and should generally be NULL), * otherwise (gdalinfo_bin.cpp use case) must be allocated with * GDALInfoOptionsForBinaryNew() prior to this function. Will be * filled with potentially present filename, open options, subdataset number... * @return pointer to the allocated GDALInfoOptions struct. * * @since GDAL 2.1 */ GDALInfoOptions CPL_DLL *GDALInfoOptionsNew(char** papszArgv, GDALInfoOptionsForBinary* psOptionsForBinary); void CPL_DLL GDALInfoOptionsFree( GDALInfoOptions *psOptions ); /** * Lists various information about a GDAL supported raster dataset. * * GDALInfoOptions* must be allocated and freed with GDALInfoOptionsNew() * and GDALInfoOptionsFree() respectively. * * @param hDataset the dataset handle. * @param psOptions the options structure returned by GDALInfoOptionsNew() or NULL. * @return string corresponding to the information about the raster dataset. * It must be freed using CPLFree(). * * @since GDAL 2.1 */ char CPL_DLL *GDALInfo( GDALDatasetH hDataset, const GDALInfoOptions *psOptions ); }}} Similarly for GDALTranslate(): {{{ /*! Options for GDALTranslate(). Opaque type */ typedef struct GDALTranslateOptions GDALTranslateOptions; typedef struct GDALTranslateOptionsForBinary GDALTranslateOptionsForBinary; GDALTranslateOptions CPL_DLL *GDALTranslateOptionsNew(char** papszArgv, GDALTranslateOptionsForBinary* psOptionsForBinary); void CPL_DLL GDALTranslateOptionsFree( GDALTranslateOptions *psOptions ); void CPL_DLL GDALTranslateOptionsSetProgress( GDALTranslateOptions *psOptions, GDALProgressFunc pfnProgress, void *pProgressData ); GDALDatasetH CPL_DLL GDALTranslate(const char *pszDestFilename, GDALDatasetH hSrcDataset, const GDALTranslateOptions *psOptions, int *pbUsageError); }}} Similarly for GDALWarp(): {{{ /*! Options for GDALWarp(). Opaque type */ typedef struct GDALWarpAppOptions GDALWarpAppOptions; typedef struct GDALWarpAppOptionsForBinary GDALWarpAppOptionsForBinary; GDALWarpAppOptions CPL_DLL *GDALWarpAppOptionsNew(char** papszArgv, GDALWarpAppOptionsForBinary* psOptionsForBinary); void CPL_DLL GDALWarpAppOptionsFree( GDALWarpAppOptions *psOptions ); void CPL_DLL GDALWarpAppOptionsSetProgress( GDALWarpAppOptions *psOptions, GDALProgressFunc pfnProgress, void *pProgressData ); void CPL_DLL GDALWarpAppOptionsSetWarpOption( GDALWarpAppOptions *psOptions, const char* pszKey, const char* pszValue ); GDALDatasetH CPL_DLL GDALWarp( const char *pszDest, GDALDatasetH hDstDS, int nSrcCount, GDALDatasetH *pahSrcDS, const GDALWarpAppOptions *psOptions, int *pbUsageError ); }}} == SWIG bindings (Python / Java / C# / Perl) changes == For all bindings, the above functions are mapped to SWIG with : {{{ struct GDALInfoOptions { %extend { GDALInfoOptions(char** options) { return GDALInfoOptionsNew(options, NULL); } ~GDALInfoOptions() { GDALInfoOptionsFree( self ); } } }; %rename (InfoInternal) GDALInfo; char *GDALInfo( GDALDatasetShadow *hDataset, GDALInfoOptions *infoOptions ); }}} {{{ struct GDALTranslateOptions { %extend { GDALTranslateOptions(char** options) { return GDALTranslateOptionsNew(options, NULL); } ~GDALTranslateOptions() { GDALTranslateOptionsFree( self ); } } }; %rename (TranslateInternal) wrapper_GDALTranslate; %newobject wrapper_GDALTranslate; %inline %{ GDALDatasetShadow* wrapper_GDALTranslate( const char* dest, GDALDatasetShadow* dataset, GDALTranslateOptions* translateOptions, GDALProgressFunc callback=NULL, void* callback_data=NULL) { int usageError; /* ignored */ GDALTranslateOptionsSetProgress(translateOptions, callback, callback_data); return GDALTranslate(dest, dataset, translateOptions, &usageError); } }}} {{{ struct GDALWarpAppOptions { %extend { GDALWarpAppOptions(char** options) { return GDALWarpAppOptionsNew(options, NULL); } ~GDALWarpAppOptions() { GDALWarpAppOptionsFree( self ); } } }; /* Note: we must use 2 distinct names since there's a bug/feature in swig */ /* that doesn't play nicely with the (int object_list_count, GDALDatasetShadow** poObjects) input typemap */ %inline %{ int wrapper_GDALWarpDestDS( GDALDatasetShadow* dstDS, int object_list_count, GDALDatasetShadow** poObjects, GDALWarpAppOptions* warpAppOptions, GDALProgressFunc callback=NULL, void* callback_data=NULL) { int usageError; /* ignored */ GDALWarpAppOptionsSetProgress(warpAppOptions, callback, callback_data); return GDALWarp(NULL, dstDS, object_list_count, poObjects, warpAppOptions, &usageError) != NULL; } %} %newobject wrapper_GDALWarpDestName; %inline %{ GDALDatasetShadow* wrapper_GDALWarpDestName( const char* dest, int object_list_count, GDALDatasetShadow** poObjects, GDALWarpAppOptions* warpAppOptions, GDALProgressFunc callback=NULL, void* callback_data=NULL) { int usageError; /* ignored */ GDALWarpAppOptionsSetProgress(warpAppOptions, callback, callback_data); return GDALWarp(dest, NULL, object_list_count, poObjects, warpAppOptions, &usageError); } %} }}} For Python bindings, convenience wrappers are created to allow specifying options in a more user friendly way. {{{ def InfoOptions(options = [], format = 'text', deserialize = True, computeMinMax = False, reportHistograms = False, reportProj4 = False, stats = False, approxStats = False, computeChecksum = False, showGCPs = True, showMetadata = True, showRAT = True, showColorTable = True, listMDD = False, showFileList = True, allMetadata = False, extraMDDomains = None): """ Create a InfoOptions() object that can be passed to gdal.Info() options can be be an array of strings, a string or let empty and filled from other keywords.""" def Info(ds, **kwargs): """ Return information on a dataset. Arguments are : ds --- a Dataset object or a filename Keyword arguments are : options --- return of gdal.InfoOptions(), string or array of strings other keywords arguments of gdal.InfoOptions() If options is provided as a gdal.InfoOptions() object, other keywords are ignored. """ }}} {{{ def TranslateOptions(options = [], format = 'GTiff', outputType = GDT_Unknown, bandList = None, maskBand = None, width = 0, height = 0, widthPct = 0.0, heightPct = 0.0, xRes = 0.0, yRes = 0.0, creationOptions = None, srcWin = None, projWin = None, projWinSRS = None, strict = False, unscale = False, scaleParams = None, exponents = None, outputBounds = None, metadataOptions = None, outputSRS = None, GCPs = None, noData = None, rgbExpand = None, stats = False, rat = True, resampleAlg = None, callback = None, callback_data = None): """ Create a TranslateOptions() object that can be passed to gdal.Translate() Keyword arguments are : options --- can be be an array of strings, a string or let empty and filled from other keywords. format --- output format ("GTiff", etc...) outputType --- output type (gdal.GDT_Byte, etc...) bandList --- array of band numbers (index start at 1) maskBand --- mask band to generate or not ("none", "auto", "mask", 1, ...) width --- width of the output raster in pixel height --- height of the output raster in pixel widthPct --- width of the output raster in percentage (100 = original width) heightPct --- height of the output raster in percentage (100 = original height) xRes --- output horizontal resolution yRes --- output vertical resolution creationOptions --- list of creation options srcWin --- subwindow in pixels to extract: [left_x, top_y, width, height] projWin --- subwindow in projected coordinates to extract: [ulx, uly, lrx, lry] projWinSRS --- SRS in which projWin is expressed strict --- strict mode unscale --- unscale values with scale and offset metadata scaleParams --- list of scale parameters, each of the form [src_min,src_max] or [src_min,src_max,dst_min,dst_max] exponents --- list of exponentation parameters outputBounds --- assigned output bounds: [ulx, uly, lrx, lry] metadataOptions --- list of metadata options outputSRS --- assigned output SRS GCPs --- list of GCPs noData --- nodata value (or "none" to unset it) rgbExpand --- Color palette expansion mode: "gray", "rgb", "rgba" stats --- whether to calcule statistics rat --- whether to write source RAT resampleAlg --- resampling mode callback --- callback method callback_data --- user data for callback """ def Translate(destName, srcDS, **kwargs): """ Convert a dataset. Arguments are : destName --- Output dataset name srcDS --- a Dataset object or a filename Keyword arguments are : options --- return of gdal.InfoOptions(), string or array of strings other keywords arguments of gdal.TranslateOptions() If options is provided as a gdal.TranslateOptions() object, other keywords are ignored. """ }}} {{{ def WarpOptions(options = [], format = 'GTiff', outputBounds = None, outputBoundsSRS = None, xRes = None, yRes = None, targetAlignedPixels = False, width = 0, height = 0, srcSRS = None, dstSRS = None, srcAlpha = False, dstAlpha = False, warpOptions = None, errorThreshold = None, warpMemoryLimit = None, creationOptions = None, outputType = GDT_Unknown, workingType = GDT_Unknown, resampleAlg = None, srcNodata = None, dstNodata = None, multithread = False, tps = False, rpc = False, geoloc = False, polynomialOrder = None, transformerOptions = None, cutlineDSName = None, cutlineLayer = None, cutlineWhere = None, cutlineSQL = None, cutlineBlend = None, cropToCutline = False, copyMetadata = True, metadataConflictValue = None, setColorInterpretation = False, callback = None, callback_data = None): """ Create a WarpOptions() object that can be passed to gdal.Warp() Keyword arguments are : options --- can be be an array of strings, a string or let empty and filled from other keywords. format --- output format ("GTiff", etc...) outputBounds --- output bounds as (minX, minY, maxX, maxY) in target SRS outputBoundsSRS --- SRS in which output bounds are expressed, in the case they are not expressed in dstSRS xRes, yRes --- output resolution in target SRS targetAlignedPixels --- whether to force output bounds to be multiple of output resolution width --- width of the output raster in pixel height --- height of the output raster in pixel srcSRS --- source SRS dstSRS --- output SRS srcAlpha --- whether to force the last band of the input dataset to be considered as an alpha band dstAlpha --- whether to force the creation of an output alpha band outputType --- output type (gdal.GDT_Byte, etc...) workingType --- working type (gdal.GDT_Byte, etc...) warpOptions --- list of warping options errorThreshold --- error threshold for approximation transformer (in pixels) warpMemoryLimit --- size of working buffer in bytes resampleAlg --- resampling mode creationOptions --- list of creation options srcNodata --- source nodata value(s) dstNodata --- output nodata value(s) multithread --- whether to multithread computation and I/O operations tps --- whether to use Thin Plate Spline GCP transformer rpc --- whether to use RPC transformer geoloc --- whether to use GeoLocation array transformer polynomialOrder --- order of polynomial GCP interpolation transformerOptions --- list of transformer options cutlineDSName --- cutline dataset name cutlineLayer --- cutline layer name cutlineWhere --- cutline WHERE clause cutlineSQL --- cutline SQL statement cutlineBlend --- cutline blend distance in pixels cropToCutline --- whether to use cutline extent for output bounds copyMetadata --- whether to copy source metadata metadataConflictValue --- metadata data conflict value setColorInterpretation --- whether to force color interpretation of input bands to output bands callback --- callback method callback_data --- user data for callback """ def Warp(destNameOrDestDS, srcDSOrSrcDSTab, **kwargs): """ Warp one or several datasets. Arguments are : destNameOrDestDS --- Output dataset name or object srcDSOrSrcDSTab --- an array of Dataset objects or filenames, or a Dataset object or a filename Keyword arguments are : options --- return of gdal.InfoOptions(), string or array of strings other keywords arguments of gdal.WarpOptions() If options is provided as a gdal.WarpOptions() object, other keywords are ignored. """ }}} gdal.Info() can be used either with setting the attributes of gdal.InfoOptions() or inlined arguments of gdal.Info(). Arguments can be specified as array of strings, command line syntax or dedeicated keywords. So various combinations are possible : {{{ options = gdal.InfoOptions(format = 'json', computeChecksum = True) gdal.Info(ds, options) }}} {{{ options = gdal.InfoOptions(options = ['-json', '-checksum']) gdal.Info(ds, options) }}} {{{ options = gdal.InfoOptions(options = '-json -checksum') gdal.Info(ds, options) }}} {{{ gdal.Info(ds, format = 'json', computeChecksum = True) }}} {{{ gdal.Info(ds, options = ['-json', '-checksum']) }}} {{{ gdal.Info(ds, options = '-json -checksum') }}} Same for gdal.Translate() == Utilities == Utilities are modified to call the respective function. == Documentation == All new methods/functions are documented. == Test Suite == gdal.Info method is tested in test_gdalinfo_lib.py. gdal.Translate method is tested in test_gdal_translate_lib.py. == Compatibility Issues == None expected. Command line utilities will keep the same interface. It will be checked by ensuring their tests in autotest/utilities still pass. == Open question == What name should be given for librarified ogr2ogr? OGR2OGR() or OGRTranslate() ? The order of arguments in GDALTranslate(), GDALWarp() and OGR2OGR() is currently dest then source(s). {{{ GDALDatasetH CPL_DLL GDALTranslate(const char *pszDest, GDALDatasetH hSrcDataset, GDALTranslateOptions *psOptions, int *pbUsageError) GDALDatasetH CPL_DLL GDALWarp( const char *pszDest, GDALDatasetH hDstDS, int nSrcCount, GDALDatasetH *pahSrcDS, GDALWarpAppOptions *psOptions, int *pbUsageError ) GDALDatasetH CPL_DLL OGR2OGR( const char *pszDest, GDALDatasetH hDstDS, GDALDatasetH hSrcDS, OGR2OGROptions *psOptions, int *pbUsageError ) }}} It is similar to GDALCreateCopy(const char* pszDestFilename, GDALDatasetH hSrcDS, ....), so at least there's a form of consistency at the API level. Any comments? == Related ticket == == Implementation == Implementation will be done by Faza Mahamood and Even Rouault The proposed implementation lies in the "rfc59.1" branch of the https://github.com/rouault/gdal2/tree/rfc59.1. The list of changes : https://github.com/rouault/gdal2/compare/rfc59.1 == Voting history +1 from DanielM and EvenR