FDO RFC 30 - Data Value Type Conversion

This page contains a request for comments document (RFC) for the FDO Open Source project. More FDO RFCs can be found on the RFCs page.

Status

RFC Template Version	(1.0)
Submission Date	November 21, 2008
Last Modified	Brent Robinson Timestamp
Author	Brent Robinson
RFC Status	Adopted
Implementation Status	Pending
Proposed Milestone	3.5
Assigned PSC guide(s)	Greg Boone
Voting History	April 10, 2009
+1	Greg, Traian, Jason, Jackie, Haris, Orest
+0
-0
-1

Overview

The purpose of this RFC is to enhance the FDO API to support converting FDO data values between various types.

Motivation

There are a number of places in core FDO and Provider source code where data values are converted from one type to another. These data values include property values for features and property constraint values. Also, as various FDO client applications are written, some of them may also need to do data type conversion.

One current example is when an SDF file has a feature class with an int64 identity property called ID, and an FdoISelect is performed with the following string filter:

ID = 5

When the filter is parsed, the value 5 becomes an int32 value. Since the filter is on the identity property, the SDF provider optimizes the select by doing an indexed lookup. However, for the lookup to work, the value must be converted to the property type (int32->int64). This conversion is done in ​SdfQueryOptimizer::ConvertDataValue()

Another example is the ​FdoIDescribeSchema implementation for the RDBMS providers ( in ​FdoSmLpSchemaCollection::FixDataValueType()), which extracts check constraints from SQL expressions obtained from the RDBMS, and converts them into FDO Property Value Constraints. These constraints contain data values that represent ranges or lists of allowed values. The data types determined by the expression parser do not always exactly match the data type for the property that owns the constraint. When the types differ, the data values are converted to the property's data type.

Data value type conversions are not always trivial and the supporting runtime functions sometimes differ between Win32 and Linux (e.g. converting between string and int64). Implementing these conversions in different places leads to the following problems:

• there is no consistent handling of type conversions. For example, different parts of FDO might handle overflows (value is not valid for the destination type) differently

• code duplication. More expensive maintenance if the same fix has to be applied in multiple places (this can easily happen when the type conversion code is cloned to different places)

• more difficult for developers to find the type conversion functions currently available, since these can be in various places

Data values are encapsulated in the ​FdoDataValue class and its derivations. There is a derivation for each data type supported by FDO. Example data types include various integer types (8bit, 16bit, 32bit, 64bit), string and datetime.

The ​FdoDataValue class effectively provides this type conversion now, by supporting the conversion from strings to other data types and vice-versa. For example, a double can be converted to an int32 by converting it to a string and then to an int32. However, providing a more direct conversion, without converting to and from strings, would provide better performance. It also allows very large or small values to be converted more accurately.

It is also possible to convert between various numeric types through simple casting. However, this leads to source code with sizeable switch statements when converting between two arbitrary data types.

Application and Provider developers also need a certain amount of control over how the conversion is performed, in regards to:

Truncation - how to handle values that are too large for the destination type (e.g. convert 100000 to int16). In most cases, the conversion should fail but there are some cases where truncation is ok. For example, if 100000 is the inclusive maximum for a range constraint, then converting it to max int16 (32767) would be ok.

Shifting - how to handle values that lose precision when converted (e.g. convert 1.9 to int32). In some cases, rounding the value is ok. However, if it is a value parsed from an FDO filter string (e.g "ID = 1.9"), then rounding it is not ok since that changes the filter to "ID = 2".

Error Handling - the simplest way to handle errors is to throw an exception. However, there are cases where the caller might just want to test whether the conversion can be done, without generating an error. In these cases, trapping and discarding the exception each time would be cumbersome.

Proposed Solution

This RFC proposes that data type conversions be encapsulated in new API functions on the ​FdoDataValue class.

Direct conversion from a source to a destination type will be supported by adding Create functions to each destination type, which take a value of the source type as a parameter.

The following describes the changes to the unmanaged API. However, equivalent changes will also be made to the managed API.

Main Functions

The following functions will be added:

FDO_API static FdoDataValue* FdoDataValue::Create(
     FdoDataType dataType, 
     FdoDataValue* src, 
     FdoBoolean nullIfIncompatible = false,
     FdoBoolean shift = true,
     FdoBoolean truncate = false
);
 
FDO_API static FdoBooleanValue* FdoBooleanValue::Create(
     FdoDataValue* src, 
     FdoBoolean nullIfIncompatible = false,
     FdoBoolean shift = true,
     FdoBoolean truncate = false
 );
 
FDO_API static FdoByteValue* FdoByteValue::Create(
     FdoDataValue* src, 
     FdoBoolean nullIfIncompatible = false,
     FdoBoolean shift = true,
     FdoBoolean truncate = false
 );
 
FDO_API static FdoDateTimeValue* FdoDateTimeValue::Create(
     FdoDataValue* src, 
     FdoBoolean nullIfIncompatible = false,
     FdoBoolean shift = true,
     FdoBoolean truncate = false
 );
 
FDO_API static FdoDecimalValue* FdoDecimalValue::Create(
     FdoDataValue* src, 
     FdoBoolean nullIfIncompatible = false,
     FdoBoolean shift = true,
     FdoBoolean truncate = false
 );
 
FDO_API static FdoDoubleValue* FdoDoubleValue::Create(
     FdoDataValue* src, 
     FdoBoolean nullIfIncompatible = false,
     FdoBoolean shift = true,
     FdoBoolean truncate = false
 );
 
FDO_API static FdoInt16Value* FdoInt16Value::Create(
     FdoDataValue* src, 
     FdoBoolean nullIfIncompatible = false,
     FdoBoolean shift = true,
     FdoBoolean truncate = false
 );
 
FDO_API static FdoInt32Value* FdoInt32Value::Create(
     FdoDataValue* src, 
     FdoBoolean nullIfIncompatible = false,
     FdoBoolean shift = true,
     FdoBoolean truncate = false
 );
 
FDO_API static FdoInt64Value* FdoInt64Value::Create(
     FdoDataValue* src, 
     FdoBoolean nullIfIncompatible = false,
     FdoBoolean shift = true,
     FdoBoolean truncate = false
 );
 
FDO_API static FdoSingleValue* FdoSingleValue::Create(
     FdoDataValue* src, 
     FdoBoolean nullIfIncompatible = false,
     FdoBoolean shift = true,
     FdoBoolean truncate = false
 );
FDO_API static FdoStringValue* FdoStringValue::Create(
     FdoDataValue* src, 
     FdoBoolean nullIfIncompatible = false,
     FdoBoolean shift = true,
     FdoBoolean truncate = false
 );
 
FDO_API static FdoCLOBValue* FdoCLOBValue::Create(
     FdoDataValue* src, 
     FdoBoolean nullIfIncompatible = false,
     FdoBoolean shift = true,
     FdoBoolean truncate = false
 );
 
FDO_API static FdoBLOBValue* FdoBLOBValue::Create(
     FdoDataValue* src, 
     FdoBoolean nullIfIncompatible = false,
     FdoBoolean shift = true,
     FdoBoolean truncate = false
 );

where:

dataType – specifies the destination type. This parameter is only specified for ​FdoDataValue::Create(). For the other functions, the destination type is determined by the return type.

src – specifies the source type and value. It is converted to an ​FdoDataValue of the destination type.

nullIfIncompatible – determines what happens if the source and destination types are incompatible (e.g. pass ​FdoDateTimeValue as src to ​FdoBooleanValue::Create()):

true: returns a null ​FdoDataValue (​FdoDataValue::IsNull() = true). Note that the pointer returned is not null, the ​FdoDataValue returned has its null flag set

false: throw an exception.

See the chart in ​FDO DataValue Conversion Compatibility for information on which pairs of types are compatible and which are incompatible.

shift – determines what happens if conversion would cause the value to change. This can happen when converting between numeric types and the destination type does not have enough precision to hold the value as is (e.g convert 1.9 from Double to Int32 ). Applicable only when both source and destination types are one of Byte, Decimal, Double, Int16, Int32, Int64 or Single:

true: shift the value by rounding it to the precision allowed by the destination type

false: depends on the value of nullIfIncompatible:

true: return a null ​FdoDataValue (​FdoDataValue::IsNull() = true). Note that the pointer returned is not null, the ​FdoDataValue returned has its null flag set

false: throw an exception

truncate – determines what happens if the value is outside the valid range for the destination type (e.g. convert 1000000 from ​FdoInt32Value to ​FdoInt16Value). Applicable only when both source and destination types are one of Boolean, Byte, Decimal, Double, Int16, Int32, Int64 or Single:

true: truncate the value to:

the maximum value for the destination type if the input value is greater than the maximum value

the minimum value for the destination type if the input value is less than the minimum value

false: depends on the value of nullIfIncompatible:

true: return a null ​FdoDataValue (​FdoDataValue::IsNull() = true). Note that the pointer returned is not null, the ​FdoDataValue returned has its null flag set

false: throw an exception

When the source type is numeric and the destination type is Boolean, the behaviour differs from the above and is as follows:

true: convert 0 to false and other values to true

false: convert 0 to false and 1 to true. Throw an exception for other values

Supporting Functions

The following functions will be added:

FDO_API FdoInt64 FdoStringP::ToInt64()

which converts a string to int64 value, and

FDO_API FdoStringP::FdoStringP( FdoInt64 value )

to convert an int64 to string. These functions hide the Windows and Linux specific differences in doing these conversions.

Implications

All of the proposed changes are additions to the FDO API so there are no backward compatibility issues. The new functions will need to be included in FDO API documentation.

Test Plan

A unit test suite will be added to test the API defined above.

Funding/Resources

To be determined