XML of the response to a GetObservation request sent to SOS 1.0.0 server doesn't validate against the schema

[nsavard]

A SOS 1.0.0 server that received a GetObservation request returns an XML response that doesn't validate against the related schemas. You can find below the faulty elements. I checked with XMLSpy and the schemas, I got the same results.

The relevant tests are: getObservation:core-SOS.GetObservation-ResponseMatchingSRSData.1 (s0003) getObservation:core-SOS.GetObservation-ResponseMatchingResponseFormatData.1 (s0003) getObservation:core-SOS.GetObservation-ResponseMatchingProcedureData.1 (s0003) getObservation:core-SOS.GetObservation-ResponseMatchingFeatureOfInterestData.1 (s0003) getObservation:core-SOS.GetObservation-ResponseMatchingResultData.1 (s0003)

cvc-complex-type.2.4.a: Invalid content was found starting with element 'observedProperty'. One of '{"​http://www.opengis.net/om/1.0":resultQuality, "​http://www.opengis.net/om/1.0":observedProperty}' is expected.

Validation error:

cvc-complex-type.3.2.2: Attribute 'gml:id' is not allowed to appear in element 'gml:featureMember'.

Validation error:

cvc-complex-type.2.4.a: Invalid content was found starting with element 'obs'. One of '{"​http://www.opengis.net/gml":_Feature}' is expected.

Validation error:

cvc-complex-type.2.4.a: Invalid content was found starting with element 'observedProperty'. One of '{"​http://www.opengis.net/om/1.0":resultQuality, "​http://www.opengis.net/om/1.0":observedProperty}' is expected.

Validation error:

cvc-complex-type.3.2.2: Attribute 'gml:id' is not allowed to appear in element 'gml:featureMember'.

Validation error:

cvc-complex-type.2.4.a: Invalid content was found starting with element 'obs'. One of '{"​http://www.opengis.net/gml":_Feature}' is expected.

Validation error:

cvc-complex-type.2.4.a: Invalid content was found starting with element 'observedProperty'. One of '{"​http://www.opengis.net/om/1.0":resultQuality, "​http://www.opengis.net/om/1.0":observedProperty}' is expected.

Validation error:

cvc-complex-type.3.2.2: Attribute 'gml:id' is not allowed to appear in element 'gml:featureMember'.

Validation error:

cvc-complex-type.2.4.a: Invalid content was found starting with element 'obs'. One of '{"​http://www.opengis.net/gml":_Feature}' is expected.

Validation error:

cvc-complex-type.2.4.a: Invalid content was found starting with element 'observedProperty'. One of '{"​http://www.opengis.net/om/1.0":resultQuality, "​http://www.opengis.net/om/1.0":observedProperty}' is expected.

Validation error:

cvc-complex-type.3.2.2: Attribute 'gml:id' is not allowed to appear in element 'gml:featureMember'.

Validation error:

cvc-complex-type.2.4.a: Invalid content was found starting with element 'obs'. One of '{"​http://www.opengis.net/gml":_Feature}' is expected.

Validation error:

cvc-complex-type.2.4.a: Invalid content was found starting with element 'observedProperty'. One of '{"​http://www.opengis.net/om/1.0":resultQuality, "​http://www.opengis.net/om/1.0":observedProperty}' is expected.

Validation error:

cvc-complex-type.3.2.2: Attribute 'gml:id' is not allowed to appear in element 'gml:featureMember'.

Validation error:

cvc-complex-type.2.4.a: Invalid content was found starting with element 'obs'. One of '{"​http://www.opengis.net/gml":_Feature}' is expected.

15 validation errors detected.

[nsavard]

Just as an help, here are some comments based on the XML below:

1) NameSpace is missing in front of "observedProperty" and "obs" 2) gml:id attribute is not permitted as part of the "featureMember" element

<om:procedure xlink:href="urn:ogc:def:procedure:35"/> <observedProperty>WaterQuality</observedProperty> <om:result>

<gml:featureMember xmlns:gml="http://www.opengis.net/gml" gml:id="371">

<obs>

<gml:boundedBy>

<gml:Envelope srsName="urn:ogc:crs:epsg:4326" srsDimension="2">

<gml:lowerCorner>-75 45</gml:lowerCorner> <gml:upperCorner>-75 45</gml:upperCorner>

</gml:Envelope>

</gml:boundedBy> <msGeometry>

<gml:Point srsName="urn:ogc:crs:epsg:4326" srsDimension="2">

<gml:pos>-75 45</gml:pos>

</gml:Point>

</msGeometry> <Value>89.9</Value>

</obs>

</gml:featureMember>

</om:result>

</om:Observation>

</om:member>

[tomkralidis]

The major issue has to do with what we put into om:result.

OM 1.0.0 allows for om:result to be xs:any. So we put a gml:featureMember.

Because OM 1.0.0 refs GML 3.1.1, the identification of gml:featureMember in om:result triggers XML validators (I'm using XMLSpy 2008 sp1) to further validate that as gml. The GML produced is not valid.

If we wanted the gml:featureMember to be valid with om:result, we would have to:

• create an application schema to extend gml:featureMember with our attributes
• have said application ref'd in the xsi:schemaLocation attribute

I think this produces alot of overhead. People can hit WFS if they wanted a 'pure' GML output. Also, as I don't see us supporting DescribeObservationType, so I think it's too much effort for the value we would get out of embedding GML proper.

I propose the following:

for om:result, output a flat representation as per the attributes:

<om:result>
 <myns:value1>foo</myns:value1>
 <myns:value2>foo2</myns:value2>
</om:result>

As for the coordinates, we could throw these in om:Observation/om:featureOfInterest.

myns would be automagically generated

Because we don't have a schema defined for the attributes, XML validators continue to validate, so we're ok.

Thoughts?

[assefa]

Tom,

I understand you suggestion but I am a bit hesitant on it.

The advantage that I think we have when generating the feature members is that It can be read by tools such OGR, making it easier to develop client tools on top the resulting XML. Do you think this is worthed?

Is DescribeObservationType similar to DescribeFeatureType?

We already have the ability to generate a valid gml3.1 and extend the definition through the DescribeFeatureType in wfs. I am wondering if we can take advantage of it.

[tomkralidis]

Assefa: _very_ good point. Thanks for bringing this up.

SOS DescribeObservationType is the same in principle as WFS DescribeFeatureType.

• implement msSOSDescribeObservationType, leveraging what we can from msWFSDescribeFeatureType
• alter msSOSGetObservation so that it validates against a DescribeObservationType response

What do you think? If we agree, then I will set the milestone for 5.4, as this would require some level of effort.

[assefa]

works with me.

[tomkralidis]

Setting to 5.4. Note that once #2662 is implemented, msSOSGetObservation can ref accordingly.

[tomkralidis]

Made some fixes in #8511

[tomkralidis]

make that r8511

[tomkralidis]

r8514 implements a first pass at fully validating the document, by passing WFS DescribeFeatureType for the given observedProperty.

I don't have a validating parser handy, but if this works as expected, we could probably leverage this approach to support DescribeObservationType and DescribeFeatureOfInterestType.

msautotest test updated in r8515.

[tomkralidis]

Almost there for resultModel=om:Measurement (the default). Here's the issue:

When pointing to a WFS 1.1.0 DescribeFeatureType against the given typename, the response defines _all_ columns defined in the underlying dataset.

In our SOS implementation, only columns which have a sos_column_alias defined are output in the GetObservation response.

So, if some fields are not defined this way, the document is not valid, as the validator expects to see those fields in the XML response.

Of course, if a mapfile defines all columns with sos_column_alias, the document validates perfectly.

I hope I explained this clearly enough.

Assefa: any ideas on dealing with this? This would save us a bunch of effort instead of writing our own schema output to validate GetObservation requests.

I'll work on resultModel=om:Observation after this.

[assefa]

Tom,

I guess setting the gml_include_items to those items used by SOS would work too?

Are you thinking that we should use the gml_include_items/exclude ... in the SOS logic?

[tomkralidis]

Replying to assefa:

Tom,

I guess setting the gml_include_items to those items used by SOS would work too?

Are you thinking that we should use the gml_include_items/exclude ... in the SOS logic?

Yes, that would work. IF they matched. So if sos_mycol_alias = "MyColumn", and gml_include_items = "all", then the WFS DescribeFeatureType would expect ms:mycol, when the SOS is outputting ms:MyColum.

As for using gml_include_items/exclude in SOS, why did we go with defining sos_col_alias again (can't remember offhand)?

Options:

• remove sos_item_alias metadata and rely on gml_include/exclude_items. This would match things between the two (for the purposes of this ticket). This _may_ cause backwards compatibility issue
• write a msSOSDescribeObservationType in mapogcsos.c for forget about leveraging the WFS call
• pass a flag to the WFS DescribeFeatureType call like "sos=true", and update the code to look for sos_item_alias in order to output the column name

Any others?

[assefa]

Not sue if there are other options.

I am more in favor of supporting the gml metadata and have it consistent with WFS. I am not sure why It was not done like that initially. Is this too disruptive to be done for 5.4 ?

[tomkralidis]

Replying to assefa:

Not sue if there are other options.

I am more in favor of supporting the gml metadata and have it consistent with WFS. I am not sure why It was not done like that initially. Is this too disruptive to be done for 5.4 ?

Are we talking about removing support for sos_item_alias and drive everything from gml_include_items / gml_exclude_items? Thoughts:

• backward compatibility: for folks with sos_item_alias defined, and not gml_include_items/gml_exclue_items, they would end up with no column data being output. We would have to make sure there are notes and migration paths for this (which would essentially be set gml_include_items/gml_exclude_items)

• do you see this as a new feature or a bug fix per this ticket? If new feature, then tomorrow's the deadline. If bug fix, we have more time

• do you have any pointers on how to integate include_items in mapogcsos.c ?

[assefa]

Tom,

Have you use URL substitution thing in Mapserver? I tried to see if I could modify metedata values by passing them through the URL (with the idea of setting gml_include_items from the URL). I was not successful. Do you know if setting metedata is possible or even make sense?

[tomkralidis]

Replying to assefa:

Tom,

Have you use URL substitution thing in Mapserver? I tried to see if I could modify metedata values by passing them through the URL (with the idea of setting gml_include_items from the URL). I was not successful. Do you know if setting metedata is possible or even make sense?

Do you mean like: http://host/url&service=WFS&version=1.1.0&request=DescribeFeatureType&map_obs_metadata_gml_include_items=all

This would probably do it, but I can't get it to work either.

Does/can the WFS (or any OWS) code do variable substitution this way?

[assefa]

Do you mean like: http://host/url&service=WFS&version=1.1.0&request=DescribeFeatureType&map_obs_metadata_gml_include_items=all

Yes. We could setup the gml_include_items based on sos items that are available.

This would probably do it, but I can't get it to work either.

Does/can the WFS (or any OWS) code do variable substitution this way?

Don't think so. If it worked, I though It would be less disruptive for 5.4 and we could use directly the the gml_xxx items in the SOS for the 6.0 release.

[massimof]

Hello

I have noticed that also the validation of GetCapabilities xml response against the schema fails.

Should a new ticket be created or it is handled under this same?

{{{Element <sos:intendedApplication> is not allowed at this location under element <sos:ObservationOffering>. Element <sos:procedure> is not allowed at this location under element <sos:ObservationOffering>. Element <sos:observedProperty> is not allowed at this location under element <sos:ObservationOffering>. Element <sos:featureOfInterest> is not allowed at this location under element <sos:ObservationOffering>. Element <sos:responseFormat> is not allowed at this location under element <sos:ObservationOffering>. Element <sos:resultModel> is not allowed at this location under element <sos:ObservationOffering>. Element <sos:responseMode> is not allowed at this location under element <sos:ObservationOffering>. Further elements required under element <sos:ObservationOffering>. }}}

[massimof]

Sorry, the formatting was not good, I retry:

Element <sos:intendedApplication> is not allowed at this location under element <sos:ObservationOffering>.
Element <sos:procedure> is not allowed at this location under element <sos:ObservationOffering>.
Element <sos:observedProperty> is not allowed at this location under element <sos:ObservationOffering>.
Element <sos:featureOfInterest> is not allowed at this location under element <sos:ObservationOffering>.
Element <sos:responseFormat> is not allowed at this location under element <sos:ObservationOffering>.
Element <sos:resultModel> is not allowed at this location under element <sos:ObservationOffering>.
Element <sos:responseMode> is not allowed at this location under element <sos:ObservationOffering>.
Further elements required under element <sos:ObservationOffering>.

[tomkralidis]

Replying to massimof:

Hello

I have noticed that also the validation of GetCapabilities xml response against the schema fails.

Should a new ticket be created or it is handled under this same?

{{{Element <sos:intendedApplication> is not allowed at this location under element <sos:ObservationOffering>. Element <sos:procedure> is not allowed at this location under element <sos:ObservationOffering>. Element <sos:observedProperty> is not allowed at this location under element <sos:ObservationOffering>. Element <sos:featureOfInterest> is not allowed at this location under element <sos:ObservationOffering>. Element <sos:responseFormat> is not allowed at this location under element <sos:ObservationOffering>. Element <sos:resultModel> is not allowed at this location under element <sos:ObservationOffering>. Element <sos:responseMode> is not allowed at this location under element <sos:ObservationOffering>. Further elements required under element <sos:ObservationOffering>. }}}

Capabilities validates fine for me in svn trunk. If you are still having issues, please send to mapserver-users.

[massimof]

Capabilities validates fine for me in svn trunk. If you are still having issues, please send to mapserver-users.

OK, thanks.

By the way the version I'm using is 5.2.1

[tomkralidis]

Pushing to 6.0

[nsavard]

Tom, do you have any new development on this issue?

[tomkralidis]

Norm, no further development. We have to figure out per comment:11.

Assefa: any thoughts?