GetObservation SWE DataBlock support

[cplist]

Can you expand on the swecommon response a bit more? Is this the DataBlock issue of encoding data?

Right. I'm sorry to hound folks about this, but I thought I'd throw it out there to see if it was in the realm of possibility.

E.g. Say I have 3 records of SST for a given request. Instead of returning 3 sets of GML, I'd return one set that contained a DataBlock of all the data.

Here's an example of 1 day's worth of air_pressure. ​http://preview.tinyurl.com/33hfks Notice that the eventTime filter works great. Thanks, guys! (attached to this ticket as madis.xml)

What I'm asking for (in addition to the moon) is the ability to trim down on all the repeated data so that I would end up, instead w/ one block that contained a single result block. Like this SOS service does, ​http://preview.tinyurl.com/3avorn (not the same sensor). (attached to this ticket as airdas.xml)

[tomkralidis]

I started taking a look at this. Basically, we're looking at adding another output format to SOS Server. We currently output a loose, simple O&M 0.14.7 encoding.

To add support for a SWE DataBlock, we'd have to:

1./ advertise a responseFormat in GetCapabilities in addition to <ogc:responseFormat>text/xml; subtype=om/0.14.7</ogc:responseFormat>. I've emailed the OGC SWE working group to see how to advertise multiple different O&M encodings

2./support said responseFormat in GetObservation requests. Looks like this starts to happen in mapogcsos.c, around line 2070.

So, if responseFormat=text/xml; subtype=om/0.14.7, then we proceed per the existing output encoding.

If responseFormat=WhateverTheNewFormatWillBeCalled, then we output as a SWE DataBlock. In this case, ALL data is output as ONE om:member, with an om:resultDefinition (which is basically the header), with ONE result as a dump of the data, which, in a default case, could look like CSV data, as per attachment:ticket:2248:airdas.xml​.

For the om:resultDefinition, we would output the column name and other metadata. Something like:

"sos_foo_definition" "urn:ogc:phenomenon:angle"
"sos_foo_uom"        "deg"

...where foo is the column name. If not specified, we could use defaults, maybe based on the datatype or other.

For the encoding, we could allow for optional WEB or LAYER metadata like:

"sos_encoding_blockSeparator" "FOO" 
"sos_encoding_decimalSeparator" "FOO2"
"sos_encoding_tokenSeparator" "FOO3"

...where, if not specified, the defaults would be <swe:TextBlock blockSeparator=" " decimalSeparator="." tokenSeparator=","/>, a la CSV format.

Actually, looking at this again, I think we should constrain decimalSeparator to "." (and not allow this as an optional metadata directive) because underlying data records with decimals will be encoded as such.

Would this be too much flexibility for encoding the output? Would folks take advantage of this? Or should we stick with the CSV-ish default, as per many of the SOS output examples anyway.

For the CSV default, is there any chance we could pass the internal MapServer data structure to OGR, as OGR has CSV write support?

Overall, as an OGC supported format, I think the DataBlock would get alot of mileage for those passing data around in SOS, and would reduce alot of repeatable field names, etc.

Charlton/Assefa/Daniel/all: comments most welcome.

[tomkralidis]

Correspondence from the swe.wg list:

You may use the <resultModel>-Tag in the Capabilities document of the SOS to indicate that your SOS is able to encode the same sensor data with different types of Observations. (all advertised Observation types conform to the same O&M spec version).

Have a look at our 52°North SOS implementation:

​http://mars.uni-muenster.de:8080/52nSOSv2/sos?REQUEST=GetCapabilities&SERVICE=SOS

This service indicates that one can retrieve the data either encoded in the simple "Measurement" or in the generic "Observation" format (which is more compact because it uses the SWE DataBlock). Of course, the 52north SOS supports the GetObservation operation. But (conform to the spec) it works with HTTP-POST only. We list up some links to example SOS requests here: ​http://mars.uni-muenster.de:8080/52nSOSv2

Use the link below to receive some Observations encoded with the Measurement type: ​http://mars.uni-muenster.de:8080/52nSOSv2/testClient_GetObservation_Measurement.html Just change the resultModel-parmeter in the request to "Observation" and you will receive the same data but with the alternative encoding.

So we don't have to add another responseFormat (i.e. keep the one we have), but just add responseModels.

[tomkralidis]

Small update in r6937:

I've added a resultModel "Observation", since this is what currently comes from GetObservation results (I've also added the responseMode "inline" since all responses come back as such).

If/when DataBlock is supported, then a resultModel "Measurement" will be added. Also, GetObservation requests will then need to support the resultModel parameter.

[assefa]

Tom,

couple of notes on this:

• Looking in the 52north example, "Measurement" seems to be the current default of Mapserver and Observation seems to be output DataBlock. Is this correct ? If so we need to advertise Measurement.

• In general I like the idea of MapServer being able to output different models and the Observation model with the DataBlock seems the most common way of doing things.

In the case of MapServer are the changes required to do something like this :

om:member>

<om:Observation>

<om:samplingTime>

<gml:TimeInstant>

<gml:timePosition>1995-01-02 00:00:00</gml:timePosition>

</gml:TimeInstant>

</om:samplingTime> <om:procedure xlink:href="urn:ogc:def:procedure:NS01EE0014"/>

<observedProperty>WATER QUALITY</observedProperty> <om:result>

<gml:featureMember xmlns:gml="http://www.opengis.net/gml">

<gml:NS01EE0014>

<gml:boundedBy>

<gml:Envelope srsName="urn:ogc:crs:epsg:4326" srsDimension="2">

<gml:lowerCorner>-65.0483 44.4619</gml:lowerCorner> <gml:upperCorner>-65.0483 44.4619</gml:upperCorner>

</gml:Envelope>

</gml:boundedBy> <msGeometry>

<gml:Point srsName="urn:ogc:crs:epsg:4326" srsDimension="2">

<gml:pos>-65.0483 44.4619</gml:pos>

</gml:Point>

</msGeometry> <TIME>1995-01-02 00:00:00</TIME>

<AMMON_DIS>-1</AMMON_DIS> <CAR_DIS_I>-1</CAR_DIS_I> <CAR_DIS_O>-1</CAR_DIS_O> <COLOUR>85</COLOUR> <NIT_DI_NO>-1</NIT_DI_NO> <NIT_TOTAL>0.06</NIT_TOTAL>

<TURBIDITY>-1</TURBIDITY>

</gml:NS01EE0014>

</gml:featureMember>

</om:result>

</om:Observation>

</om:member>

to

<om:resultDefinition>

<swe:DataBlockDefinition>

<swe:components>

<ns:SimpleDataRecord xmlns:ns="http://www.opengis.net/swe/0.0" xsi:type="ns:SimpleDataRecordType">

<ns:field>

<ns:Time definition="urn:ogc:data:time:iso8601"/>

</ns:field> <ns:field>

<ns:Text definition="urn:ogc:data:AMMON_DIS"/>

</ns:field> <ns:field>

<ns:Text definition="urn:ogc:data:CAR_DIS_I"/>

</ns:field> <ns:field>

<ns:Text definition="urn:ogc:data:CAR_DIS_O"/>

</ns:field> ...

</ns:SimpleDataRecord>

</swe:components> <swe:encoding>

<swe:TextBlock decimalSeparator="." tokenSeparator="," blockSeparator=" "/>

</swe:encoding>

</swe:DataBlockDefinition>

</om:resultDefinition>

<!-- appropriate values here -->

<om:result>2007-06-11T10:00:00+02,urn:ogc:def:object:feature:foi_alber,38.0 </om:result>

</om:Observation>

• No way to know the attribute types and measurements, unless we add metadata for that

• I also saw that the response to GetObservation in the example was organized where each result member was referring to one procedure. Not sure if we need to do that or how it affects MapServer sos

I am willing to put some time in the next weeks to address this.

[tomkralidis]

Assefa:

1./ We currently advertise Measurement as well, but I think we have it the wrong way. Observation is what outputs a DataBlock and Measurement is what we output now.

See mapogcsos.c line 2026 for a small stub on this, which traps Measurement as a resultModel. I've locally changed this to Observation, but won't commit (this minor change) until further discussion and more significant development.

2./ Our current way of outputting GetObservation requests (i.e. om:Measurement) is / should be the default output. If resultModel=om:Observation is found, then we output a DataBlock.

See comment:1 for definition and uom metadata for each field.

Yes, it looks like the DataBlock is referring to one procedure. In the case where we output data from > 1 procedures, I would suspect that we encode those as multiple members in a single response.

FYI in discussions with FrankW, he mentioned that it would be easier and more efficient to construct the DataBlock data (CSV-ish) format by hand as opposed to the overhead of passing to OGR.

I can help out on this one as well in the near term.

[assefa]

Tom,

Ok for the Measurement/Observation output.

When we have several procedures should we do something like this ?

<om:member>

<om:Observation>

<om:samplingTime>

<gml:TimePeriod xsi:type="gml:TimePeriodType">

<gml:beginPosition>2007-06-11T10:00:00+02</gml:beginPosition> <gml:endPosition>2007-06-11T10:00:00+02</gml:endPosition>

</gml:TimePeriod>

</om:samplingTime> <om:procedure xlink:href="urn:ogc:object:sensor:IFGI:ifgi-sensor-1a"/>

<om:procedure xlink:href="urn:ogc:object:sensor:IFGI:ifgi-sensor-1b"/>

<om:procedure xlink:href="urn:ogc:object:sensor:IFGI:ifgi-sensor-1c"/>

I would like to take a stab at this: what do you think the appropriate way approaching this, do we need a document that details the changes? You mention further discussions and I would like to know how to proceed.

[tomkralidis]

Assefa: if we have multiple procedures, would it be broken out to one om:member per procedure?

I think we need some pseudocode to figure out how to go about encoding a DataBlock. Do you think this needs to be an RFC (I'm not sure)? Or a wiki page with details.

[tomkralidis]

Assefa: check out ​http://mars.uni-muenster.de/swerep1.0/swe/trunk/om/0.0.0/observation.xsd

..which shows that one Observation has only one procedure element. This tells me that we have group the output data by procedure. What do you think?

[cplist]

I'm CC-ing folks who may be able to provide some insight here. I am absolutely NOT a SWE DataBlock guru, but they are. If I have some quick replies, I'll push them here.

[cplist]

My SWE expert comrades give that URL you provided, Tom, a thumbs up.

[assefa]

From the document, It seems to be one output per procedure ( I hope I am reading it properly :)

I also believe that we do not need an RFC for this. What I would like to see is a discussion doc where we have the different map file settings (one layer/one procedure ; one layer/several procedures, when sos_procedure_item is used ...) and what we expect the results to be for a GetObservation calls. We could maybe start with you sample data (config.map/sos_test.shp), define different ways to setup up an sos service and go from there. Can this doc be a wiki page on the mapserver doc site ? Let me know what you think.

[assefa]

Tom,

1) Here is we currently do re. procedure and GetObservation request

a) set up on the layer using the sos_procedure (eg sos_procedure" "NS01EE0017")

This produces one procedure per om:member. The same procedure is repeated for all member results coming from the layer

b) set up using sos_procedure with several procudures (eg "sos_procedure" "NS01EE0017 NS01EE0016 NS01EE0015").

This gives something like <om:procedure xlink:href="urn:ogc:def:procedure:NS01EE0017 NS01EE0016 NS01EE0015"/>

c) set up using sos_procedure_item

This produces something similar to case a) with one procedure per om:member (and the value of the procedure extracted from the data)

I think this 1b) is the one that seems inconsistent BUT we can also use a combination of sos_procedure and sos_procedure_item (​http://mapserver.gis.umn.edu/docs/howto/sos_server/#use-of-sos-procedure-and-sos-procedure-item) and that is what we should use in 1b)

So looking into these cases, I think we cover all (most ?) of what I have seen as user data model:

• user has several sensors that can fit the model of one sensor per layer: this would be done using 1a)

• user has several sensors and db contains info from all sensors: this could be done using 1b) and 1c)

2) * Looking into the code, I think the main function that would be affected would be msSOSAddMemberNode which would differ if the output is Observation or Measurement.

• an mentioned in comment:1 we can use the metadata per field to define the definition and the uom.

• I seems also that some SOS (for example the SOS server used in comment:2) always include the time element as part of the data definition (and datablock). In the case of Mapserver, or we could let the user decide to putput or not by adding as as one of the output fields

Any comments/concerns ? As I mentioned, I would like to go ahead with these changes but I am not sure if there are other aspects that should be considered.

[tomkralidis]

Replying to assefa:

From the document, It seems to be one output per procedure ( I hope I am reading it properly :)

I also believe that we do not need an RFC for this. What I would like to see is a discussion doc where we have the different map file settings (one layer/one procedure ; one layer/several procedures, when sos_procedure_item is used ...) and what we expect the results to be for a GetObservation calls. We could maybe start with you sample data (config.map/sos_test.shp), define different ways to setup up an sos service and go from there. Can this doc be a wiki page on the mapserver doc site ? Let me know what you think.

Really? Which document are you referring to? I thought one response XML document can have many om:member elements, which house one procedure, but can have multiple records _within_ the om:member

Sure, we can start up a 'SOS Server Examples' page on the main site.

[tomkralidis]

Replying to assefa:

Tom,

1) Here is we currently do re. procedure and GetObservation request

a) set up on the layer using the sos_procedure (eg sos_procedure" "NS01EE0017")

This produces one procedure per om:member. The same procedure is repeated for all member results coming from the layer

b) set up using sos_procedure with several procudures (eg "sos_procedure" "NS01EE0017 NS01EE0016 NS01EE0015").

This gives something like <om:procedure xlink:href="urn:ogc:def:procedure:NS01EE0017 NS01EE0016 NS01EE0015"/>

c) set up using sos_procedure_item

This produces something similar to case a) with one procedure per om:member (and the value of the procedure extracted from the data)

I think this 1b) is the one that seems inconsistent BUT we can also use a combination of sos_procedure and sos_procedure_item (​http://mapserver.gis.umn.edu/docs/howto/sos_server/#use-of-sos-procedure-and-sos-procedure-item) and that is what we should use in 1b)

So looking into these cases, I think we cover all (most ?) of what I have seen as user data model:

• user has several sensors that can fit the model of one sensor per layer: this would be done using 1a)

• user has several sensors and db contains info from all sensors: this could be done using 1b) and 1c)

2) * Looking into the code, I think the main function that would be affected would be msSOSAddMemberNode which would differ if the output is Observation or Measurement.

• an mentioned in comment:1 we can use the metadata per field to define the definition and the uom.

• I seems also that some SOS (for example the SOS server used in comment:2) always include the time element as part of the data definition (and datablock). In the case of Mapserver, or we could let the user decide to putput or not by adding as as one of the output fields

Any comments/concerns ? As I mentioned, I would like to go ahead with these changes but I am not sure if there are other aspects that should be considered.

OK. I agree with the above, I think we cover most cases of observational data to go into SOS. The difference with msSOSAddMemberNode when resultModel=om:Observation, is that we will have to group the result set by procedure, I believe.

[assefa]

Replying to tomkralidis:

Really? Which document are you referring to? I thought one response XML document can have many om:member elements, which house one procedure, but can have multiple records _within_ the om:member

I was referring to ​http://mars.uni-muenster.de/swerep1.0/swe/trunk/om/0.0.0/observation.xsd. Maybe I did not express it correctly. Here is what I understand:

• Observation Collection
  • contains 1 to N om: members
    • 1 om:member contains 1 to N om:Observation (or om:Measurement)
      • 1 om:Observation contains one procedure

As for the grouping here I have attached pseudo code that could be used:

What do you think ?

[assefa]

text file

[tomkralidis]

Replying to assefa:

Replying to tomkralidis:

Really? Which document are you referring to? I thought one response XML document can have many om:member elements, which house one procedure, but can have multiple records _within_ the om:member

I was referring to ​http://mars.uni-muenster.de/swerep1.0/swe/trunk/om/0.0.0/observation.xsd. Maybe I did not express it correctly. Here is what I understand:

• Observation Collection
  • contains 1 to N om: members
    • 1 om:member contains 1 to N om:Observation (or om:Measurement)
      • 1 om:Observation contains one procedure

As for the grouping here I have attached pseudo code that could be used:

What do you think ?

Thanks for the clarification.

• Observation Collection
  • contains 1 to N om: members
    • 1 om:member contains ONLY 1 om:Observation (or om:Measurement)
      • 1 om:Observation contains one procedure
      • 1 om:Observation contains one om:result

The main difference with msSOSAddMemberNode here will be, when a client requests resultModel=om:Observation, then msSOSAddMemberNode outputs one Observation per procedure, which may have many records, as opposed to the way we currently output GetObservation (hereafter referred to as the om:Measurement method), which outpus one Observation per record.

Does this make sense?

[assefa]

I added initial support (r7064) :

• supports is enable with resultModel=Observation in the GetObservation request
• supports grouping output per procedure
• uses metadata for definition and uom and separators

If you have the chance give it a try.

[tomkralidis]

I've tested this and it seems to work as expected at first glance.

I've made some initial fixes to remove build warnings and to fix namespace and schema pointers in r7065.

I will now start to go over the output for XML validity and the like.

[tomkralidis]

More fixes in r7607. Updated the SOS Server documentation to describe the new metadata elements supported.

Looks like the base functionality is there -- good work Assefa!

Reassigning to myself to work on XML validity.

[tomkralidis]

Replying to tomkralidis:

More fixes in r7607. Updated the SOS Server documentation to describe the new metadata elements supported.

Looks like the base functionality is there -- good work Assefa!

Reassigning to myself to work on XML validity.

Oops, I meant r7067

[assefa]

Thanks for the work Tom.

Couple of points

• the sos_offering_timeextent is an optional metedata. It is used in the case of om:observation to provide the sampling time parameter inside an observation (memeber:observation:time). The time output is mandatory according to the xsd. Would it be a good idea to make the metadata also mandatory ?

• I really would like to be able to give links from the mapserver docs to couple of SOS servers like the one you have sent and also be able to see the map file that is used. It would really help potential users to have concrete examples. Not sure where though to host this.

[tomkralidis]

Replying to assefa:

Thanks for the work Tom.

Couple of points

• the sos_offering_timeextent is an optional metedata. It is used in the case of om:observation to provide the sampling time parameter inside an observation (memeber:observation:time). The time output is mandatory according to the xsd. Would it be a good idea to make the metadata also mandatory ?

Yes, in the SWE world, most, if not all, observations would have a date/time associated.

Getting the final, authoritative, validating SOS and OM schemas from OGC is taking some time, as these haven't quite made it out publically. Although we're very close, it is against these final, public schemas which I will validate against.

• I really would like to be able to give links from the mapserver docs to couple of SOS servers like the one you have sent and also be able to see the map file that is used. It would really help potential users to have concrete examples. Not sure where though to host this.

Good question. How about a page called "SOS Server Examples" under ​http://mapserver.gis.umn.edu/docs/tutorial or ​http://mapserver.gis.umn.edu/docs/link ?

[assefa]

Tom, where would the application be setup ? Can we put in the docs a link to your server ? I am going to look here at DM if we can setup some "permanent" site where we could host these demo apps.

[tomkralidis]

Assefa: We can for now. The link is my own test area, but I can't guarantee it's availability as it's on a dev box. It would be nice to have a dedicated "lab" type dev box which houses a built latest svn (and maybe more), to be able to demo.

[cplist]

Excellent work, fellas! I finally got a chance to run this against our SOS dataset, and it looks dreamy. See it in all its splendor on ​http://sos-web.tamu.edu/sos-cgi/top_obs by making your way to the TIME SERIES block below the table -- particularly the 1st 2 SOS links. The eventTime works as advertised! :)

We'll begin digesting the SOS realtime soon and have more progressive feedback to offer, hopefully soon.

Again, gracias!

[tomkralidis]

I checked this against ​http://schemas.opengis.net/sweCommon/1.0.0/data.xsd and it looks like we're inline with what a DataBlock looks like.

Closing this bug as the remainder of XML checking will be done in #2246