When an SLD has multiple rules, try and combine them into a FILTER for db layers

[bartvde]

I'll attach an example SLD illustrating the issue.

We have a heavy database layer with a column named WEGBEHSRT. It has the possible values of: P, R, T and X. X is about 80% of the database table. The SLD only selects P, R and T, so about 20% of the table. But since there are multiple classes, the filters are not combined into a FILTER statement on the database layer.

In the ideal situation Mapserver would generate something like:

FILTER "(WEGBEHSRT = 'P' OR WEGBEHSRT = 'R' OR WEGBEHSRT = 'T')"

which would be much lighter to process for Mapserver.

[bartvde]

example SLD

[assefa]

possible patch to combine simple class expressions and the layer's filter parameter.

[assefa]

committed in trunk (r9584)

[bartvde]

Hmm, this does not work for me. If I use the SLD from this bug report, I get back an empty image, and I get the following in the debug file:

[Thu Dec 10 15:02:24 2009] [error] [client 145.50.90.204] msEvalRegex(): Regular expression error. String (P) failed expression test.
[Thu Dec 10 15:02:24 2009] [error] [client 145.50.90.204] msEvalRegex(): Regular expression error. String (R) failed expression test.
[Thu Dec 10 15:02:24 2009] [error] [client 145.50.90.204] msEvalRegex(): Regular expression error. String (T) failed expression test.
[Thu Dec 10 15:02:24 2009] [error] [client 145.50.90.204] msEvalRegex(): Regular expression error. String (P) failed expression test.
[Thu Dec 10 15:02:24 2009] [error] [client 145.50.90.204] msEvalRegex(): Regular expression error. String (R) failed expression test.
[Thu Dec 10 15:02:24 2009] [error] [client 145.50.90.204] msEvalRegex(): Regular expression error. String (T) failed expression test.
[Thu Dec 10 15:02:25 2009] [error] [client 145.50.90.204] freeLayer(): freeing layer at 0xa323a00.
[Thu Dec 10 15:02:25 2009] [error] [client 145.50.90.204] msConnPoolClose(): General error message. Closing connection geodatabase.ad.rws.nl,port:5151,5151,RWS_LEZEN,RWS_LEZEN,SDE.DEFAULT even though ref_count=3.

[assefa]

Bart,

could you set the debug to 5 in the map file as well as the web imagepath to see what the map loos like after the sld is applied. I have in my case:

CONFIG "MS_ERRORFILE" "f:/projects/mapserver-buildkit-2008/mapserver/msautotest/wxs/wms_sld_mine.log" DEBUG 5 ... WEB

IMAGEPATH "f:/tmp/ms_tmp/" ....

and I get in the log file:

[Thu Dec 10 14:26:07 2009].481000 msApplySLD(): Map file after SLD was applied f:/tmp/ms_tmp/4b214b4f_5b8_0._sld.map[Thu Dec 10 14:26:07 2009]

That would help to see if the filter element on that layer was not set properly.

[bartvde]

Somehow both a filter and an expression are created, which is causing this problem:

  LAYER
    CLASSITEM "WEGBEHSRT"
    CONNECTION "geodatabase.ad.rws.nl,port:5151,5151,RWS_LEZEN,RWS_LEZEN,SDE.DEFAULT"
    CONNECTIONTYPE SDE
    DATA "WEGEN.WEGEN_WEGVAKKEN,SHAPE"
    DUMP TRUE
      FILTER (( (WEGBEHSRT= 'P')  OR  (WEGBEHSRT= 'R')  OR  (WEGBEHSRT= 'T') ))
    MAXSCALEDENOM 100000
    METADATA
      "GML_GEOMETRY_TYPE"       "multiline"
      "WMS_METADATAURL_TYPE"    "TC211"
      "GS_PUBLICATION"  "internet"
      "GML_ROUTE_TYPE"  "Character"
      "WMS_METADATAURL_HREF"    "http://test.intranet.rijkswaterstaat.nl/services/geoservices/metadata/WEGEN.WEGEN_WEGVAKKEN"
      "ows_featureid"   "OBJECTID"
      "WFS_MAXFEATURES" "50"
      "OWS_EXTENT"      "10000 305000 280000 619000"
      "OWS_TITLE"       "wegvakken"
      "WMS_METADATAURL_FORMAT"  "text/xml"
      "GML_INCLUDE_ITEMS"       "all"
      "GML_GEOMETRIES"  "geometry"
    END
    MINSCALEDENOM 1000
    NAME "AAA70"
    PROCESSING "CLOSE_CONNECTION=DEFER"
    PROJECTION
      "init=epsg:28992"
    END
    STATUS ON
    TEMPLATE "blank.html"
    TOLERANCE 5
    TYPE LINE
    UNITS METERS
    CLASS
      NAME "P"
      EXPRESSION ("[WEGBEHSRT]" = "P")
      STYLE
        ANGLE 360
        COLOR 255 215 0
        SYMBOL "sld_line_symbol"
        WIDTH 1
      END
      TITLE "P"
    END
    CLASS
      NAME "R"
      EXPRESSION ("[WEGBEHSRT]" = "R")
      STYLE
        ANGLE 360
        COLOR 165 165 165
        SYMBOL "sld_line_symbol"
        WIDTH 1
      END
      TITLE "R"
    END
    CLASS
      NAME "T"
      EXPRESSION ("[WEGBEHSRT]" = "T")
      STYLE
        ANGLE 360
        COLOR 247 180 103
        SYMBOL "sld_line_symbol"
        WIDTH 1
      END
      TITLE "T"
    END
  END

[assefa]

Bart,

Not sure to understand the last comment "Somehow both a filter and an expression are created, which is causing this problem": the filter is the combination and the expressions are based on the sld

Looking at the layer posted, I am not sure to see what is the problem: is the filter correct (I mean syntax seems correct) but wondering if the filter is working with that SDE layer. What would be the Filter and the expressions for that layer if you would set it manually?

[bartvde]

Hi Assefa, you're right my second comment was a bit strange (I realized after posting but could not delete :-)).

If I run the generated MAP file through shp2img, there is no issue and the image is generated correctly.

However, when using the GetMap request with the SLD, I get an empty image and the errors in the log file about msEvalRegex.

Do you have any explanation why there would be this difference? What could cause the msEvalRegex errors?

[assefa]

Sorry I missed this bug this morning.

The regex output is "normal" in the sense that ogc filter will use a mapserver function msEvalRegex to validate if a certain value in the filter is of type nummeric or string (to be able to properly construct the expressions). It is a "harmless" error message.

For the map draw, I was a bit baffled but looking closely in the sld, I realized that the stroke-width was set to 0. Could you set it to 1 and try?

[bartvde]

I just realized I probably got shp2img from /usr/bin/ (so 5.2) instead of the 5.6 one, so I'll have to recheck with the 5.6 binary to be sure.

[bartvde]

This is working fine, sorry for the noise. Turns out that I was requesting an extent at which the layer was not visible, hence the white image. At least the new shp2img gives a nice warning for this :-)

[bart@hades mapserver-r9599]$ ./shp2img -m /data/OGC_UMN_services/tmp/ms_tmp/4b31d470_1509_1._sld.map -o /tmp/bart.png -l AAA70 -e 83415.05040544 405705.12820511 142443.92395354 432499.99999998 -all_debug 10
msLoadMap(): 0.099s
msLayerIsVisible(): Skipping layer (AAA70) because LAYER.MAXSCALE is too small for this MAP scale
msLayerIsVisible(): Skipping layer (AAA70) because LAYER.MAXSCALE is too small for this MAP scale
msDrawMap(): Drawing Label Cache, 0.000s
msDrawMap() total time: 0.013s
msSaveImage() total time: 0.035s
msFreeMap(): freeing map at 0xa0fc308.
freeLayer(): freeing layer at 0xa103f78.
freeLayer(): freeing layer at 0xa105e38.
freeLayer(): freeing layer at 0xa10b4e8.
freeLayer(): freeing layer at 0xa10c8f0.
shp2img total time: 0.158s

[Mavka]

This is bad idea to combine multiple rules in FILTER. There is for LayerFeatureConstraints node from SLD.

I want to change FILTER in run-time and use SLD for styles. But LayerFeatureConstraints still not supported. And Run-time Substitution via &MAP.LAYER[0].FILTER=... not work because of this patch (higher priority).

[assefa]

Could you please give me more details or an example package I can use to reproduce your error. I am not sure I understand the issue described. Thnaks

[Mavka]

explain sld FeatureConstraints

[Mavka]

full sld with FeatureConstraints

[Mavka]

It is difficult to make testing package - this ticket relate to databases only. But I have tried to explain it in 'report.pdf' and 'ornito.sld'.