Class expressions: embedding Python into MapServer

[sgillies@…]

MapServer Enhancement Proposal

Python Expressions for MapServer 4.6
====================================

:Authors:
    Sean Gillies, sgillies@frii.com

:Date: 15 December 2004

Summary
-------
Optionally embed Python into MapServer so that Python expressions can be
evaluated in msEvalExpression(), joining the existing expression types
as a new, 4th type.

Motivation
----------
MapServer's existing expressions have proved themselves useful but have
their limits.  Adding new function such as the new length() is a drain on
developer time and likely to add new bugs. Embedding Python allows us to
leverage a rich and mature expression syntax.  Furthermore, embedding
Python lets us sidestep the shared lexer bottleneck.

Embedding Python
----------------
Python's C API makes embedding Python easy.  There is a PyRun_String()
function with a py_eval_input mode that is specifically designed for
embedding Python expression evaluation into other applications.  Linking
MapServer against the Python library, which Python mapscript users do
routinely, is all that is required.  No other software required.

Details
-------

* Will be implemented in a manner that breaks trail for PHP expressions.

* Python expressions are to be optional, configured like 
  --with-python-expressions --with-python=PATH

* We will load all feature items and feature geometry into an object
  named "f" so that they can be accessed like

    "int(f.FID) == 1001" or "int(f['FID']) == 1001"

  Note that since MapServer currently treats all attributes as strings,
  users will have to explicitly cast values in expressions.
  
* All the built-in string, float, int stuff is immediately available,
  allowing for

    "f.FNAME.upper() == 'FOO'" or "f.FNAME.find('F') >= 0"

* We get to take advantage of negative assertions and sequence
  inclusion:

    "f.FNAME.upper() != 'FOO'" or "f.FNAME not in ['Foo', 'Bar']"

* We may allow import of other Python modules into the namespace for
  evaluation.  Candidates include the math and regular expression
  modules.  Python web frameworks like Zope, Quixote, etc have already
  identified for us which ones are both useful and safe.

* Python mapscripters will be able to eval within the context of their
  scripts' local namespace.  No limits to what they can use in their
  expressions.

[dmorissette]

Can you please explain what you mean by:

> * Will be implemented in a manner that breaks trail for PHP expressions.

[sgillies@…]

Daniel, what I mean is that PHP expressions, should they be embedded,
can hang from the same hook.  In msEvalExpression there should be a new case, 
let us call it "rich expressions".  If you configured for python expressions
you'd get them when you used a rich expression, if you configured for php, you'd
eval a php expression.

Frank's idea originally, and intended to prevent the kind of awkwardness we now
have with filters having different meanings in different layer contexts.

[sdlime]

I don't have any problems with this. You'll need a way to identify Python 
expressions within the initial parsing. Will also need to fix msLayerWhichItems 
to retrieve all attributes. MapServer tries hard to only retrieve the columns 
it needs, which may be overly complicating things. MapServer currently scans 
expressions (and the various ...item properties) to build an item list and that 
won't be possible with Python expressions.

I'll be very curious to see how this works AND what performance is like (I hope 
better). I'd love to turn to a 3rd party expression handler but never found 
anything stand-alone (like regex). I could see eventually, like v.5, replacing 
the current handler altogether (keep string and regex, but new logical 
expression handler). That said, what other options are there besides Python, 
like a general SQL expression handler?

Steve

[fwarmerdam]

Steve, 

I am doubtful that Python (or other) expression handling will be significantly 
faster than what is already there, but hopefully they won't be significantly
slower. 

I do maintain a retargetable SQL WHERE clause parser/evaluator for use in OGR
and a few other projects.  It could be used to provide WHERE clause style 
filtering/expressions in MapServer though I am not sure there is a compelling
reason to do so.  It has most core operators, but not much in the way of 
functions.  

[sdlime]

I don't want to get too far off topic. I was hoping there was a ANSI-compliant 
SQL parser that was built to be plugged into 3rd party apps. Oh well...

The inclusion of the python engine seems a good idea. I'd like to use it as an 
opportunity to simplify the current engine as well (if possible). Attribute 
handling is one area to think about.

Steve

[sgillies@…]

To get us back on track, here is a pseudocode (Python) example of how this will
all work.  Python's C API parallels Python code, so the actual implementation
will be very much like this.

In msShapeGetClass:
-------------------

    # Get items and values for feature
    items = whichItems(layer)
    values = whichValues(layer, feature)
 
    # Iterate over classes
    for c in classes:
        if msEvalExpression(c.expression, items, values):
            # this is our class
            break
        

In msEvalExpression:
--------------------

    # Make an attribute record and put it into "my_locals" for use
    # in expression evaluation
    f = AttributeRecord(fields, values)
    my_locals = locals()
    my_locals.update({'f': f})

    # This stands in for the case/switch logic
    if isRichExpr(expression):
        
        # 2nd and 3rd args to Python eval() are global and local
        # namespaces as dictionaries.  This is exactly the same
        # for the Python C API, and this is how we "pass" the 
        # feature attribute record object into the evaluation.
        if eval(expression, globals(), my_locals):
            # this is the class
            break

[sgillies@…]

Hi all, I've talked myself into putting this off for a while.  As I see it, there
are two other good options

A) Use PostGIS or Oracle or SDE which have real joins, embedded functions, and   
   subselects.  Proven, reliable, scalable.
B) Classify in mapscript (assigning a "classindex" and "text" to shapes) and then
   popping the shapes into an inline layer using addFeature().  The only downside
   here is the amount of memory that may need to be allocated to store features.

For the time being, I am going with A when I can and B when I cannot.  I will be
continuing to use the proposed "f" namespace for feature attributes in my PCL
project, so that will have all the bugs worked out by the time I return to this
issue.

[sgillies@…]

Reviewing my bugs and dropping the priority of this one to the lowest.  Very
doubtful that I'll start on this.

[sgillies@…]

definitely will not start on this.