(Prelim) Gradient Support for Mapserver 4.4.1

[bill@…]

As I posted on the mapserver-users list, I have put together a hack to get
gradient colors from a single style.  Here is the post:

...

Regarding the gradient coloring, it works like this:  you choose a numeric
field in your data (I've tested with PostGIS, but anything you can use for
classitem or labelitem should work), and create a style like this:

STYLE
  COLOR 60 60 60
  MINCOLOR 0 0 0
  MAXCOLOR 255 255 0
  MINVALUE 0.0
  MAXVALUE 300000.0
  GRADIENTITEM "sale_price"
END

That takes the sale_price field from the shapes values, maps its value to
a percentage between MINVALUE and MAXVALUE and then picks the color that's
appropriate from the color range.

For a quick snapshot of how this looks, I've mapped relative size of
parcels (as a percentage) to these colors:

                MINCOLOR 127 29 200 #Purple
                MAXCOLOR 255 255 0  #Yellow

You can see the results here: http://www.binko.net/gradient.png

As I said, this is still in the proof of concept stage, and I haven't
tested against any rendering except GD (although in theory it should
work).

I have gotten some feedback from the -users group, but I'd appreciate any from
the core developers.  I will post a patch against 4.4.1 as well.

Bill

[bill@…]

Prelim Gradient Patch against mapserver-4.4.1

[bill@…]

Example Image

[bill@…]

Just a couple of developer comments here.

The way I did this was:
1) add the elements to the mapfile lexer etc (yuk!)
2) grab them in mapfile.c and put them on the styleObj
3) In msDrawShape, before we do anything, we call a new msMapGradient(style,
shape) function for each style.  That function changes the style's color
according the the shape value (defined in GRADIENTITEM) and the parameters.  It
also must reset the pen, since that will be cached if it isn't set to MS_PEN_UNSET

This probably is not very thread-friendly, at least not if multiple threads will
access the same styleObj.

[bill@…]

Feedback I've gotten from -users:

1) Add support for MINSYMBOLSIZE and MAXSYMBOLSIZE that uses the same
GRADIENTITEM .  This seems straightforward and useful: I think it can be done
easily - just add the file elements and everything else is in msMapGradient().
2) Add support for Some kind of increments.  Personally, I like simple things,
so something like "INCREMENT 10.0" would work for me.  Other solutions have been
discussed.
3) Add support for legends.  This is needed, but seems to be beyond me for right
now.  If anyone who has worked in legends wants my input, please let me know.

BTW: if someone will send me the connect string for CVS (even read only) I will
post a 4.5 patch.

[sdlime]

Hi All, was on vacation last week, sorry for the delayed comments. Looks like a 
sweet addition. This should be a 4.5 enhancement as opposed to 4.4 but it looks 
like that's were it's headed anyway. The CVS access information is at:

  http://cvs.gis.umn.edu/cvs.html

I gotta think about where parameters should live a bit though. Something feels 
awkward to me about all this living in a style object. What you're really 
talking about is a creating a bunch of virtual classes right?

Steve

[bill@…]

I've heard both sides of that question: whether these should be "virtual
classes" or not.  To me, this is more of a Dynamic Color.  In fact, I originally
named one of the properties I added "DYNCOLOR".  

The fundemental problem I was trying to solve (and for my purposes, I feel I
succeeded) was that I had features with an attribute that was a non-discrete
value, and I wanted to map it simply.  It turns out that its equally valid for
any attribute that has a large number of discrete values.

Because there are not a fixed number of "classes" (unless we add the "INCREMENT"
option or an equivalent), I don't really think that virtual class describes it
well.  But what do I know: I just got here :)

Bill

[sdlime]

Couldn't the gradient be used to build classes with styles etc... someplace 
outside msDrawShape? Would be better performance-wise potentially. Could be 
done in msDrawShape or even msDrawMap or msPrepareMap... Just thinking out loud 
at the moment. This gets particularly hairy with annotation and markers since 
the markers and properties get cached along with a label.

Steve

[bill@…]

Doing it higher may be difficult, as it requires the value from the shape itself
to calculate the color.  If you want a set number of classes, you can probably
get away with it.  What's nice about this is that it give you continuous color.

Is it potentially a performance issue? Perhaps.  I haven't tried with more than
a few thousand shapes.  All it really does is replace the pen color on each
shape.  I'm not sure what the performance hit on that would be.  The arithmetic
involved in calculating the color should not be a real factor.

[fwarmerdam]

Steve,

For what it's worth, I also think of it as a single class with a dynamic
color.   I think this becomes significant when we look at how this class/layer
should be represented in the legend.  In particular, we want the layer to appear
to have a continuous color, and we don't want the legend gummed up with dozens
or hundreds of classes.   

So there is the dilema.  Is this a specialized style with variable color, 
or is it a bunch of classes. 

I would like to stress that this concept of treating a variable (raster pixel or
vector attribute) as a continous value and coloring appropriately is not a
fringe capability.  It is quite core, especially with the science crowd. 

[sdlime]

I was thinking of it sort of like the color ramp support in something like
arcview. That is, request 5 classes and let the software automatically classify
data based on an attribute. I see the difference now. Let's go ahead with a
patch against 4.5 as is and think about adding an autoclass feature as a totally
seperate effort...

Steve

[bill@…]

Patch against CVS HEAD (4.5)

[bill@…]

(From update of attachment 315)
this applies the same GRADIENT
abilities to CVS HEAD as of 4/13/05

[sgillies@…]

Please don't forget to update the code that copies and writes styleObj.

[sdlime]

Bill: Can you confirm Sean's comment is addressed?

Steve

[bill@…]

I have a new patch ready to test with the copy/write code in there.  However,
the move to GD >= 2.0.16 has kindof hammered me.  My Mandrake 10.0 Official
system doesn't have an update beyond 2.0.15, so I will need to upgrade to 10.1.

Basically, I had a patch that seemed to work, until I got the update on 1225
earlier today.  I can update as of this morning, and re-apply the patch and use
that, or it will be tomorrow (at least) before I can move up to 10.1.

Bill

[sdlime]

Maybe I'm missing something but the patch doesn't look like a patch, rather a 
binary file...

Steve

[fwarmerdam]

Steve, 

It looks like it is gzipped.  Just save it as patch.gz and then do "gzip -d
patch.gz". 

[sdlime]

Guess I was missing something then, thanks Frank. I've applied the patch and 
verified that at least everything still compiles fine. The changes have been 
committed to 4.5. Probably should create (or change this one) a documentation 
bug.

Steve

[bill@…]

Very sorry!  I gzipped them out of habit: I should have asked the convention here.

I just use:
zcat patch.1305.gz | patch 
from within the mapserver directory.

I won't gzip any new ones.

As for my status on this, I need to grab the latest libgd2, build it (but not
install it) and test my new patch with that.  I upgraded my system libgd2 to the
latest Mandrake 10.0 will support, and it broke my mapserver binary (with
segfaults).  Not sure why, but I'm sick of <expletive>ing with it. :)

I'll get that done this weekend, test the new patch and have it up by Sunday if
that's ok.  I will also move to MDK 10.1 soon, but I have "real" work on this
box that can't be interrupted for that kind of a change.

Happy Tax Day! (Oh Joy!)

[sgillies@…]

I saw the CVS commit cross the wire.  No sign of the needed change to mapcopy.c.

[bill@…]

This patch adds support for copying etc. (Against 4.5)

[bill@…]

I managed to get libgd2.0.33 build and installed.

The patch seems to work fine.  However, I don't know how to exercise the
copy/write code.  If someone could test those, I'd appreciate it.

[sdlime]

I did the stuff in the last patch by hand. Specifically:

  - added copy support (also added item index copying for all styleObj items)
  - added a line to freeStyle to clean up gradientitem if set
  - added writing support, although I set it so all (mincolor/maxcolor 
included) gradient parameters are output only if gradientitem is set

Steve

[bill@…]

Great Steve,
Are you going to commit those? I'm still seeing my changes locally with no
changes for Copy/Write support on mapcopy.c and mapfile.c

[sdlime]

I committed those changes last week, here are the revisions from mapfile.c:

 * $Log: mapfile.c,v $
 * Revision 1.299  2005/04/15 19:32:33  julien
 * Bug 1103: Set the default tolerance value based on the layer type.
 *
 * Revision 1.298  2005/04/15 18:52:01  sdlime
 * Added write support for the gradient parameters to writeStyle. Parameters 
are only written if a gradientitem is set.
 *
 * Revision 1.297  2005/04/15 17:52:47  sdlime
 * Updated freeStyle to free the gradientitem if set.
 *
 * Revision 1.296  2005/04/15 17:10:36  sdlime
 * Applied Bill Benko's patch for bug 1305, gradient support.

I don't have any outstanding commits and it looks fine to me.

Steve

[sdlime]

One thing that would be nice would be to take advantage of this from 
MapScript. Unfortunately since this is item driven that's not possible with 
dynamic features. Now, I've not gone to look at the source. Might it be 
possible to expose a method in MapScript that would take 1) a style with the 
gradient min/max values and colors and 2) a value and have it return a color?

That way you could use your code to compute colors and MapScript would do the 
assignment rather than the code in msDrawShape. This could be easy to do 
depending on your implementation. Thoughts?

I ask because I have an immediate need to do something like this.

Steve

[bill@…]

Sure, Steve

This would mean a trivial change to msMapGradient() to split it into two parts:
one that gets the value off of the shape, and the other that maps the value to
the gradient.

Exposing to Mapscript is something you'd have to manage: I've no idea on it.

I can make that change and send you a patch against current CVS HEAD.  Do you
want it on this bug?

[bill@…]

Splits the graident mapping function for Mapscripts support

[bill@…]

In a discussion with Frank, he suggested that I bring this as an option for this
bug as several people are still concerned with the number of options and the
work Gradient:

> As for your first two concerns, I'm fine with any modifications to those.
> I was thinking recently that it could be simplified to:
>
> STYLE
>    COLORRANGE 0 0 0  255 255 0 # black to yellow
>    DATARANGE 0.0 100.0
>    RANGEITEM "foobar"
> END

This would simplify things: would "RANGE" be better?  It would allow things like
"ALPHACOLORRANGE", "OUTLINECOLORRANGE" and "SYMBOLSIZERANGE" to only need one
more element if people wanted to add them.

What about having two colors on one element: is that ok? -- too confusing?

Just thoughts
Bill

[sdlime]

Hmmm... I guess when I see Frank's suggestion I like it. I can't see where 
folks would get that confused, switching the terminology to "RANGE" helps with 
that. I'd support the change (coupled with the function spliting changes 
mentioned in previous comments).

Steve

[bill@…]

Gimp Mockup of Legend

[bill@…]

Steve,

Just so you don't put Frank's reputation behind that idea, I suggested it and he
suggested posting it here (although I think he liked the changes).

I'm going to go ahead and make those modifications and stop messing with the
legends.  If anyone can give feedback on the mockup, I'll get those working
eventually.  I don't want the basic functionality held up with the new release
pending etc.

Bill

[bill@…]

Patch to use new ColorRange? syntax and split function

[sdlime]

Ok, thanks for the clarification, now we can blame you! I've applied the patch
and will work on the MapScript extension, should be trivial.

Steve

[bill@…]

Documentation Patch for mapfile-reference.xml

[sdlime]

This is a short thread I had with Sean regarding the gradient/color ramp support
in 4.6. Didn't want to lose the content.

Steve

-------

Yup, I see what you mean. As I was using it I was wishing for a non-linear
interpolation, perhaps a log-based function. I suppose in map file terms we
could do something like:

COLORRAMP
   NAME 'blue2red'
   METHOD LINEAR
   MINCOLOR 0 0 255
   MAXCOLOR 255 0 0
   MINVALUE 0
   MAXVALUE 100
   INTERVALS 8
END

and then refer to the ramp by name in a STYLE:

STYLE
   ...
   COLOR 'blue2red'
END

with RANGEITEM moving to a LAYER level parameter, into the COLORRAMP itself (as
ITEM), or perhaps even encoding it in the color:

   COLOR 'blue2red:DEPTH'

What do you guys think? I don't have a good feeling for what the demand for this
functionality, like I said as I used it I found myself wanting more.

Steve

>>> Sean Gillies <sgillies@frii.com> 5/11/2005 9:29:58 AM >>>
OK, now I understand, although your example doesn't technically need 
the color ramp feature.

I think Bill's use case is crying out for a ColorRange or ColorMap 
class, rather than extension of the Style class.  Defining a new class 
now would allow for room to grow in the future when Bill wants to 
implement color ramps that are not just linear interpolations between 2 
colors.  Maybe we'll want something other than a straight line in color 
space, or want to dither to web-safe colors.  Here's an example of how 
such a thing would be used in your example:

     ramp = ColorRamp(firstcolor=colorObj(255,0,0), 
lastcolor=colorObj(0,0,255),
                      item="TEMP" minvalue=0 maxvalue=10)

     while (1):
         po = pointObj(x, y)
         style.color.setRGB(ramp.getColor(val))
         po.draw(...)

Of course, for mapserv, we'll need a place for the ColorRamp to live.  
I think it's much better to have this in the Layer than in the style.  
Maybe the style references the color ramp in the same way it currently 
references symbols.

I like the styleObj as it is, just a "dumb" bag of properties.  That's 
my $.02.  My other $.02 is that Bill's feature is something that should 
be tried out in MapServer 4.7 for a while to get it just right.  We 
should be fixing performance bugs and improving documentation at this 
point, not trying to polish a brand new feature.

Sean

On May 11, 2005, at 1:33 AM, Steve Lime wrote:

> Bill's stuff assumes that features come from regular data sources (e.g.
> shapefiles) and have an attribute with values that fall within a range
> (e.g. temperature). If you're working with features build in MapScript
> you don't have that luxury. Code might look like:
>
> $layer = ...
> $class = $layer->getClass(0);
> $style = $class->getStyle(0);
>
> while(...) {
>   # $x, $y and $val come from a database for instance
>   $point->{x} = $x;
>   $point->{y} = $y;
>   $style->setRangeColor($val);
>   $point->draw(...);
> }
>
> Steve
>
>>>> Sean Gillies <sgillies@frii.com> 05/10/05 11:24 PM >>>
> Steve,
>
> I haven't been following development of this feature and so I don't
> really understand the difference between your way and Bill's.  Can you
> show me a usage example?
>
> Sean
>
>

[fwarmerdam]

In general terms, I like moving the colormap out into it's own object,
partly because at some point I would like to be able to load pre-defined
color ramps from a file. 

 

[sdlime]

How big a deal would it be to not release this as part of 4.6? I don't think 
we'd have to back it out, just not document it as an *official* feature.

Steve

[bill@…]

Guys,

I have no problem with this change -- I don't see the need to reuse the
ColorRamps/ColorRange/Gradient (any other names?), as they are generally
specific to the range of the values on your Class -- however, moving them
doesn't reduce their usefulness, and I've always been flexible on where they live.

In general, making a "grouping" that contains all of the parameters looks more
well defined, and was what I orignally had.

I'd be careful with the word ColorRamp, as it seems to have a definition in ESRI
products (although this might be exactly what they mean -- I don't know).

As for putting it in 4.6/4.7, I don't care.  I work off of CVS HEAD and will for
a while.  It makes sense to me that this shouldn't go into a release until the
legend support is done.  (Speaking of which, I'd like more feedback on the
attached mockup.)

The only thing I'd ask is this:  If this support is removed from CVS HEAD, I
will need to maintain it as a patch -- I have maps that will need to continue to
function that use this.  I don't mind changing their mapfiles to match the new
format (even if its unstable for a while).  I'm even willing to make the changes
once we actually agree on a format.  However, maintaining the patches is a pain
in the ass -- if you want me to support this going forward, I'd appreciate being
able to commit.  If not, I'll be happy to work with whoever will be owning it.

As far as the linear nature, I agree that applying a logarithmic interpolation
would be nice, and I'm sure there are others.  I have a working layer that does
just that: however, I just apply the LOG() inside of PostGIS :-)

Bill

[dmorissette]

I haven't looked at the way this is implemented in details, but I'd like to
second the idea of using separate objects for the colorramp parameters instead
of bloating the style object.

With respect to releasing this or not in 4.6, well, even if it's there and not
documented people will use it and may rely on it, it might be best to document
that the feature is experimental and that there are plans to remove/change it in
the next release.

[bill@…]

Just to make sure we're all on the same page, the current CVS HEAD accepts
ColorRange attributes in the following format:

> STYLE
>    COLORRANGE 0 0 0  255 255 0 # black to yellow
>    DATARANGE 0.0 100.0
>    RANGEITEM "foobar"
> END

I'm fine with any format that captures the following information:

1) Start and End Colors
2) Start and End of Data Range
3) Layer Attribute to use to calculate the color for the current shape

To support Non-linear interpolation, additional attributes may be needed.

We might find that putting all of the attributes EXCEPT for the "RANGEITEM"
(perhaps renamed to RAMPITEM or whatever) outside of the class.  That way you
could use the same definitions for different layers just changing which
attribute you're keying off of.

Perhaps one item at the CLASS (or STYLE) level defined as such:

COLORRAMP "blue2red" "Attribute1"

and everything else like this (at a higher level):

COLORRAMP
   NAME 'blue2red'
#   METHOD LINEAR -- not implemented yet
   MINCOLOR 0 0 255
   MAXCOLOR 255 0 0
   MINVALUE 0
   MAXVALUE 100
#   INTERVALS 8 -- not implemented yet
END

[fwarmerdam]

I would agree that we should document this feature as experimental with the
expectation that it will change (and hopefully stabilize) for 4.8.  The feature
type in general is needed in my opinion, but I for one haven't quite settled
my mind on how the final result should be accomplished.

[sgillies@…]

So, is there a rollback or does the code stay in?  My preference would be a 
rollback until it's done the way we are agreeing it should be done.  Because
once it's in, the incentive to do it right diminishes.

[fwarmerdam]

My opinion is that the feature is already quite useful, and that it will
be hard to get feedback if is rolledback.  Furthermore, rolling it back
may well be messy. 

So, I vote for leave it in, mark it as experimental/transitional.

[sdlime]

I agree, we should not rollback...

[sdlime]

Perhaps we can address formally again in 5.4...

Steve

[apetkov]

I am interested in using this funcionality, since I would like to display grayscale rasters (that are computed on the fly) with values in the range [-255 255].

I tried it with 5.2.0, and it seems to work. Here is the relevant snippet from the map file:

CLASS
  NAME "img diff"
  STYLE
      COLORRANGE 0 0 0 255 255 0
      DATARANGE -255 255
  END
END 

Now, is it possible to display a legend like in the example above, with a continuous colorbar?

[tomkralidis]

This would also be beautiful from the WMS GetLegendGraphic side of things, especially for raster. I would also suggest this as a command line utility to generate gradient-ed legends offline, then being able to ref them in KEYIMAGE.

[apetkov]

Replying to tomkralidis:

This would also be beautiful from the WMS GetLegendGraphic side of things, especially for raster. I would also suggest this as a command line utility to generate gradient-ed legends offline, then being able to ref them in KEYIMAGE.

Thank you Tom--this is exactly what I meant. I would like to be able to generate a similar graphic as a result of a GetLegendGraphic? request, using a COLORRANGE directive, instead of defining a separate class for each group of values.

[sholl]

Ping,

is there any momentum within this issue? At least it seems to work, but the docs do not reflect anything about it. I even did not know about this possibility though.

Could this nice feature be somehow documented?

TIA for your commets.

Stephan

[havatv]

I have created a documentation ticket (#3917). I hope it is possible to document this useful feature in some way, even if it is still preliminary.

[jmckenna]

Note that I had worked on this documentation long ago, but was told by the MapServer PSC not to add it, as the feature has not been approved by the PSC (see mapserver-dev mailing list archives).

[sdlime]

Jeff's correct. We had talked about some changes we'd like to see back at the sprint but didn't feel it was appropriate to make them so close to the release. They'll be in 6.2 for sure though.

Steve