Trim trailing zeros from fractional part of float field in dbf portion of shapefile

[bfraser]

Numbers stored in a dbf field of type float have trailing zeros after the decimal point, and are retrieved (and rendered) by mapserver at full precision with all the trailing zeros. This can make labels, especially contour labels, very ugly.

Zeros trailing the decimal should be automatically trimmed. For example, 12.500000 should be rendered as 12.5 and 2000.0000000 should be rendered as 2000 with no trailing decimal.

A useful option would be to allow the setting the rendered precision in the mapfile, perhaps similar to the method in templates of [item precision=2].

[sdlime]

You know, the easiest fix might be to trim the trailing 0's (as we do with spaces) for the double/float field types all the time. For use when drawing or labeling it really shouldn't matter that the trailing 0's are gone. You can add them back if necessary with query output. This provide more performance (and less complication) than trying to do something where precision can be specified as necessary. Could use a %g conversion string in sprintf (mapxbase.c, lines 734-735) rather than building a format string from the size and precision attributes. Anyone care to try that?

Wonder what the other drivers do... Hmmm.

Other thoughts?

Steve

[sdlime]

If anyone is interested I've been working to expand the expression parser capabilities beyond just logical expressions. One use is turning TEXT expressions into true expressions. I have a sandbox based on trunk from a few weeks ago where this is working nicely. Checkout:

​http://svn.osgeo.org/mapserver/sandbox/sdlime/common-expressions/mapserver/

The broader idea is supporting a common expression syntax across drivers and this is just an offshoot. A RFC is forthcoming...

Anyway, in this version you can write something like this:

LAYER
  ...
  CLASS
    ...
    TEXT (commify(toString([area]*0.000247105381,"%.2f") + ' ac')
  END
END

This takes the 'area' attribute, converts from sq. meters to acres, formats the result using standard C printf syntax, commify's that result (e.g. 1000 => 1,000) and finally appends a ' ac'. Pretty cool IMHO.

There are a few debugging statements still present (this code works very different from existing versions) so stick to shp2img (or remove the printf's in maplayer.c).

Steve

[dmorissette]

I didn't try the code, but I like the toString() idea which takes a printf format string. I was actually going to suggest that we try using printf formats in response to your comment in ticket #60.

In your example above, instead of + 'ac' , could we not add the units in the format string, e.g.

    TEXT (commify(toString([area]*0.000247105381,"%.2f ac"))

or maybe that would confuse the commify call?

[bfraser]

I'm all for it, but I didn't think printf formats would allow me to

• specify the maximum number of digits to the right of the decimal, AND
• trim trailing zeros

I'd like to somehow specify "precision=1;trim" to get contour labels like:

DBF: Label:

---

2000.000000 "2000"

3.499998 "3.5" 0.000000 "0"

-0.510000 "-0.5"

And I suppose it should be internationalized for those who swap "." and ","

Brent

[sdlime]

Replying to dmorissette:

I didn't try the code, but I like the toString() idea which takes a printf format string. I was actually going to suggest that we try using printf formats in response to your comment in ticket #60.

In your example above, instead of + 'ac' , could we not add the units in the format string, e.g.

    TEXT (commify(toString([area]*0.000247105381,"%.2f ac"))

or maybe that would confuse the commify call?

Correct, without the commify call you could place the units in the format string.

[dmorissette]

I fear Brent is right. Unfortunately I can't find a %f or %g format that handles all the cases above the way he wants. :(

[sdlime]

We could create a "round" function that operates on numbers and then combine that with the toString with a %g format...

[bfraser]

Here's the code I hacked into my mapxbase.c, around line 563

        pReturnField = psDBF->pszStringField+i;

        { /* Remove trailing '0's */  //bwf
            char *p;
            int count;
            p = strchr (pReturnField,'.');       // Find decimal point.
            if (p != NULL) {
                count = 3;              // Adjust for more or less decimals.
                while (count >= 0) {    // Maximum decimals allowed.
                     count--;
                     if (*p == '\0')    // If there's less than desired.
                         break;
                     p++;               // Next character.
                }
            
                *p-- = '\0';            // Truncate string.
                while (*p == '0') {     // Remove trailing zeros.
                    *p-- = '\0';
                }
                if (*p == '.') {        // If all decimals were zeros, remove ".".
                    *p = '\0';
                }
            }

        }

Ok for testing/illustrating result, but copied from the Internet (so provenance/copyright unknown), hard-coded precision, and no i18n.

Brent

[bfraser]

And the above function truncate, it doesn't round. So that's four bad things...

Brent

[sdlime]

Another problem is that your hack is applied to every numeric field (I think) when in practice you don't need to do that. Rounding really is a separate function so let me try and add that. It would take the form round(number, precision)...

Steve

[bfraser]

Ok, but don't forget the trim function (for "0" and possibly "."). Something like:

  trim( round(contour_val,1), "0." )

to give

Value:       Result:
 2000.000000 "2000"
    3.499998 "3.5"
    0.000000 "0"
   -0.510000 "-0.5"

Hmmm, the 0.000 case is a little special...

Brent

[sdlime]

I think the %g format works like the trim... I'll create a test case with your sample values...

Steve

[sdlime]

Ok, I added a round() function and it seems to work nicely when combined with the toString() function. See attached image. It's in the sandbox...

Steve

[sdlime]

Note that we have to define a standard method to convert a number to a string so that an expression like:

TEXT ([num]) or TEXT ([AREA]/5000.0)

are valid. I set the default conversion to be "%g" since it tends to produce the most readable numeric output.

So, Brent's desired output can be achieved with:

TEXT (round([num], 0.1))

Steve

[jmckenna]

I'm following along guys, and I'm liking what I see :)

[jmckenna]

Steve,

We really need this functionality for the benchmarking exercise. What do we need to do to move this forward? If you commit to trunk I can test with some contour labels. Or let me know how you want me to test.

[sdlime]

The change set as a whole is massive (as changes go) with this being only a small part. Plus, this processing represents a regression so TEXT logical expressions won't work anymore. I can't move it without a formal SLA...

IMHO trunk shouldn't be being benchmarked anyway. Without these changes or Thomas' plugin work whats there is not representative at all of what 6.0 will be.

What about using Brent's patch against the 5.6 branch as a stop gap?

Steve

[jmckenna]

The rules of the exercise state that we must not make temporary changes in the software...will Brent's patch, or most of it, be also then included in 6.0 ? Then yes let's go for it.

[sdlime]

Brent's code would be temporary, assuming the larger parser changes in 6.0. They would replace the need for his code, so for the 5.6 branch they wouldn't be temporary.

Which MapServer version is being used in benchmarking?

Steve

[jmckenna]

"Comparisons will be made of the best available version of the software, be it a formal release or a development version"

So any version is fair game.

[jmckenna]

nothing like a live map image to hit this home:

mapserver 5.7-dev: ​http://12.189.158.78:8081/cgi-bin/mapserv?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.1076723642349,41.407828508849,2.1178733654021,41.414271246429&width=800&height=600&srs=EPSG:4326&FORMAT=image/png

mapserver 5.7-dev with the mapxbase.c change: ​http://12.189.158.78:8081/cgi-bin/mapserv_label?MAP=/benchmarking/mapserver/spain-shapefiles.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=vector_benchmarking_2010&STYLES=&BBOX=2.1076723642349,41.407828508849,2.1178733654021,41.414271246429&width=800&height=600&srs=EPSG:4326&FORMAT=image/png

[dmorissette]

I don't think we can include Brent's code (comment:13) in the 5.6 branch since it can have side-effects for apps/users that do need the trailing zeros in their labels or query results for some reason.

(If possible, let's try to keep the ticket focused on the actual problem and its solution, and perhaps move the benchmarking-related concerns and discussions to the mailing list)

[jmckenna]

Steve we have a 6.0 sandbox build now, we will test it also.

[tbonfort]

Steve, this one should be closed as fixed I think ?

[sdlime]

Correct, this was implemented as part of RFC 64. Msautotest exists as msautotest/query/text.map.

Steve