Opened 11 years ago

Closed 9 years ago

#2950 closed enhancement (fixed)

Trim trailing zeros from fractional part of float field in dbf portion of shapefile

Reported by: bfraser Owned by: sdlime
Priority: normal Milestone: 6.0 release
Component: MapServer C Library Version: 5.2
Severity: normal Keywords: trim trailing zeros dbf shapefile
Cc: woodbri, khirst, dmorissette, jmckenna

Description

Numbers stored in a dbf field of type float have trailing zeros after the decimal point, and are retrieved (and rendered) by mapserver at full precision with all the trailing zeros. This can make labels, especially contour labels, very ugly.

Zeros trailing the decimal should be automatically trimmed. For example, 12.500000 should be rendered as 12.5 and 2000.0000000 should be rendered as 2000 with no trailing decimal.

A useful option would be to allow the setting the rendered precision in the mapfile, perhaps similar to the method in templates of [item precision=2].

Attachments (1)

text_format_sample.png (5.8 KB) - added by sdlime 9 years ago.

Download all attachments as: .zip

Change History (31)

comment:1 Changed 11 years ago by woodbri

Cc: woodbri added

comment:2 Changed 10 years ago by khirst

Cc: khirst added

comment:3 Changed 10 years ago by sdlime

Status: newassigned

comment:4 Changed 10 years ago by sdlime

Milestone: 6.0 release

comment:5 Changed 10 years ago by sdlime

Cc: dmorissette added

You know, the easiest fix might be to trim the trailing 0's (as we do with spaces) for the double/float field types all the time. For use when drawing or labeling it really shouldn't matter that the trailing 0's are gone. You can add them back if necessary with query output. This provide more performance (and less complication) than trying to do something where precision can be specified as necessary. Could use a %g conversion string in sprintf (mapxbase.c, lines 734-735) rather than building a format string from the size and precision attributes. Anyone care to try that?

Wonder what the other drivers do... Hmmm.

Other thoughts?

Steve

comment:6 Changed 9 years ago by sdlime

If anyone is interested I've been working to expand the expression parser capabilities beyond just logical expressions. One use is turning TEXT expressions into true expressions. I have a sandbox based on trunk from a few weeks ago where this is working nicely. Checkout:

http://svn.osgeo.org/mapserver/sandbox/sdlime/common-expressions/mapserver/

The broader idea is supporting a common expression syntax across drivers and this is just an offshoot. A RFC is forthcoming...

Anyway, in this version you can write something like this:

LAYER
  ...
  CLASS
    ...
    TEXT (commify(toString([area]*0.000247105381,"%.2f") + ' ac')
  END
END

This takes the 'area' attribute, converts from sq. meters to acres, formats the result using standard C printf syntax, commify's that result (e.g. 1000 => 1,000) and finally appends a ' ac'. Pretty cool IMHO.

There are a few debugging statements still present (this code works very different from existing versions) so stick to shp2img (or remove the printf's in maplayer.c).

Steve

comment:7 Changed 9 years ago by dmorissette

I didn't try the code, but I like the toString() idea which takes a printf format string. I was actually going to suggest that we try using printf formats in response to your comment in ticket #60.

In your example above, instead of + 'ac' , could we not add the units in the format string, e.g.

    TEXT (commify(toString([area]*0.000247105381,"%.2f ac"))

or maybe that would confuse the commify call?

comment:8 Changed 9 years ago by dmorissette

Cc: jmckenna added

comment:9 Changed 9 years ago by bfraser

I'm all for it, but I didn't think printf formats would allow me to

  • specify the maximum number of digits to the right of the decimal, AND
  • trim trailing zeros

I'd like to somehow specify "precision=1;trim" to get contour labels like:

DBF: Label:


2000.000000 "2000"

3.499998 "3.5" 0.000000 "0"

-0.510000 "-0.5"

And I suppose it should be internationalized for those who swap "." and ","

Brent

comment:10 in reply to:  7 Changed 9 years ago by sdlime

Replying to dmorissette:

I didn't try the code, but I like the toString() idea which takes a printf format string. I was actually going to suggest that we try using printf formats in response to your comment in ticket #60.

In your example above, instead of + 'ac' , could we not add the units in the format string, e.g.

    TEXT (commify(toString([area]*0.000247105381,"%.2f ac"))

or maybe that would confuse the commify call?

Correct, without the commify call you could place the units in the format string.

comment:11 Changed 9 years ago by dmorissette

I fear Brent is right. Unfortunately I can't find a %f or %g format that handles all the cases above the way he wants. :(

comment:12 Changed 9 years ago by sdlime

We could create a "round" function that operates on numbers and then combine that with the toString with a %g format...

comment:13 Changed 9 years ago by bfraser

Here's the code I hacked into my mapxbase.c, around line 563

        pReturnField = psDBF->pszStringField+i;

        { /* Remove trailing '0's */  //bwf
            char *p;
            int count;
            p = strchr (pReturnField,'.');       // Find decimal point.
            if (p != NULL) {
                count = 3;              // Adjust for more or less decimals.
                while (count >= 0) {    // Maximum decimals allowed.
                     count--;
                     if (*p == '\0')    // If there's less than desired.
                         break;
                     p++;               // Next character.
                }
            
                *p-- = '\0';            // Truncate string.
                while (*p == '0') {     // Remove trailing zeros.
                    *p-- = '\0';
                }
                if (*p == '.') {        // If all decimals were zeros, remove ".".
                    *p = '\0';
                }
            }

        }

Ok for testing/illustrating result, but copied from the Internet (so provenance/copyright unknown), hard-coded precision, and no i18n.

Brent

comment:14 Changed 9 years ago by bfraser

And the above function truncate, it doesn't round. So that's four bad things...

Brent

comment:15 Changed 9 years ago by sdlime

Another problem is that your hack is applied to every numeric field (I think) when in practice you don't need to do that. Rounding really is a separate function so let me try and add that. It would take the form round(number, precision)...

Steve

comment:16 Changed 9 years ago by bfraser

Ok, but don't forget the trim function (for "0" and possibly "."). Something like:

  trim( round(contour_val,1), "0." )

to give

Value:       Result:
 2000.000000 "2000"
    3.499998 "3.5"
    0.000000 "0"
   -0.510000 "-0.5"

Hmmm, the 0.000 case is a little special...

Brent

comment:17 Changed 9 years ago by sdlime

I think the %g format works like the trim... I'll create a test case with your sample values...

Steve

comment:18 Changed 9 years ago by sdlime

Ok, I added a round() function and it seems to work nicely when combined with the toString() function. See attached image. It's in the sandbox...

Steve

Changed 9 years ago by sdlime

Attachment: text_format_sample.png added

comment:19 Changed 9 years ago by sdlime

Note that we have to define a standard method to convert a number to a string so that an expression like:

TEXT ([num]) or TEXT ([AREA]/5000.0)

are valid. I set the default conversion to be "%g" since it tends to produce the most readable numeric output.

So, Brent's desired output can be achieved with:

TEXT (round([num], 0.1))

Steve

comment:20 Changed 9 years ago by jmckenna

I'm following along guys, and I'm liking what I see :)

comment:21 Changed 9 years ago by jmckenna

Steve,

We really need this functionality for the benchmarking exercise. What do we need to do to move this forward? If you commit to trunk I can test with some contour labels. Or let me know how you want me to test.

comment:22 Changed 9 years ago by sdlime

The change set as a whole is massive (as changes go) with this being only a small part. Plus, this processing represents a regression so TEXT logical expressions won't work anymore. I can't move it without a formal SLA...

IMHO trunk shouldn't be being benchmarked anyway. Without these changes or Thomas' plugin work whats there is not representative at all of what 6.0 will be.

What about using Brent's patch against the 5.6 branch as a stop gap?

Steve

comment:23 Changed 9 years ago by jmckenna

The rules of the exercise state that we must not make temporary changes in the software...will Brent's patch, or most of it, be also then included in 6.0 ? Then yes let's go for it.

comment:24 Changed 9 years ago by sdlime

Brent's code would be temporary, assuming the larger parser changes in 6.0. They would replace the need for his code, so for the 5.6 branch they wouldn't be temporary.

Which MapServer version is being used in benchmarking?

Steve

comment:25 Changed 9 years ago by jmckenna

"Comparisons will be made of the best available version of the software, be it a formal release or a development version"

So any version is fair game.

comment:27 Changed 9 years ago by dmorissette

I don't think we can include Brent's code (comment:13) in the 5.6 branch since it can have side-effects for apps/users that do need the trailing zeros in their labels or query results for some reason.

(If possible, let's try to keep the ticket focused on the actual problem and its solution, and perhaps move the benchmarking-related concerns and discussions to the mailing list)

comment:28 Changed 9 years ago by jmckenna

Steve we have a 6.0 sandbox build now, we will test it also.

comment:29 Changed 9 years ago by tbonfort

Steve, this one should be closed as fixed I think ?

comment:30 Changed 9 years ago by sdlime

Resolution: fixed
Status: assignedclosed

Correct, this was implemented as part of RFC 64. Msautotest exists as msautotest/query/text.map.

Steve

Note: See TracTickets for help on using tickets.