Opened 14 years ago
Last modified 13 years ago
#3395 new bug
Field Calculator Integer Conversion Errror
| Reported by: | alexckp | Owned by: | jef |
|---|---|---|---|
| Priority: | major: does not work as expected | Milestone: | Version 1.7.0 |
| Component: | Vectors | Version: | Trunk |
| Keywords: | Cc: | ||
| Must Fix for Release: | No | Platform: | All |
| Platform Version: | Awaiting user input: | no |
Description
I tried to use FC to extract an integer number using the following formula:
(SURV_BRG * 1000) - to int (to int(SURV_BRG * 100) * 10)
If the number is 62.233, the above formula will correctly return the number 3. However, when I encounter certain numbers, e.g. the number 65.1, the result is 9.99999999999272 instead of 0. I think this has to do with the to int function not returning a true integer and would appreciate if this problem could be solved.
Note:
See TracTickets
for help on using tickets.
I tried to reproduce the bug.
If I set the type of the new field as integer, the formula outputs 3 and 10 as expected.
If I set it as decimal, then I get the 9.99999.. even if the formula tries to output an integer.
Computing partial results in a new decimal field, with input value as 65.1, gives this:
SURV_BRG * 100 = 6510
to int(SURV_BRG ) = 65
to int(SURV_BRG * 100) = 6509
If affects Fedora, so changing platform to All.