bcusick:
Very strange! I’m using Java 1.5 and haven’t been able to get it to “mess up”. Are you using this in a calculated column (if so - stored or unstored)?
I’m using Java version 1.4.2_05-b04, mainly because my red/green flagging system doesn’t work on Java 1.5. (I had to downrev the java version)
I’ve tried stored and unstored calculation field. are you using the exact same forumla that I used?
to test it, are you just typing in the raw data, then erasing it, and then typing it again. Try it about 10-15 times, I tried it about 5 times or so and it started happening.
Defined number calc field as “return zz_testnum1.toFixed(4);” and set display format to ‘#.000000’
.009645 → .009600
.00975 → .009700
Alternated numbers about 12-13 times and worked perfectly every time.
Regards
Graham Greensall
Worxinfo Ltd
Servoy Developer
Version R2 2.2-build 328
Java version 1.4.2_08-b03 (Windows XP)
unless i’m wrong in my rounding rules, to the 4th decimal place, shouldn’t .00975 round to .0098? I tried everything and I consistently get .009645–> .00967, and .00975–> .0097
Parameters: fractionDigits - a number of digits after the decimal point.
NOTE: If a number has more digits than requested by the fractionDigits parameter, the number is rounded to the nearest number represented by fractionDigits digits. If the number is exactly halfway between two representable numbers, it is rounded away from zero (up if it is positive; down if it is negative).
Since the numbers were positive it should have rounded up - but at least it’s consistent.
grahamg:
Good point - quote from Servoy Help for ‘toFixed’
Syntax: Number.toFixed([fractionDigits])
Parameters: fractionDigits - a number of digits after the decimal point.
NOTE: If a number has more digits than requested by the fractionDigits parameter, the number is rounded to the nearest number represented by fractionDigits digits. If the number is exactly halfway between two representable numbers, it is rounded away from zero (up if it is positive; down if it is negative).
Since the numbers were positive it should have rounded up - but at least it’s consistent.
Graham Greensall
Worxinfo Ltd
Okay, so should we chalk this up as a bug? or is there a workaround? precision is very key to our solution since we are a medical R&D and manufacturing facility.
var p = 0.009645.toFixed(4);
application.output(p);
var p = 0.00975.toFixed(4);
application.output(p);
var p = 0.009751.toFixed(4);
application.output(p);
var p = 0.009645.toFixed(4);
application.output(p);
var p = 0.00975.toFixed(4);
application.output(p);
var p = 0.009751.toFixed(4);
application.output(p);
It can be that we round something up wrong for displaying.
the change you see is because you type something that will be there but then when you display it again the real value is shown what was really stored.
And that one is stored in IEEE notation so what you see then will be rounded.