Please help, I’m using servoy 2.2_08 build 328, and rounding function (toFixed) is actually extremely strange and unpredictably.
I have a number field : num1, say, value of .009645
I want to round it to only 4 decimal spots.
I used a calculation field: x= num1.toFixed(4); return x;
Sometimes, I would get the correct answer of .0096
but HALF the time, I would get all 6 decimals still and wound up with .009645.
Can someone verify this? Am i doing something wrong?
please help!
Very strange! I’m using Java 1.5 and haven’t been able to get it to “mess up”. Are you using this in a calculated column (if so - stored or unstored)?
bcusick:
Very strange! I’m using Java 1.5 and haven’t been able to get it to “mess up”. Are you using this in a calculated column (if so - stored or unstored)?
I’m using Java version 1.4.2_05-b04, mainly because my red/green flagging system doesn’t work on Java 1.5. (I had to downrev the java version)
I’ve tried stored and unstored calculation field. are you using the exact same forumla that I used?
to test it, are you just typing in the raw data, then erasing it, and then typing it again. Try it about 10-15 times, I tried it about 5 times or so and it started happening.
Hi guys,
this rounding error is still occuring with Java version 1.4.2_05-b05, servoy 2.2_08 build 328.
Can someone please verify this?
I have a entry number field : testnum1
then, I have a calculation field (testnum1round) : formula: testnum1.toFixed(4).
then I enter in .009645 , it should round to .0096 , but the result becomes .00967 ???
Also, I tried putting .00975. It should round to .0098. However, the result becomes .0097 . Please help!!! this rounding issue is really fustrating.
Thanks
Rounding working fine for me.
Defined number calc field as “return zz_testnum1.toFixed(4);” and set display format to ‘#.000000’
.009645 → .009600
.00975 → .009700
Alternated numbers about 12-13 times and worked perfectly every time.
Regards
Graham Greensall
Worxinfo Ltd
Servoy Developer
Version R2 2.2-build 328
Java version 1.4.2_08-b03 (Windows XP)
grahamg:
Rounding working fine for me.Defined number calc field as “return zz_testnum1.toFixed(4);” and set display format to ‘#.000000’
.009645 → .009600
.00975 → .009700Alternated numbers about 12-13 times and worked perfectly every time.
Regards
Graham Greensall
Worxinfo LtdServoy Developer
Version R2 2.2-build 328
Java version 1.4.2_08-b03 (Windows XP)
unless i’m wrong in my rounding rules, to the 4th decimal place, shouldn’t .00975 round to .0098? I tried everything and I consistently get .009645–> .00967, and .00975–> .0097
what’s going on?
Good point - quote from Servoy Help for ‘toFixed’
Syntax: Number.toFixed([fractionDigits])
Parameters: fractionDigits - a number of digits after the decimal point.
NOTE: If a number has more digits than requested by the fractionDigits parameter, the number is rounded to the nearest number represented by fractionDigits digits. If the number is exactly halfway between two representable numbers, it is rounded away from zero (up if it is positive; down if it is negative).
Since the numbers were positive it should have rounded up - but at least it’s consistent.
Graham Greensall
Worxinfo Ltd
grahamg:
Good point - quote from Servoy Help for ‘toFixed’Syntax: Number.toFixed([fractionDigits])
Parameters: fractionDigits - a number of digits after the decimal point.
NOTE: If a number has more digits than requested by the fractionDigits parameter, the number is rounded to the nearest number represented by fractionDigits digits. If the number is exactly halfway between two representable numbers, it is rounded away from zero (up if it is positive; down if it is negative).
Since the numbers were positive it should have rounded up - but at least it’s consistent.
Graham Greensall
Worxinfo Ltd
Okay, so should we chalk this up as a bug? or is there a workaround? precision is very key to our solution since we are a medical R&D and manufacturing facility.
no not a bug that is the spec:
this running in firefox:
and this in servoy:
var p = 0.009645.toFixed(4);
application.output(p);
var p = 0.00975.toFixed(4);
application.output(p);
var p = 0.009751.toFixed(4);
application.output(p);
gives the exact same things.
explained here:
jcompagner:
no not a bug that is the spec:this running in firefox:
and this in servoy:
var p = 0.009645.toFixed(4);
application.output(p);
var p = 0.00975.toFixed(4);
application.output(p);
var p = 0.009751.toFixed(4);
application.output(p);gives the exact same things.
explained here:
Thanks for this good insight. I’ve passed the word along to our engineering team, hopefully, they can figure what needs to be done.
thanks.
Okay, we can actually solve this problem pretty easily. Hope this will help others out there with similar problem.
digitstoround is a field that defines how many digits to round to.
Then on your round calc field (where num1 is your input field):
var multiplier = Math.pow(10, digitstoround)
return Math.round(multiplier * num1) / multiplier
works like a charm!
thanks for all your help everyone.
Hello
I have a similar problem with unwanted roundings. I have a field, datatyp is ‘number’. In preferences - local, I defined the number format #.######
when I type the following numbers:
9.99999
9.999999
10
in three diffrent records and scroll through the records I have a very strange behaviour:
First time down:
9.99999
9.99999
9.99999
back up again:
10
10
10
Second time down:
10
10
10
Is this a bug? How should I format numbers, so that I can read the exact number that I have typed in?
Thank you for your reply.
Sandra
Servoy Developer Version R2 2.2.1-build 333
Java version 1.5.0_04-b05 (Windows XP)
It can be that we round something up wrong for displaying.
the change you see is because you type something that will be there but then when you display it again the real value is shown what was really stored.
And that one is stored in IEEE notation so what you see then will be rounded.
Hello Johan Compagner
Thank you for your Answer.
jcompagner:
And that one is stored in IEEE notation so what you see then will be rounded.
What is IEEE notation, an how can I make the field to display the real number I typed in without rounding?
Sandra
IEEE is the standard floating point notation.
just set a format on youre field like ##.## and you shouldn’t notice a thing.
See http://forum.servoy.com/viewtopic.php?t … t=rounding
Have you checked what type of field Servoy has created on the backend database, and whether it is suitable for storing precise values?