Hi,
After much research in our program for a certain rounding problem we discovered the following :
Servoy debug interactive console :
=>_pwp.calc_unit_price
4.800000000000001
=>_pwp.calc_unit_price = 4.8
4.8
=>_pwp.calc_unit_price
4.800000000000001
What does that calculation do exactly?
It looks like the “normal” precision problem that you have with floating point numbers: http://floating-point-gui.de/
calc_unit_price is a table field of type ‘number’. _pwp is a table foundset.
The value 4.800000000000001 is present in the database and servoy foundset !
Interesting. What kind of database do you use? Did you create that column in Servoy?
Postgres 8.4 Column was created in Servoy. (Servoy data type ‘number’ is mapped by Servoy to Postgres data type ‘double precision’).
In Servoy code :
_pwp.calc_unit_price = 4.800000000000001
_pwp.calc_unit_price == 4.8 returns false
_pwp.calc_unit_price == 4.800000000000001 returns true
_pwp.calc_unit_price = 4.8
_pwp.calc_unit_price == 4.8 returns false
_pwp.calc_unit_price == 4.800000000000001 returns true
I am glad I once did a course numerical computations
Do you have this on all your number columns? Is there maybe a column converter or something else that touches the value?
No, no calculation and no converter. We had an end user issue with our solution we could not explain.
This field is part of the threesome calc_price, calc_quantity and calc_unit_price where :
calc_price = calc_unit_price * calc_quantity and
calc_unit_price = calc_price / calc_quantity.
End user is permitted to change both calc_price and calc_unit_price so the famous rounding problem (when rounding on 2 decimals) is nearby.
End user is able to start a rounding action on 2 decimals for a group of (_pwp) records.
Before some kind of output we have a check to ensure all unit prices have no more than 2 decimal digits.
After the rounding action the end user still failed the above check and that’s where our search started which resulted in this post.
It seems imho that Servoy does some kind of rounding check in the foundset.field assignment (Servoy thinks the value of the field has not changed , 4.800000000000001 == 4.8 ).
4.800000000000001 is a value in the javascript and postgres precision domain (16 digits).
4.800000000000001 is a value in the javascript and postgres precision domain (16 digits).
After some internet searching : Above statement could be wrong…
You can use pgAdmin to manually change fields in Postgresql to Numeric to avoid these rounding issues.
You can use pgAdmin to manually change fields in Postgresql to Numeric to avoid these rounding issues.
The field value 4.800000000000001 was the result of a Servoy computation !
If I want to change the value I know other ways :
_pwp.calc_unit_price = _pwp.calc_unit_price +1
_pwp.calc_unit_price = 4.8
after this the value of _pwp.calc_unit_price is certainly 4.8 but how would I explain this to the rest of mankind ?
To be complete :
var a
a = 4.800000000000001
a == 4.8 returns false
a == 4.800000000000001 returns true
a = 4.8
a == 4.8 returns true
a == 4.800000000000001 returns false
So : same exercise with foundset.field : not ok, with a variable : ok !
Just heard from Servoy that the Servoy (data/foundset ?) maximum precision = 1e-07.
That explains my problem case !
It’s a surprise that the Servoy precision is so much less than the javascript and database maximum precision…
Where can I find that in the Wiki (or in the sales info ) ?
lwjwillemsen:
To be complete :var a
a = 4.800000000000001
a == 4.8 returns false
a == 4.800000000000001 returns true
a = 4.8
a == 4.8 returns true
a == 4.800000000000001 returns falseSo : same exercise with foundset.field : not ok, with a variable : ok !
we do that on purpose, because else many floating point nummers are constantly changes because we don’t see them as the same…
because of the rounding it can be constantly a bit changes because of that if you first set:
formvar = 4.8
and then later on
formvar = 4.8000000000000000000001
we do a special equals that just say, that those are the same. Else for example calculations would constantly change there value based on some
Or we get complains that the records go in edit but nothing really changes (visually for a user or even in the db)
So, Servoy is a no go when numeric precision in table data beyond 1e-07 is needed ?
Again, far more precision is available in javascript and database (1e-15).