i already told in this thread that this ini file is just besides the servoy developers executable
Only for the mac it is on a different place, i guess inside the servoy.app file (i dont know for sure, havent have a mac at hand to test)
On Mac OS, it is inside the Servoy.app bundle: servoy.app/Contents/MacOS/servoy.ini
Ahhhh, now I did find it. Thank you!
Hi Patrick
By the way, I am wondering if Java does memory not handle dynamically? I would have thought that setting memory manually with min max limits etc. is a thing of the past? It doesn’t seem so though.
Regards, Robert
ptalbot:
On Mac OS, it is inside the Servoy.app bundle: servoy.app/Contents/MacOS/servoy.ini
No. And there’s no such thing as the past when it comes to optimizing/sparing memory resources.
Just keep in mind that Java is working inside a Virtual Machine which is here to abstract you from the underlying hardware, this is one of its biggest advantage (and why/how it can ‘run anywhere’), but it has its drawbacks, one of them is that like any other VM, you assign a fixed limit of memory from the start…
this shouldnt be needed, if the jvm just could give back memory.
Then that whole max setting could be erased…
Hi Johann
Just a short feedback to what we found after intensive analysis (may be to be useful for other developers with a simimal problem):
- As you said the speed of calculating the calcs is VERY much dependent on the amount of memory set in the servoy.ini file (to be found on the Mac where Patrick mentioned)
- For our case, using less the 1536 MB dramatically slows down the recalculation of (stored) calcs
- What is also interesting is that for the first time, when using 2048 MB (on our Mac OS X Server), the Java Garbage Collector IS starting to work and reduces the memory from time to time - to be seen in the status bar (bottom) of Serclipse. When assigning lesser memory, it seems that the Garbage Collector doesn’t work, the memory goes to max. and stays at max. Did you notice that as well?
- In our case, sorting the foundset also has great influence on the speed. It must be because we have calcs dependent on other calcs, and if we bring them in a “good logical” order, the speed increases. The goal is to find a sort order where dependent calcs are recalculated first, so they only get recalculated once instead o many times when ordering is “bad”
May be there is potential to optimize this on the Servoy side?
I am wondering now if our other application where a timetable parser is running for many days also can be optimized to run much faster.
PS: The Servoy.app is still shown as version 3.5 .-)
Regards
the garbage collector always works fine
The only problem i guess is that in your situation there is no garbage yet… You just query almost your whole database inside the developer/client.
for example if you use 2048MB what is the lowest value that you see when it does its stuff?
About the ordering, maybe it is the case that when you order you have your data what you need in one set.
Maybe if you dont order correctly then the same kind of data is used all over the place, but because of the huge amount you get in is already gone again the next time it is used…
So it has to re query it over and over again… (and if you touch the same calcs over and over again it will also recalc that again because it already lost its state)
jcompagner:
the garbage collector always works fine
The only problem i guess is that in your situation there is no garbage yet… You just query almost your whole database inside the developer/client.
for example if you use 2048MB what is the lowest value that you see when it does its stuff?
Around 500 MB is the lowest value it drops to. But it dosn’t behave like it does with 2048 MB when I set memory to 1024 or 1536, and that’s something I have no explanation. Why shouldn’t the value drop to around 500 MB with the lower settings like it does with the 2048 setting?
jcompagner:
About the ordering, maybe it is the case that when you order you have your data what you need in one set.
Maybe if you dont order correctly then the same kind of data is used all over the place, but because of the huge amount you get in is already gone again the next time it is used…
So it has to re query it over and over again… (and if you touch the same calcs over and over again it will also recalc that again because it already lost its state)
Yeahh, that’s what I think is the reason making it so slow. Good ordering is quite sure reducing the number of recalcing the calcs, obvious, but you first have to get the idea that this could be the problem
Regards,
Robert Huber:
Around 500 MB is the lowest value it drops to. But it dosn’t behave like it does with 2048 MB when I set memory to 1024 or 1536, and that’s something I have no explanation. Why shouldn’t the value drop to around 500 MB with the lower settings like it does with the 2048 setting?
does it really drop to 500 in the middle of doing stuff?
Then i guess it just needs memory to first being able to clear stuff up.
(i have seen that before that java takes more memory when it garbage collects and then dropping way below the memory it had before, it needs some working mem)
jcompagner:
Robert Huber:
Around 500 MB is the lowest value it drops to. But it dosn’t behave like it does with 2048 MB when I set memory to 1024 or 1536, and that’s something I have no explanation. Why shouldn’t the value drop to around 500 MB with the lower settings like it does with the 2048 setting?does it really drop to 500 in the middle of doing stuff?
Then i guess it just needs memory to first being able to clear stuff up.
(i have seen that before that java takes more memory when it garbage collects and then dropping way below the memory it had before, it needs some working mem)
It really does drop to around 500 MB in the middle of calculating! It continually raises then to near 2 GB (the max set value) and drops again to 500 MB after a short while. And exactly this behaviour must be speeding up the whole calculation process by factors! When I set the max. value to 1036 MB, I can NOT see this process coming into action!? It must have something to do with how the current Java garbage collector is programmed. And, there is surely a reason they are going to introduce a new garbage collector called G1 GC for Java 7,isn’t it?
Regards,
yes java 7 will get G1 (in java 6 you also have this already but is not enabled by default)
Hello, this post got a bit off topic, and I was hoping to hear from some community members about the original topic. Please see below…
…I assume that there are plenty of Servoy developers that recalcualte calculations in modules from within HC right? Can someone confirm that this works consistently for them? And by this I mean:
Calcualtions are in a module
HC includes the module
HC runs databaseManager.recalculate on records that have calculations that are in the module, and not in the HC solution itself.This must be common enough. Who has this working from the community?
Thanks
If we see a databaseManager.recalculate() in code it always raises a red flag. If it affects anything more than UI display or light business rules we will redo that task so as to not need to do databaseManager.recalculate(). Update tracking fields at the transactional level, compute stuff in SQL at the point when needed, use of merge tables, run an update statement and flush the clients, etc. This way we know that long term we are avoiding issues with data integrity, performance and burying business logic that too many calculations eventually leads to.
Our general view is that there is nothing elegant about an application with tons of calcs based on calcs, calcs with relations in them, calcs with globals in them, calcs getting summed, calcs not failing gracefully, etc. Finding that one business rule that you need to change or debug is a pain, you can’t be sure if your data is coming out the end of a long calculation chain is solid, and the solution will be comparatively slow.
We basically never use databaseManager.recalculate() here. It’s a situational function in my opinion and some of the uses it’s being applied to in this thread sound scary to me.
Hi David,
Thanks for the reply.
Calculated columns are common enough in models inside and outside of the Servoy ecosphere. But if the idea with Servoy calcs is don’t use them and/or or don’t count on them b/c they are too implicit and/or flaky, then that seems unreasonable to me (although I don’t doubt everything you say is true!).
I don’t want to do in the database, or in a controller, what can be done in the model layer. If I do, then what’s the point? My assumption has always been that Hibernate is in use somewhere in Servoy (and I believe it has been said that it is), although I was surprised to see it didn’t surface when the code went public. But irregardless, every major ORM supports calculated columns and they can be counted on and are used frequently in applications that implement those ORMs.
If calculations don’t work I’d like to know what doesn’t work. Up until now I have seen nothing in the documentation to suggest that they don’t work, although in my own experience, only very trivial calculations work consistently.
Thanks David…I’m now pretty frightened by my implementation; relying on features that are supposed to work, but apparently do not.
Hi David
david:
If we see a databaseManager.recalculate() in code it always raises a red flag. If it affects anything more than UI display or light business rules we will redo that task so as to not need to do databaseManager.recalculate(). Update tracking fields at the transactional level, compute stuff in SQL at the point when needed, use of merge tables, run an update statement and flush the clients, etc. This way we know that long term we are avoiding issues with data integrity, performance and burying business logic that too many calculations eventually leads to.
I seems to me that raising a red flag just for the general existence of a calculation is a bit a unspecific statement. Or do you mean it’s discussable if showing the multiplication of atribute a and attribute b as a calc is a sincere solution?
And, there are just more complex requirements that are asking for a solution. Anyone used to analyse requirements, design and model software including data, is aware that the various possible solutions at hand have to be weighted against each other - with it’s advantages and disadvantages. There is often not right or wrong.
By the way, if we are at it, there is, in my opinion, still one of the best topic dealt with requirement analysis (Problem Frames) to be started with here
http://en.wikipedia.org/wiki/Problem_Frames_Approach
or
http://www.jacksonworkbench.co.uk/steve … rames.html
I don’t think your proposed “solutions” (compute in SQL, merge tables, … are in principle better as other solutions - it would be nice to see standard problems solved with different patterns though - showing the advantages as well as disadvantages.
david:
Our general view is that there is nothing elegant about an application with tons of calcs based on calcs, calcs with relations in them, calcs with globals in them, calcs getting summed, calcs not failing gracefully, etc. Finding that one business rule that you need to change or debug is a pain, you can’t be sure if your data is coming out the end of a long calculation chain is solid, and the solution will be comparatively slow.We basically never use databaseManager.recalculate() here. It’s a situational function in my opinion and some of the uses it’s being applied to in this thread sound scary to me.
I don’t understand what exactly is your statement here? Tons of calcs is a unit I can’t assign to software engineerig. I could get an idea if you would say a blinking warning sign should appear if for example calcs exceed the number of 10% of the given attributes in a entity relationship model, or any other measurement which is based on mathematics.
I agree that the question of how solid can we expect the result to be with a not simple calc chain. I really would like to hear from all of you! But the same question arises when the implementation is done “programmatically”. Did I catch really all occuring cases, exceptions etc. Calcs can not easily solve that because there is (again) program code behind the scene to make them work.
And yes, we too (as lot’s of developer colleagues) are very hard at work to find the most simple, most robust and most logical understandable solution to a given requirement - in other words the “best” one, sometimes with, sometimes without using calcs
Regards, Robert
jbader:
Thanks David…I’m now pretty frightened by my implementation; relying on features that are supposed to work, but apparently do not.
In my view the problem with calculations is not that they don’t work – it’s that they work too well. Consequently, they can easily be abused. The whole task that calculations make easier comes with ramifications that is impossible for the calculation engine to overcome. You can’t have easy and no overhead with any platform – Servoy or whatever.
Calculations are wonderful if used appropriate to the situation. What I wrote above is our highly-developed-algorithm-based-on-many-solutions-plus-some-voodoo-magic approach to where we the draw the line when it comes to calculations. The simple version is that if databaseManager.recalculate() has to be used in a method to get things to work right for anything but a UI calc, then we’ve probably crossed the line into calc hell (also a very precise measurement).
We recently completed an almost entirely transactional based app (with very few calcs) and the speed is astounding. Overkill to use this approach on a LAN only app but if you’re looking to get speeds in smart client to be as fast as a terminal session over 2,000 miles of wire then you have to realize the limitations of calculations and design accordingly. And these kinds of design decisions are part of any programming environment.
I don’t agree that they work “too well,” in fact they frequently do not work for me at all, in my use scenario, which is why I posted this in the first place. I guess the problem with Servoy’s features not behaving for me could very well just be me, but if that were true I would expect at least one person in the community to raise their hand and say, yeah we do what you do and that works great for us, but that doesn’t seem to happen.
I’m less concerned about the performance of the calcs as we use a messaging system (that we built in Servoy,and that runs in HC) to handle record inserts and subsequent calculations. We defer everything we do to the server. Our clients only create records in queues, which servers then pick up and process further. This has been the only way we can make a WAN solution work with the amount of data we create, considering the speed we need our workflows to function. But our calcs frequently don’t fire!
we are talking here about stored calcs… unstored cacls shouldn’t have any problems and are way faster and easier to use
databaseManager.recalculate() shouldnt be used that often on stored calcs, That call only should be needed when data is changing from outside of servoy (this is logical ofcourse)
Or your calcs are nested quite deep and are depending on data from other tables that are changed even when the record where the calc is on is not in memory at all.
These kind of constructs should be avoided if possible… Because yes then you can’t expect that all data is consistent at all times according the the calc…
for example if you have orders → order_lines
and in orders you have a calc that is the order total so it just loops over the order_lines and adds the end price of every line into a total price.
But if you then also have a situation that you really only show the order_lines for example product_to_order_lines and through that relation you change the end price of an order line
then the total price of orders is out of sync. And yes you need to use recalculate on that one, because if orders where not loaded at all then servoy didn’t have any clue.
The good thing is that if you dont rely on the stored calc with queries or aggregates it doesn’t matter, because then it is just an UI thing and at the moment a user will display that order it will be updated.
These kind of situations are very difficult to handle right in what ever ORM library you use that has stored calc support.
But if you then also have a situation that you really only show the order_lines for example product_to_order_lines and through that relation you change the end price of an order line
then the total price of orders is out of sync. And yes you need to use recalculate on that one, because if orders where not loaded at all then servoy didn’t have any clue.
What is the preferred way to do this in HC? Is it to use recalculate(record) on the order table’s record? My experience is that this does not work. The only thing that does seem to work is “looking” at the record.
Recalculate returns void by the way so I have no idea what it did or didn’t do.