databaseManager.recalcualte(record) in HC, calcs in module

The forum to discuss the Headless version of Servoy. Web, Java and Servlet development questions can all be posted here.

Re: databaseManager.recalcualte(record) in HC, calcs in module

Postby Robert Huber » Thu Dec 02, 2010 10:04 am

Hi Johann

jcompagner wrote:are we looking at a client running inside developer? or a real smart client?


In my example above I talked about the client running inside developer. But the problem remains the same with the real client!
I also run it in the real client and the behaviour is the same.

jcompagner wrote:Then the admin page maxHeap doesn't do anything then it is your servoy.ini file that does it.


That is how I understood it so the maxHeap value set is only consumed by the real SmartClient!

jcompagner wrote:You can't make it more the 2GB if your client vm where you smart client runs in is 32Bit, you need a 64bit jvm for that.


I don't have a servoy.ini file or at least I can't find it. Do you mean the servoy.properties file? Which parameter to set to give the Developer client more memory? By the way, isn't that file overwritten by the developer or the application server?

jcompagner wrote:But still what i don't get is that you dont see loads of queries generated in the performance log...
You should see relation load queries of all the relations you touch, you should see normal foundset queries to get the next block of records..
that all besides the update queries of all the stored calcs that you update.. (why else would you run it)


Are you expecting we are making updates or are you talking about updates made by the calc update mechanism by Servoy?
We are asking ourselfs why you expect updates. What sort of updates? As we don't understand the mechanism for updating calcs in Servoy, we have just a vague idea what should happen.

Can you give an idea how the calc update mechanism works or is it described somewhere?

I should also note that we have 21 calcs associated with the table having the calc problem (the table is called "class_members"). 4 of the 21 calcs are stored calcs. Some calcs contain for loops. 5 modules are involved with the calcs but - as far as I know - has no effect in the SmartClient as the code there is "flattened", i. e. no module structure exists any more. Is the same true with the client from Servoy Developer?

Can we extend the amount of displayed information in the Performance Data page of the Servoy Application Server to see more details?

Best regards, Robert

PS: Wouldn't it be a good idea to give the client from within the Developer an own name, so the community could easily differentiate them!?
Robert Huber
7r gmbh, Switzerland
SAN Developer
www.seven-r.ch
User avatar
Robert Huber
 
Posts: 1239
Joined: Tue Aug 23, 2005 6:52 pm
Location: Schaffhausen, Switzerland

Re: databaseManager.recalcualte(record) in HC, calcs in module

Postby jcompagner » Thu Dec 02, 2010 11:53 am

the servoy.ini file is in the developer dir

\servoyinstall\developer\servoy.ini

by default we have somthing like: -Xmx512m that needs to be upped if you want the developer/debug client to have more memory

just right beside the servoy.exe you executed if you start the developer (on the mac they are on a bit different places)

if you do recalculate() then i expect that you want to recalculate your stored calculations and update them accordingly into the database.
That's why we have databaseManager.recalculate() to update stored cals (calculating them and sending the update statement to the database if the value did change) it doesn't have any other function.

If you never do see an update statement going to the database then the whole call to recalculate was completely unnecessary
Johan Compagner
Servoy
User avatar
jcompagner
 
Posts: 8828
Joined: Tue May 27, 2003 7:26 pm
Location: The Internet

Re: databaseManager.recalcualte(record) in HC, calcs in module

Postby birgit » Fri Dec 03, 2010 4:41 pm

Hi Johan

I did run the recalculate on a table with 13'000 records. There are 10 calculations defined for this table. The recalculate process was running about an hour. I attach the performance data. Does it tell you anything? Can you give any advise?

Then I was running recalculate on the same table and there were 20 calculations defined. After 15 Minutes I was at record number 360. And the process is running slower and slower. It will be after christmas until the end :(

Thanks and regards
Birgit
Attachments
Servoy Server Administration.pdf
Performance Data
(142.61 KiB) Downloaded 344 times
Birgit Rieder
7r gmbh, Switzerland
SAN Developer
User avatar
birgit
 
Posts: 302
Joined: Sun Jan 15, 2006 10:04 pm
Location: Switzerland

Re: databaseManager.recalcualte(record) in HC, calcs in module

Postby jcompagner » Fri Dec 03, 2010 5:32 pm

there is also a huge amount of things loaded, for example the top 2 queries are 2 relations around 14K 13K related queries. (and there are a few more of those)

You do update 2 columns first column results in 6600 updates and 1400 others.

The queries by itself dont do much, but i guess it is just that many data, I still think that you just stress the memory usage and garbage collection a lot
That is the most logical reason that it gets slower.

If we really want to look into this we really need to profile and see what really happens.
Johan Compagner
Servoy
User avatar
jcompagner
 
Posts: 8828
Joined: Tue May 27, 2003 7:26 pm
Location: The Internet

Re: databaseManager.recalcualte(record) in HC, calcs in module

Postby birgit » Fri Dec 03, 2010 5:42 pm

I also think, it is mostly the complexity of the calculations (two or three relations deep, loops and that there are calcs which call other calcs). So, running will not be fast. But that it does not get slower, I'd like to increment the memory it can use. Where do I do this? We cannot find a servoy.ini file. Can I create one? Can you give me an example file?

Thank you for the help and regards
Birgit Rieder
7r gmbh, Switzerland
SAN Developer
User avatar
birgit
 
Posts: 302
Joined: Sun Jan 15, 2006 10:04 pm
Location: Switzerland

Re: databaseManager.recalcualte(record) in HC, calcs in module

Postby jcompagner » Fri Dec 03, 2010 5:55 pm

i already told in this thread that this ini file is just besides the servoy developers executable
Only for the mac it is on a different place, i guess inside the servoy.app file (i dont know for sure, havent have a mac at hand to test)
Johan Compagner
Servoy
User avatar
jcompagner
 
Posts: 8828
Joined: Tue May 27, 2003 7:26 pm
Location: The Internet

Re: databaseManager.recalcualte(record) in HC, calcs in module

Postby ptalbot » Fri Dec 03, 2010 6:01 pm

On Mac OS, it is inside the Servoy.app bundle: servoy.app/Contents/MacOS/servoy.ini
Patrick Talbot
Freelance - Open Source - Servoy Valued Professional
https://www.servoyforge.net
Velocity rules! If you don't use it, you don't know what you're missing!
User avatar
ptalbot
 
Posts: 1654
Joined: Wed Mar 11, 2009 5:13 am
Location: Montreal, QC

Re: databaseManager.recalcualte(record) in HC, calcs in module

Postby birgit » Fri Dec 03, 2010 6:07 pm

Ahhhh, now I did find it. Thank you! :D
Birgit Rieder
7r gmbh, Switzerland
SAN Developer
User avatar
birgit
 
Posts: 302
Joined: Sun Jan 15, 2006 10:04 pm
Location: Switzerland

Re: databaseManager.recalcualte(record) in HC, calcs in module

Postby Robert Huber » Tue Dec 07, 2010 5:44 pm

Hi Patrick

By the way, I am wondering if Java does memory not handle dynamically? I would have thought that setting memory manually with min max limits etc. is a thing of the past? It doesn't seem so though.

Regards, Robert

ptalbot wrote:On Mac OS, it is inside the Servoy.app bundle: servoy.app/Contents/MacOS/servoy.ini
Robert Huber
7r gmbh, Switzerland
SAN Developer
www.seven-r.ch
User avatar
Robert Huber
 
Posts: 1239
Joined: Tue Aug 23, 2005 6:52 pm
Location: Schaffhausen, Switzerland

Re: databaseManager.recalcualte(record) in HC, calcs in module

Postby ptalbot » Wed Dec 08, 2010 12:09 am

No. And there's no such thing as the past when it comes to optimizing/sparing memory resources.
Just keep in mind that Java is working inside a Virtual Machine which is here to abstract you from the underlying hardware, this is one of its biggest advantage (and why/how it can 'run anywhere'), but it has its drawbacks, one of them is that like any other VM, you assign a fixed limit of memory from the start...
Patrick Talbot
Freelance - Open Source - Servoy Valued Professional
https://www.servoyforge.net
Velocity rules! If you don't use it, you don't know what you're missing!
User avatar
ptalbot
 
Posts: 1654
Joined: Wed Mar 11, 2009 5:13 am
Location: Montreal, QC

Re: databaseManager.recalcualte(record) in HC, calcs in module

Postby jcompagner » Wed Dec 08, 2010 10:44 am

this shouldnt be needed, if the jvm just could give back memory.
Then that whole max setting could be erased..
Johan Compagner
Servoy
User avatar
jcompagner
 
Posts: 8828
Joined: Tue May 27, 2003 7:26 pm
Location: The Internet

Re: databaseManager.recalcualte(record) in HC, calcs in module

Postby Robert Huber » Mon Dec 13, 2010 5:23 pm

Hi Johann

Just a short feedback to what we found after intensive analysis (may be to be useful for other developers with a simimal problem):
- As you said the speed of calculating the calcs is VERY much dependent on the amount of memory set in the servoy.ini file (to be found on the Mac where Patrick mentioned)
- For our case, using less the 1536 MB dramatically slows down the recalculation of (stored) calcs
- What is also interesting is that for the first time, when using 2048 MB (on our Mac OS X Server), the Java Garbage Collector IS starting to work and reduces the memory from time to time - to be seen in the status bar (bottom) of Serclipse. When assigning lesser memory, it seems that the Garbage Collector doesn't work, the memory goes to max. and stays at max. Did you notice that as well?
- In our case, sorting the foundset also has great influence on the speed. It must be because we have calcs dependent on other calcs, and if we bring them in a "good logical" order, the speed increases. The goal is to find a sort order where dependent calcs are recalculated first, so they only get recalculated once instead o many times when ordering is "bad"

May be there is potential to optimize this on the Servoy side?

I am wondering now if our other application where a timetable parser is running for many days also can be optimized to run much faster.

PS: The Servoy.app is still shown as version 3.5 .-)

Regards
Robert Huber
7r gmbh, Switzerland
SAN Developer
www.seven-r.ch
User avatar
Robert Huber
 
Posts: 1239
Joined: Tue Aug 23, 2005 6:52 pm
Location: Schaffhausen, Switzerland

Re: databaseManager.recalcualte(record) in HC, calcs in module

Postby jcompagner » Mon Dec 13, 2010 6:40 pm

the garbage collector always works fine
The only problem i guess is that in your situation there is no garbage yet.. You just query almost your whole database inside the developer/client.
for example if you use 2048MB what is the lowest value that you see when it does its stuff?

About the ordering, maybe it is the case that when you order you have your data what you need in one set.
Maybe if you dont order correctly then the same kind of data is used all over the place, but because of the huge amount you get in is already gone again the next time it is used..
So it has to re query it over and over again.. (and if you touch the same calcs over and over again it will also recalc that again because it already lost its state)
Johan Compagner
Servoy
User avatar
jcompagner
 
Posts: 8828
Joined: Tue May 27, 2003 7:26 pm
Location: The Internet

Re: databaseManager.recalcualte(record) in HC, calcs in module

Postby Robert Huber » Mon Dec 13, 2010 6:58 pm

jcompagner wrote:the garbage collector always works fine
The only problem i guess is that in your situation there is no garbage yet.. You just query almost your whole database inside the developer/client.
for example if you use 2048MB what is the lowest value that you see when it does its stuff?


Around 500 MB is the lowest value it drops to. But it dosn't behave like it does with 2048 MB when I set memory to 1024 or 1536, and that's something I have no explanation. Why shouldn't the value drop to around 500 MB with the lower settings like it does with the 2048 setting?

jcompagner wrote:About the ordering, maybe it is the case that when you order you have your data what you need in one set.
Maybe if you dont order correctly then the same kind of data is used all over the place, but because of the huge amount you get in is already gone again the next time it is used..
So it has to re query it over and over again.. (and if you touch the same calcs over and over again it will also recalc that again because it already lost its state)


Yeahh, that's what I think is the reason making it so slow. Good ordering is quite sure reducing the number of recalcing the calcs, obvious, but you first have to get the idea that this could be the problem :-o

Regards,
Robert Huber
7r gmbh, Switzerland
SAN Developer
www.seven-r.ch
User avatar
Robert Huber
 
Posts: 1239
Joined: Tue Aug 23, 2005 6:52 pm
Location: Schaffhausen, Switzerland

Re: databaseManager.recalcualte(record) in HC, calcs in module

Postby jcompagner » Wed Dec 15, 2010 12:06 pm

Robert Huber wrote:Around 500 MB is the lowest value it drops to. But it dosn't behave like it does with 2048 MB when I set memory to 1024 or 1536, and that's something I have no explanation. Why shouldn't the value drop to around 500 MB with the lower settings like it does with the 2048 setting?


does it really drop to 500 in the middle of doing stuff?
Then i guess it just needs memory to first being able to clear stuff up.
(i have seen that before that java takes more memory when it garbage collects and then dropping way below the memory it had before, it needs some working mem)
Johan Compagner
Servoy
User avatar
jcompagner
 
Posts: 8828
Joined: Tue May 27, 2003 7:26 pm
Location: The Internet

PreviousNext

Return to Servoy Headless Client

Who is online

Users browsing this forum: No registered users and 2 guests