So our servoy solution is getting slower and slower throughout the day as we have 15-20 users in at a time. Management has taken to restarting the Servoy Service 2x a day to get it to stay fast.
I have timers on methods, and in the morning it takes less than a second to do one feature that by noon it takes 7 seconds to do the same thing. By 2pm it can be 20 seconds.
Ideas? It’s not only only one feature that slows down, it’s the whole UI. Changing tabs, everything. My theory is the application.updateUI() commands or the dialog plugin from IT2BE (which I use application.updateUI() with to get it to fire fast and stop consistently), but I don’t have a machine that has 20 users on it to prove that with.
Anything like this known about servoy?
ellen
Ellen Meserow
Consultant, User Interfaces/Databases
Meserow Consulting ellen@meserow.com
phone 206 852 8675
fax 208 730 2049
What version of Servoy/Java/Platform (server/clients)?
Also what are the memory requirements (see about window) of these clients in the morning/noon/afternoon.
And what do these methods do that you track ?
Sorry – This is Servoy 3.5.3. I’ve given the clients a max of 128 and the server a max of 512. Java.exe is only using 128ish of that 512 – I assume that’s where to look for Servoy’s service in Task manager.
Restart of the service solves everything. We did try having everyone exit manually without restarting the service and the last user in doesn’t get faster. Nor does he get faster when he quits Servoy and re-opens it.
Only with restart of the service does it go back to the morning speeds.
The method I mentioned is using fsupdater to mark record flags. But the whole UI slows down, changing of tabs and everything – I only mentioned that one method because it has a visible timer wrapped around it and can quantifiably be compared.
Note: Database Performances statistics on the Server webpage do not degrade.
During the day do you see an increase in database connections (see database servers page in servoy-admin) ?
Also what about transactions (see transactions page in servoy-admin).
Do you see a lot of client disconnects in the servoy log ?
I.e. what I am looking for is if your database connection pool is shrinking during the day due to stuck database transactions or other causes.
During the day do you see an increase in database connections (see database servers page in servoy-admin) ?
Also what about transactions (see transactions page in servoy-admin).
Do you see a lot of client disconnects in the servoy log ?
I.e. what I am looking for is if your database connection pool is shrinking during the day due to stuck database transactions or other causes.
We are not using transactions at all - as with MS SQL 2000 if one user task manage ends/force quits his Servoy session while in transaction, it hangs the entire network. Yes, that’s true. Learned the hard way and confirmed by Jan Blok because of the way that SQL Server 2000 (and a couple other backends he said) manage transactions.
No client disconnects in servoy log.
Also, I disabled the dialog plugin from IT2BE with application.updateUI in the method and that slowed the rate of degradation significantly. I seriously suspect app.updateUI in a large multi-user environment, or the dialog plugin. Have written IT2BE about this.
If the solution is slowing down maybe it could be linked to the clients never seeming to be idle and therefore the clients are doing something which has an effect on overall solution performance.
Q: Does the client idle time alter through the day in relation to the solution becoming slower ?
I am going to try to identify the issue together with Ellen.
The strange thing is, although I can imagine that the plug-in/application.updateUI is suspect, these two are both client related methods.
That makes the claim ‘the more users, the more slowdown’ a strange one…
Thanks for your hint Harry, a good candidate as well…
If you say, the server slows down, you actually mean that performance on the client slows down, right? Is this true for “freshly” connected clients or does it happen after a while of connection time of a single client?
Have you ever enabled client tracing and checked one of the client traces? A typical slow down factor, for example, are calc errors. Those should be visible in a client trace.
If you say, the server slows down, you actually mean that performance on the client slows down, right? Is this true for “freshly” connected clients or does it happen after a while of connection time of a single client?
Good question, from what I understand it happens after a while, per client, but for all clients.
However, reading the above message again it could be something else.
Sorry guys, small delay by my departure to Mardi Gras in New Orleans and then some political activity trying to get Obama elected here. Ah, life.
Back to Servoy –
Quick thing – I don’t have time to maake a sample solution today, but I am absolutely sure that I had a real situation about six months ago where application.updateUI inside an oft-fired loop caused this exact same kind of slowdown situation. It didn’t crash, it just slowed down over time – even when you were navigated away from the loop in question. I am pretty sure there’s something up with application update.UI().
That said, when we disabled the dialog plugin with it’s application.updateUI line that comes before it starts and stops, the slowdown continued – but MUCH less quickly degrading and much less severe.
BTW, I cannot swear that the more users the more degradation. That was an instinct, unproven. However in testing we never saw this slowdown, or never noticed it. It was only when we got many users in that we found it. We have now instituted a midnight restart of the service, and we are out of our busy period – with only 5 concurrent users at a time or so. So we can’t test as well.
As for Patrick’s questions – I do mean that the clients slowdown when I say the server slows down. And a “fresh” login when the degradation is underway will have the slowness problem. Even if we get all the users out of the system, and then log in as the sole user, the degradation is there at the exact same amount as before with lots of users. The only solution is a restart of the service. Then everyone gets in with fast speed again.
Memory usage of the Dialog plug-in (to show progress of the creation of records as in the sample solution) remains at 114bytes.
No matter what I do…
The Servoy classes responsible for creating the new rows however show increasing memory usage with every task when I use the controller.
The nice thing is that when I use the foundset, it shows increasing memory usage so long as new rows are created but drops again after a .clearFoundset().
Which makes sense.
Question: could it be that you are working on the controller instead of the foundset?
ellenmeserow:
…
Restart of the service solves everything. We did try having everyone exit manually without restarting the service and the last user in doesn’t get faster. Nor does he get faster when he quits Servoy and re-opens it.
Only with restart of the service does it go back to the morning speeds.
…
Could you give it a try with adminpage setting: servoy.useObjectPool to false?
We have same problem over here too.
But only with Servoy 3.5 customers.
We have 2 customers with Servoy 3.5 with 50 or so clients with nice servers with 4Gb of RAM en multi-cpu’s.
We use almost the same solution on another customer with Servoy 3.1 with 70 clients, and all runs well since ages.
Everytime the performance slows down, I noticed that java.exe on the server is consuming a lot of CPU. Starting up a client will take minutes, and the CPU goes to almost 100% met all 4 cpu’s, during the whole of the startup of a fat-client. When we are not starting up clients java.exe keeps running up to 25% thru 30% constantly…then we restart Servoy Application Server…and every thing turns to normal, after the restart about 40 people start up the Servoy clients almost at the same time (during a period of 10 minutes), and everything runs smooth…untill the next performance crash half a day later.
I have doing some ajustments with Jan Blok…yesterday night I have done:
servoy.useObjectPool false
setting servoy.vmClientArgs
Today the whole day i haven’t heard any complaints. So I will you updated if this works.
So glad someone else is seeing this. We’re excited to see how those changes work for you – we’re implementing them too but won’t be able immediately to see if they work, as our concurrent users has gone down to three to five right now (compared to the 25 we saw during january when people do their budgets here). Keep us updated, and THANKS! (to you too Jan Blok!)
I have a 25 user 3.5.x system on Windows 2003 Server, and we sometimes get users complain about a slowdown after a few days…
In the past blamed this on a Windows security updates. A window pops up saying there are updates… The guy looking after the server installs them and servoy speeds up again. Maybe it has nothing to do with the updates, but we happen to reboot Servoy server which clears whatever causes the problem. Will watch this a bit closer…
Both properties you can find on the admin page of Servoy 3.5.x
servoy.useObjectPool true, makes large datasets smaller to be sent over the network because the same primitive objects in a set are only serialized once, but it seems not all JVM’s garbage collectors are too happy about this. In upcoming servoy releases we will make this property false by default.
servoy.vmClientArgs we tried ourselves (on our servers) an option (which we shared with tweetie) which makes objects live longer in the smartclient, which is “-XX:SoftRefLRUPolicyMSPerMB=3600000”
If you experience performance problems on the server, set useObjectPool to false, the second option is only for people trying to squeeze out the last bit of performance on the smartclient.
At this moment it looks like the performance problems are gone.
We have set these above mentioned setting with 2 of our customers…and they stopped complaining. We havent have to restart since last Thursday.
I’ll be monitoring this further.
Any news on the stuff Marcel mentioned?
IT2Be:
Found two interesting things:
Memory usage of the Dialog plug-in (to show progress of the creation of records as in the sample solution) remains at 114bytes.
No matter what I do…
The Servoy classes responsible for creating the new rows however show increasing memory usage with every task when I use the controller.
The nice thing is that when I use the foundset, it shows increasing memory usage so long as new rows are created but drops again after a .clearFoundset().
We tried just changing the object pool setting, not the other – and we are still having a slowdown – though not NEARLY as much. Our test for this starts with a quick method which in the morning (after an automated midnight restart of the Servoy service) takes less than a second. Before the objectpool change it would take up to 20 seconds by late afternoon if we had lots of users in (around 20 users). Yesterday, after 16 hours of uptime and a consistent 5 concurrent users (different users, but around 5 all day), it took 5 seconds.
So we’re going to add the string to VMclientargs today – it may not go into effect until tomorrow (depending on when restarting the service is convenient for users). We’ll report back.
Ignore the above – I just put in the VMClientArgs on our server and restarted and noticed that ObjectPool was back on true! I have no idea how this reverted, other than perhaps in our auto-restart of the service at midnight it picks up a different settings file than the one its using now (there are two on the machine since we are still battling the 64 bit problem notated in http://forum.servoy.com/viewtopic.php?p=49261)?
After this restart it held the false setting, and the new -XX:SoftRefLRUPolicyMSPerMB=3600000 setting in VMClientArgs. Will watch tomorrow to see if it becomes true again.
In any case that five second slowdown doesn’t count cause ObjectPool was set to true. We’re now in a real test.