CPU High (100%)

[briese-it]

Hello, since 1 month we have problems with our Servoy Server.

Every day the java. exe on the server is going up to 100%.
Between 9:00 and 11AM the CPU has 100% peaks and is always going down to about 3% after a few seconds.
Between 11AM and 1PM the CPU is going to 100% and after a few minutes servoy is restarting his service by himself.
Same story after lunch time.....

Since beginning of the problems 1 month ago I updated the Servoy Server from 7.4.5 to 7.4.6 and yesterday to 8.0.3, tested with different Java versions,too without a result.

There are between 50 and 120 users working on the server (depending on working hours in different time zones) with SmartClient and WebClient.

Today I made a dump when the server had a short 100% cpu peak.
See attached dump and screenshots.

Server hardware:
Virtual machine on ESXi Cluster (4 Servers with 2 x CPU E5-2660 and 256RAM)
Servoy-Server: 8CPU with 64GB RAM
SQL-Server 2008

[Gordon McLean]

Hi
Did you remember to re allocate the memory and java heap stuff in the servoy.sh/servoy.bat file ? I had a similar experience and embarrassingly having thrown larger and larger servers at the solution found simply changing the parameters here fixed the problem completely

There are details here
https://wiki.servoy.com/display/public/ ... Management

As a by the by the server graph was fine for 4-6 hours and then would grow over 40mins to an hour to the point it was 100% and would stay there. I concluded it was a memory leak and spent a lot of time trying to track it down. Allocating sufficient memory not the default 1mb I think it was fixed the problem instantly. Credit to Servoy it did run with a lot of users despite being starved of memory.

Cheers
Gordon

[briese-it]

Hello Gordon,
many thanks for these informations. I'll give it a try and come back with the results

[briese-it]

Until now everything is fine. Seems that it was really a memory leak.
Thanks Gordon .

[Andrei Costescu]

It didn't have to be a memory leak. It is possible that it just needed more memory to work properly (and it was at the limit).
And because it didn't have enough it would run the full garbage collector all the time (much to often - and that uses CPU of course) to free up heap - which was filled up again quickly and then heap needed to be garbage collected again and so on..

It would be a memory leak if without changes in the number of users served/solution/usage... the needed memory would keep growing until eventually the VM crashes or gives OOM exceptions (how fast would depend on how big of the memory leak was).

[briese-it]

Hello Andrei.
I don't know.... First time when the problem appears there were no changes, no solution update and not more users online than before. Maybe the users are starting more queries all days since one month... I don't know.
But anyway, now it's ok.

[Gordon McLean]

My experience was exactly as Andrei explained, you sound like your running out of memory and the CPU is struggling to keep up.

I found it was this line in the application_server/servoy_server.sh file

$CMD_LINE_START -Djava.awt.headless=true -Xmx18g -Xms1500m -XX:MaxPermSize=2500m -classpath .:lib/activation.jar:lib/antlr.jar:lib/apache-mime4j.jar:lib/BrowserLauncher2.jar:l$

changed Xms and MaxPermSize

CPU went from looking like a memory leak to working with in 4-6% instantly, scaled down the server as a result and the costs with it.

Best
Gordon

[Andrei Costescu]

You'll see in the future how it goes. You can't easily say if it's a leak or not without further investigation.