CPU High (100%)

Questions and Answers on installation, deployment, management, locking, tranasactions of Servoy Application Server

CPU High (100%)

Postby briese-it » Tue Apr 19, 2016 9:43 am

Hello, since 1 month we have problems with our Servoy Server.

Every day the java. exe on the server is going up to 100%.
Between 9:00 and 11AM the CPU has 100% peaks and is always going down to about 3% after a few seconds.
Between 11AM and 1PM the CPU is going to 100% and after a few minutes servoy is restarting his service by himself.
Same story after lunch time.....

Since beginning of the problems 1 month ago I updated the Servoy Server from 7.4.5 to 7.4.6 and yesterday to 8.0.3, tested with different Java versions,too without a result.

There are between 50 and 120 users working on the server (depending on working hours in different time zones) with SmartClient and WebClient.

Today I made a dump when the server had a short 100% cpu peak.
See attached dump and screenshots.

Server hardware:
Virtual machine on ESXi Cluster (4 Servers with 2 x CPU E5-2660 and 256RAM)
Servoy-Server: 8CPU with 64GB RAM
SQL-Server 2008
Attachments
diagram month.PNG
peaks month
diagram month.PNG (157.34 KiB) Viewed 1326 times
diagram actual.PNG
actual CPU peaks
diagram actual.PNG (177.08 KiB) Viewed 1326 times
dump.txt
Dump
(154.39 KiB) Downloaded 36 times
Michael Harms
Briese Schiffahrts GmbH & Co.KG, Germany
- Servoy 8.1.4 Running on Windows 2008R2 DataCenter - MSSQL2008
User avatar
briese-it
 
Posts: 122
Joined: Mon Jun 20, 2011 1:50 pm
Location: Leer, Germany

Re: CPU High (100%)

Postby Gordon McLean » Tue Apr 19, 2016 11:53 am

Hi
Did you remember to re allocate the memory and java heap stuff in the servoy.sh/servoy.bat file ? I had a similar experience and embarrassingly having thrown larger and larger servers at the solution found simply changing the parameters here fixed the problem completely :oops:

There are details here
https://wiki.servoy.com/display/public/ ... Management

As a by the by the server graph was fine for 4-6 hours and then would grow over 40mins to an hour to the point it was 100% and would stay there. I concluded it was a memory leak and spent a lot of time trying to track it down. Allocating sufficient memory not the default 1mb I think it was fixed the problem instantly. Credit to Servoy it did run with a lot of users despite being starved of memory.

Cheers
Gordon
Gordon McLean
Clickdigital.com
Gordon McLean
 
Posts: 191
Joined: Wed Aug 03, 2005 12:24 pm
Location: UK

Re: CPU High (100%)

Postby briese-it » Tue Apr 19, 2016 12:12 pm

Hello Gordon,
many thanks for these informations. I'll give it a try and come back with the results :-)
Michael Harms
Briese Schiffahrts GmbH & Co.KG, Germany
- Servoy 8.1.4 Running on Windows 2008R2 DataCenter - MSSQL2008
User avatar
briese-it
 
Posts: 122
Joined: Mon Jun 20, 2011 1:50 pm
Location: Leer, Germany

Re: CPU High (100%)

Postby briese-it » Wed Apr 20, 2016 12:50 pm

Until now everything is fine. Seems that it was really a memory leak.
Thanks Gordon :-).
Michael Harms
Briese Schiffahrts GmbH & Co.KG, Germany
- Servoy 8.1.4 Running on Windows 2008R2 DataCenter - MSSQL2008
User avatar
briese-it
 
Posts: 122
Joined: Mon Jun 20, 2011 1:50 pm
Location: Leer, Germany

Re: CPU High (100%)

Postby Andrei Costescu » Wed Apr 20, 2016 12:59 pm

It didn't have to be a memory leak. It is possible that it just needed more memory to work properly (and it was at the limit).
And because it didn't have enough it would run the full garbage collector all the time (much to often :) - and that uses CPU of course) to free up heap - which was filled up again quickly and then heap needed to be garbage collected again and so on..

It would be a memory leak if without changes in the number of users served/solution/usage... the needed memory would keep growing until eventually the VM crashes or gives OOM exceptions (how fast would depend on how big of the memory leak was).
Andrei Costescu
Servoy
Andrei Costescu
 
Posts: 907
Joined: Tue Jun 26, 2007 3:14 pm

Re: CPU High (100%)

Postby briese-it » Wed Apr 20, 2016 1:09 pm

Hello Andrei.
I don't know.... First time when the problem appears there were no changes, no solution update and not more users online than before. Maybe the users are starting more queries all days since one month... I don't know.
But anyway, now it's ok.
Michael Harms
Briese Schiffahrts GmbH & Co.KG, Germany
- Servoy 8.1.4 Running on Windows 2008R2 DataCenter - MSSQL2008
User avatar
briese-it
 
Posts: 122
Joined: Mon Jun 20, 2011 1:50 pm
Location: Leer, Germany

Re: CPU High (100%)

Postby Gordon McLean » Wed Apr 20, 2016 5:06 pm

My experience was exactly as Andrei explained, you sound like your running out of memory and the CPU is struggling to keep up.

I found it was this line in the application_server/servoy_server.sh file

$CMD_LINE_START -Djava.awt.headless=true -Xmx18g -Xms1500m -XX:MaxPermSize=2500m -classpath .:lib/activation.jar:lib/antlr.jar:lib/apache-mime4j.jar:lib/BrowserLauncher2.jar:l$

changed Xms and MaxPermSize

CPU went from looking like a memory leak to working with in 4-6% instantly, scaled down the server as a result and the costs with it.

Best
Gordon
Gordon McLean
Clickdigital.com
Gordon McLean
 
Posts: 191
Joined: Wed Aug 03, 2005 12:24 pm
Location: UK

Re: CPU High (100%)

Postby Andrei Costescu » Wed Apr 20, 2016 5:27 pm

You'll see in the future how it goes. You can't easily say if it's a leak or not without further investigation.
Andrei Costescu
Servoy
Andrei Costescu
 
Posts: 907
Joined: Tue Jun 26, 2007 3:14 pm


Return to Servoy Server

Who is online

Users browsing this forum: No registered users and 1 guest

cron