Servoy Using more and more CPU on an X serve

My X served pretty much crashed a week ago and stopped web serving and servoy serving due to Servoy server using more and more CPU.

I have been monitoring this since and day by day it seems to creep up and not go down when the servoy server is idle.

Restarting the servoy server remedies the problem.

The only thing i have changed in the last month or two is switching to the Port 80 tunnel.

Has anyone else had a similar problem. What diagnostic information would be helpful to get a handle on it?

David

Hi David,

Can you provide us with some more info?
Like what version of Servoy, your servoy_server.sh, any batchprocessors running, how many users, etc.

OS 10.5.3 and now 4 (but started under 3

3.5.6

#!/bin/sh

while true
do
	java -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Xmx256m -Xms64m -XX:MaxPermSize=128m -classpath .:lib/ohj-jewt.jar:lib/MRJAdapter.jar:lib/compat141.jar:lib/commons-codec.jar:lib/commons-httpclient.jar:lib/activation.jar:lib/antlr.jar:lib/commons-collections.jar:lib/commons-dbcp.jar:lib/commons-fileupload-1.0.jar:lib/commons-logging.jar:lib/commons-pool.jar:lib/concurrent.jar:lib/dom4j.jar:lib/help.jar:lib/hibernate3.jar:lib/j2db.jar:lib/j2dbdev.jar:lib/jdbc2_0-stdext.jar:lib/jmx.jar:lib/jndi.jar:lib/js.jar:lib/jta.jar:lib/BrowserLauncher2.jar:lib/jug.jar:lib/log4j.jar:lib/mail.jar:lib/ohj-jewt.jar:lib/oracle_ice.jar:lib/server-bootstrap.jar:lib/servlet-api.jar:lib/wicket-extentions.jar:lib/wicket.jar:lib/wicket-calendar.jar:lib/xerces.jar:lib/concurrent.jar:lib/slf4j-api.jar:lib/slf4j-log4j.jar:lib/joda-time.jar:lib/rmitnl.jar:lib/networktnl.jar com.servoy.j2db.server.ApplicationServer "$@" 1>> server.log 2>> server.log
	EXITCODE=$?
	if [ "$EXITCODE" != 99 ]; then exit $EXITCODE; fi
done

Only about 4-5 users on at any time, but often none or one or two.

No batch processes.

It may be a conincidence, but it started happening after moving to the http tune i think, although it took a few weeks for the server to reach over 60% of CPU on the java service.

This is remedied by restarting the servoy service, it gets to about 20% of CPU usage constantly after about a week and remains there even if no users are logged in.

I will keep and eye on the threads, but at present it is reading about 49 with 6% CPU usage having restarted it last night with only one user logged in. I thin the threads go up with the CPU usage.
I attach a current graph of CPU usage and it will rise tomorrow i am sure.

David

David

can you test without the tunnel for a while?

Hi,

We’re running the tunnel internally on our servers as well, but do not see the problem you’re having.

The tunnel barely uses any CPU, so I cannot imagine that it is causing the CPU problems. It would be interesting to know which threads are using up the CPU, maybe you can monitor it with jconsole.

Seb,

Still cant bottom this one out.

On a practical note, is there a linux command line option to restart servoy, so i can put it on the cron each night. At least this would stop the CPU usage climbing to a disastrous level.

David

How sure are you that this is related to the Tunnel?

Any change of testing it without the Tunnel?

There might also be memory leaks in plugins you use. Any specific plugins in this installation?

Paul

I use the Calendar bean plugin and the built in mail plugin.

I also use the Java progress bar.

thats about it i think.

I have not switched the tunnel back off, but it definitely started when i started using the tunnel.

What would be the best log to capture or data when it happens again?, as it will do over the next few days?

When you started this topic, you weren’t so sure that is was directly related to the tunnel:

It may be a conincidence, but it started happening after moving to the http tune i think,

Can you try first without the tunnel for a while?

Paul

Paul to make the change back,

do i just remove

Djava.net.preferIPv4Stack=true

from my .sh file and then comment out the line in my servoy properties file.

Given i have change the port it goes over to 80, i assume i can leave this the same even with the tunnel off???

Let me know and i will change it back and see what happens. Its just its a live system so i dont want to bring my users down!

Thanks

David

Which version of Servoy are you using. Assuming you are using the latest version (with the latest version of the tunnel (2.0.4)) just remove the

SocketFactory.rmiServerFactory=com.servoy.j2db.server.rmi.tunnel.ServerTunnelRMISocketFactoryFactory

line from servoy.properties.

That should do the trick.

Another thing to try, one user noticed that the lack of entropy was causing server slowdowns.

Can you add a flag

-Djava.security.egd=file:/dev/urandom

to the java command of the servoy_server.sh command (assuming that’s the one you use to start the servoy server) and see if it helps?

See also the thread http://forum.servoy.com/viewtopic.php?f=5&t=6354&p=31329.

Might be totally unrelated, but just in case read the other thread and see if the above flag helps.

Regards,
Sebastiaan

I am back on the case of this now.

I switched a tunnel on a new server this week (mac os 10.4.10, servoy 3.5.7)

and this is waht happened to the CPU trace:
[attachment=0]Picture 4.png[/attachment]

SO the answer to switching on and off the tunnel is this is what happens! when you switch it on. this server had been stable for 6 months.

Restarting servoy brings the CPU usage right down again.

I think i will switch the tracing on a something is up with the tunnel!

David

Did you already try adding the -Djava.security.egd=file:/dev/urandom flag to the java commandline in the servoy_server.sh as mentioned below? Please try this first!

You can also try adding:

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=11111 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

And using jconsole to connect to localhost:11111 to see what is going on.

The tunnel really does nothing CPU intensive at all, so I find this very strange, and I’m going to need a lot more information to be able to see what’s going no here.

Is there anything in your logs?
Is it possible for me to connect to the server myself and inspect it with jconsole?

Regards,
Sebastiaan