How many threads does Servoy 8.2.x Server need?

swingman · December 8, 2017, 11:23am

Hi all,

I’m looking at two Servoy servers, one running 7.4.x and one running 8.2.1 – Mac OS X and smart client, both with about 25 users connected.

Looking the activity monitor, the 7.4.x server uses less than 200 threads and has been up for almost a month.
The 8.2.1 server has been up about 4 hours and is using 700 threads right now and it is slowly increasing. I expect it to hit the OS thread limit and restart itself some time in the afternoon.

The 8.2.1 is running on the 1.8.0_151.jdk while the 7.4.x server is running an older 1.8.0_65.jdk.

Another difference is that the 8.2.1 solution uses the streamFilesToServer / streamFilesFromServer functionality…

Anybody come across any similar problem?
What should I look for / change?

swingman · December 11, 2017, 10:01am

I have made various tests over the weekend, with one smart client, with one batch processor, tried 3 different JDKs.
I set the ~RMI timeout to 15 seconds and this seemed to help slightly, but I would still get a new thread every 4 minutes.

Just after startup I had (X.X.X.XXX is the IP of my single smart client):

Thread: RMI TCP Connection(1)-X.X.X.XXX, state: WAITING, total cpu time: 33.913ms, total user time: 32.251ms
Thread: RMI TCP Connection(2)-X.X.X.XXX, state: WAITING, total cpu time: 53.636ms, total user time: 51.49ms
Thread: RMI TCP Connection(3)-X.X.X.XXX, state: WAITING, total cpu time: 901.693ms, total user time: 848.269ms
Thread: RMI TCP Connection(4)-X.X.X.XXX, state: WAITING, total cpu time: 1.833ms, total user time: 1.604ms

After 7 hours I would have this:

Thread: RMI TCP Accept-1099, state: WAITING, total cpu time: 212.237ms, total user time: 123.037ms
Thread: RMI TCP Connection(1)-X.X.X.XXX, state: WAITING, total cpu time: 33.913ms, total user time: 32.251ms
Thread: RMI TCP Connection(2)-X.X.X.XXX, state: WAITING, total cpu time: 53.636ms, total user time: 51.49ms
Thread: RMI TCP Connection(3)-X.X.X.XXX, state: WAITING, total cpu time: 901.693ms, total user time: 848.269ms
Thread: RMI TCP Connection(4)-X.X.X.XXX, state: WAITING, total cpu time: 1.833ms, total user time: 1.604ms
Thread: RMI TCP Connection(5)-X.X.X.XXX, state: WAITING, total cpu time: 1.562ms, total user time: 1.334ms
Thread: RMI TCP Connection(6)-X.X.X.XXX, state: WAITING, total cpu time: 1.601ms, total user time: 1.369ms
Thread: RMI TCP Connection(7)-X.X.X.XXX, state: WAITING, total cpu time: 1.479ms, total user time: 1.255ms
Thread: RMI TCP Connection(8)-X.X.X.XXX, state: WAITING, total cpu time: 1.535ms, total user time: 1.295ms
Thread: RMI TCP Connection(9)-X.X.X.XXX, state: WAITING, total cpu time: 7.663ms, total user time: 6.819ms
Thread: RMI TCP Connection(10)-X.X.X.XXX, state: WAITING, total cpu time: 118.727ms, total user time: 109.448ms
Thread: RMI TCP Connection(11)-X.X.X.XXX, state: WAITING, total cpu time: 1.463ms, total user time: 1.231ms
Thread: RMI TCP Connection(12)-X.X.X.XXX, state: WAITING, total cpu time: 1.338ms, total user time: 1.121ms
Thread: RMI TCP Connection(13)-X.X.X.XXX, state: WAITING, total cpu time: 1.135ms, total user time: 0.915ms
Thread: RMI TCP Connection(14)-X.X.X.XXX, state: WAITING, total cpu time: 1.356ms, total user time: 1.132ms
Thread: RMI TCP Connection(15)-X.X.X.XXX, state: WAITING, total cpu time: 1.413ms, total user time: 1.165ms
Thread: RMI TCP Connection(16)-X.X.X.XXX, state: WAITING, total cpu time: 1.786ms, total user time: 1.473ms
Thread: RMI TCP Connection(17)-X.X.X.XXX, state: WAITING, total cpu time: 1.257ms, total user time: 1.101ms
Thread: RMI TCP Connection(18)-X.X.X.XXX, state: WAITING, total cpu time: 1.374ms, total user time: 1.151ms
Thread: RMI TCP Connection(19)-X.X.X.XXX, state: WAITING, total cpu time: 1.423ms, total user time: 1.183ms
Thread: RMI TCP Connection(20)-X.X.X.XXX, state: WAITING, total cpu time: 1.211ms, total user time: 1.074ms
Thread: RMI TCP Connection(21)-X.X.X.XXX, state: WAITING, total cpu time: 1.374ms, total user time: 1.151ms
Thread: RMI TCP Connection(22)-X.X.X.XXX, state: WAITING, total cpu time: 1.044ms, total user time: 0.856ms
Thread: RMI TCP Connection(23)-X.X.X.XXX, state: WAITING, total cpu time: 1.362ms, total user time: 1.137ms
Thread: RMI TCP Connection(24)-X.X.X.XXX, state: WAITING, total cpu time: 1.351ms, total user time: 1.13ms
Thread: RMI TCP Connection(25)-X.X.X.XXX, state: WAITING, total cpu time: 1.502ms, total user time: 1.262ms
Thread: RMI TCP Connection(26)-X.X.X.XXX, state: WAITING, total cpu time: 1.233ms, total user time: 1.065ms
Thread: RMI TCP Connection(27)-X.X.X.XXX, state: WAITING, total cpu time: 1.391ms, total user time: 1.199ms
Thread: RMI TCP Connection(28)-X.X.X.XXX, state: WAITING, total cpu time: 1.773ms, total user time: 1.537ms
Thread: RMI TCP Connection(29)-X.X.X.XXX, state: WAITING, total cpu time: 1.332ms, total user time: 1.109ms
Thread: RMI TCP Connection(30)-X.X.X.XXX, state: WAITING, total cpu time: 1.366ms, total user time: 1.11ms
Thread: RMI TCP Connection(31)-X.X.X.XXX, state: WAITING, total cpu time: 1.351ms, total user time: 1.126ms
Thread: RMI TCP Connection(32)-X.X.X.XXX, state: WAITING, total cpu time: 1.33ms, total user time: 1.111ms
Thread: RMI TCP Connection(33)-X.X.X.XXX, state: WAITING, total cpu time: 1.433ms, total user time: 1.193ms
Thread: RMI TCP Connection(34)-X.X.X.XXX, state: WAITING, total cpu time: 1.407ms, total user time: 1.157ms
Thread: RMI TCP Connection(35)-X.X.X.XXX, state: WAITING, total cpu time: 1.365ms, total user time: 1.121ms
Thread: RMI TCP Connection(36)-X.X.X.XXX, state: WAITING, total cpu time: 1.386ms, total user time: 1.154ms
Thread: RMI TCP Connection(37)-X.X.X.XXX, state: WAITING, total cpu time: 1.384ms, total user time: 1.151ms
Thread: RMI TCP Connection(38)-X.X.X.XXX, state: WAITING, total cpu time: 1.342ms, total user time: 1.107ms
Thread: RMI TCP Connection(39)-X.X.X.XXX, state: WAITING, total cpu time: 1.401ms, total user time: 1.149ms
Thread: RMI TCP Connection(40)-X.X.X.XXX, state: WAITING, total cpu time: 1.129ms, total user time: 0.93ms
Thread: RMI TCP Connection(41)-X.X.X.XXX, state: WAITING, total cpu time: 1.571ms, total user time: 1.326ms
Thread: RMI TCP Connection(42)-X.X.X.XXX, state: WAITING, total cpu time: 1.722ms, total user time: 1.373ms
Thread: RMI TCP Connection(43)-X.X.X.XXX, state: WAITING, total cpu time: 1.56ms, total user time: 1.25ms
Thread: RMI TCP Connection(44)-X.X.X.XXX, state: WAITING, total cpu time: 1.345ms, total user time: 1.122ms
Thread: RMI TCP Connection(45)-X.X.X.XXX, state: WAITING, total cpu time: 1.35ms, total user time: 1.109ms
Thread: RMI TCP Connection(46)-X.X.X.XXX, state: WAITING, total cpu time: 1.293ms, total user time: 1.099ms
Thread: RMI TCP Connection(47)-X.X.X.XXX, state: WAITING, total cpu time: 1.355ms, total user time: 1.114ms
Thread: RMI TCP Connection(48)-X.X.X.XXX, state: WAITING, total cpu time: 1.402ms, total user time: 1.173ms
Thread: RMI TCP Connection(49)-X.X.X.XXX, state: WAITING, total cpu time: 1.36ms, total user time: 1.108ms
Thread: RMI TCP Connection(50)-X.X.X.XXX, state: WAITING, total cpu time: 1.355ms, total user time: 1.13ms
Thread: RMI TCP Connection(51)-X.X.X.XXX, state: WAITING, total cpu time: 1.367ms, total user time: 1.132ms
Thread: RMI TCP Connection(52)-X.X.X.XXX, state: WAITING, total cpu time: 1.805ms, total user time: 1.451ms
Thread: RMI TCP Connection(53)-X.X.X.XXX, state: WAITING, total cpu time: 1.363ms, total user time: 1.116ms
Thread: RMI TCP Connection(54)-X.X.X.XXX, state: WAITING, total cpu time: 1.457ms, total user time: 1.174ms
Thread: RMI TCP Connection(55)-X.X.X.XXX, state: WAITING, total cpu time: 1.158ms, total user time: 0.899ms
Thread: RMI TCP Connection(56)-X.X.X.XXX, state: WAITING, total cpu time: 1.354ms, total user time: 1.099ms
Thread: RMI TCP Connection(57)-X.X.X.XXX, state: WAITING, total cpu time: 1.301ms, total user time: 1.072ms
Thread: RMI TCP Connection(58)-X.X.X.XXX, state: WAITING, total cpu time: 1.407ms, total user time: 1.16ms
Thread: RMI TCP Connection(59)-X.X.X.XXX, state: WAITING, total cpu time: 1.315ms, total user time: 1.08ms
Thread: RMI TCP Connection(60)-X.X.X.XXX, state: WAITING, total cpu time: 1.103ms, total user time: 0.881ms
Thread: RMI TCP Connection(61)-X.X.X.XXX, state: WAITING, total cpu time: 1.312ms, total user time: 1.075ms
Thread: RMI TCP Connection(62)-X.X.X.XXX, state: WAITING, total cpu time: 1.404ms, total user time: 1.09ms
Thread: RMI TCP Connection(63)-X.X.X.XXX, state: WAITING, total cpu time: 1.37ms, total user time: 1.152ms
Thread: RMI TCP Connection(64)-X.X.X.XXX, state: WAITING, total cpu time: 1.434ms, total user time: 1.187ms
Thread: RMI TCP Connection(65)-X.X.X.XXX, state: WAITING, total cpu time: 1.206ms, total user time: 1.004ms
Thread: RMI TCP Connection(66)-X.X.X.XXX, state: WAITING, total cpu time: 0.786ms, total user time: 0.664ms
Thread: RMI TCP Connection(67)-X.X.X.XXX, state: WAITING, total cpu time: 1.279ms, total user time: 1.052ms
Thread: RMI TCP Connection(68)-X.X.X.XXX, state: WAITING, total cpu time: 0.929ms, total user time: 0.766ms
Thread: RMI TCP Connection(69)-X.X.X.XXX, state: WAITING, total cpu time: 1.308ms, total user time: 1.065ms
Thread: RMI TCP Connection(70)-X.X.X.XXX, state: WAITING, total cpu time: 1.331ms, total user time: 1.088ms
Thread: RMI TCP Connection(71)-X.X.X.XXX, state: WAITING, total cpu time: 1.183ms, total user time: 0.948ms
Thread: RMI TCP Connection(72)-X.X.X.XXX, state: WAITING, total cpu time: 1.278ms, total user time: 1.052ms
Thread: RMI TCP Connection(73)-X.X.X.XXX, state: WAITING, total cpu time: 1.687ms, total user time: 1.366ms
Thread: RMI TCP Connection(74)-X.X.X.XXX, state: WAITING, total cpu time: 1.33ms, total user time: 1.072ms
Thread: RMI TCP Connection(75)-X.X.X.XXX, state: WAITING, total cpu time: 1.273ms, total user time: 1.046ms
Thread: RMI TCP Connection(76)-X.X.X.XXX, state: WAITING, total cpu time: 1.528ms, total user time: 1.189ms
Thread: RMI TCP Connection(77)-X.X.X.XXX, state: WAITING, total cpu time: 1.364ms, total user time: 1.103ms
Thread: RMI TCP Connection(78)-X.X.X.XXX, state: WAITING, total cpu time: 1.292ms, total user time: 1.058ms
Thread: RMI TCP Connection(79)-X.X.X.XXX, state: WAITING, total cpu time: 1.289ms, total user time: 1.055ms
Thread: RMI TCP Connection(80)-X.X.X.XXX, state: WAITING, total cpu time: 1.188ms, total user time: 0.97ms
Thread: RMI TCP Connection(81)-X.X.X.XXX, state: WAITING, total cpu time: 1.26ms, total user time: 1.03ms
Thread: RMI TCP Connection(82)-X.X.X.XXX, state: WAITING, total cpu time: 1.372ms, total user time: 1.137ms
Thread: RMI TCP Connection(83)-X.X.X.XXX, state: WAITING, total cpu time: 1.279ms, total user time: 1.052ms
Thread: RMI TCP Connection(84)-X.X.X.XXX, state: WAITING, total cpu time: 1.304ms, total user time: 1.064ms
Thread: RMI TCP Connection(85)-X.X.X.XXX, state: WAITING, total cpu time: 1.288ms, total user time: 1.06ms
Thread: RMI TCP Connection(86)-X.X.X.XXX, state: WAITING, total cpu time: 1.417ms, total user time: 1.166ms
Thread: RMI TCP Connection(87)-X.X.X.XXX, state: WAITING, total cpu time: 1.19ms, total user time: 1.012ms
Thread: RMI TCP Connection(88)-X.X.X.XXX, state: WAITING, total cpu time: 1.451ms, total user time: 1.179ms
Thread: RMI TCP Connection(89)-X.X.X.XXX, state: WAITING, total cpu time: 1.265ms, total user time: 1.024ms
Thread: RMI TCP Connection(90)-X.X.X.XXX, state: WAITING, total cpu time: 1.343ms, total user time: 1.113ms
Thread: RMI TCP Connection(91)-X.X.X.XXX, state: WAITING, total cpu time: 1.282ms, total user time: 1.051ms
Thread: RMI TCP Connection(92)-X.X.X.XXX, state: WAITING, total cpu time: 1.264ms, total user time: 1.041ms
Thread: RMI TCP Connection(93)-X.X.X.XXX, state: WAITING, total cpu time: 1.295ms, total user time: 1.065ms
Thread: RMI TCP Connection(94)-X.X.X.XXX, state: WAITING, total cpu time: 1.351ms, total user time: 1.114ms
Thread: RMI TCP Connection(95)-X.X.X.XXX, state: WAITING, total cpu time: 1.261ms, total user time: 1.023ms
Thread: RMI TCP Connection(96)-X.X.X.XXX, state: WAITING, total cpu time: 1.196ms, total user time: 0.996ms
Thread: RMI TCP Connection(97)-X.X.X.XXX, state: WAITING, total cpu time: 1.247ms, total user time: 1.042ms
Thread: RMI TCP Connection(98)-X.X.X.XXX, state: WAITING, total cpu time: 1.218ms, total user time: 1.019ms
Thread: RMI TCP Connection(103)-X.X.X.XXX, state: WAITING, total cpu time: 1.278ms, total user time: 1.07ms
Thread: RMI TCP Connection(105)-X.X.X.XXX, state: WAITING, total cpu time: 1.162ms, total user time: 0.964ms
Thread: RMI TCP Connection(108)-X.X.X.XXX, state: WAITING, total cpu time: 1.368ms, total user time: 1.074ms
Thread: RMI TCP Connection(111)-X.X.X.XXX, state: WAITING, total cpu time: 1.154ms, total user time: 0.949ms
Thread: RMI TCP Connection(112)-X.X.X.XXX, state: WAITING, total cpu time: 1.283ms, total user time: 1.06ms
Thread: RMI TCP Connection(113)-X.X.X.XXX, state: WAITING, total cpu time: 1.261ms, total user time: 1.034ms

swingman · December 11, 2017, 11:42am

These connections only get cleaned up once I quit the Smart Client. Should not a garbage collector get rid of stale connections?

jcompagner · December 13, 2017, 11:11am

all those TCP connections are connections from/to a client, they will only be cleared up when the client releases a connections
This is all java RMI core stuff that Servoy can’t do much about.

swingman · December 13, 2017, 10:45pm

Thanks Johan, I was suspecting there is something outside Servoy causing this.
This issue has given me big stability problems – I had to restart the Servoy server daily, otherwise I would run out of threads and it would restart on its own.

I upgraded the Server tonight from Mac OS X 10.11.x to 10.12.6 (Sierra) and Servoy seems to behave much better, making fewer RMI connections. The Servoy server configuration is identical and the Java version identical. The real test will come tomorrow when the server is under load, but it looks like there must have been some issue with 10.11.x.

This solution has an on_exit script asking if the user really wants to Quit… Should I remove this to ensure Servoy can be shut down remotely and free up RMI connections?

swingman · December 20, 2017, 9:16am

Is there anything I can add in the configuration either of the smart client or the server to get these unused connections Garbage collected?
My 8.2.x server does not seem to do it on it’s own…

jcompagner · December 20, 2017, 9:31am

no you can’t there are completely managed by rmi and after that java itself.
The only thing you can do is disconnect a smart client…

swingman · December 20, 2017, 11:21am

OK. What I don’t understand is why this was not a problem with Servoy 7.4.x. It has only become a problem after upgrading to 8.2.x.
I’m going to let the server run now – to see if it still shuts itself down. I’m Java 9.0.1 now ( Have tried various Java 1.8.0s)

I restarted it manually while on 1900+ threads yesterday – not wanting the users to lose work.
I did add a

-XX:+UseG1GC

argument to the wrapper to see if it would do anything. Still under 400 threads after 12 hours.

jcompagner · December 20, 2017, 12:33pm

Garbage collection doesn’t have anything todo with Threads.
Threads needs to be terminated by who ever started the thread (so RMI in this case)

Also that UseG1GC just sets that G1 Garbage Collector (i think that is default on java 9), that is just how memory is used and how it is cleaned up of memory is released.

That has nothing todo with the RMI threads.

Are you use it is just about the rmi threads? Is the memory just not getting full and does it bang out with out of mem?

Then we need a memory dump (can be done with JVisualVM or adding stuff like:
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=“c:\temp\dump2.hprof”

to the vm arguments.

Maybe in a memory dump we can see what is really holding on to stuff.

swingman · December 20, 2017, 12:41pm

Thanks Johan, I will add these the next time I have to restart.

jcompagner · December 20, 2017, 12:52pm

i wonder if this is also a problem with 812 instead of 822 …

jcompagner · December 20, 2017, 2:22pm

i tried to reproduce this locally by quickly starting up a application server
then starting a few smart clients and shut them down again.
then i see a few of those RMI Connection threads but those are gone quite quickly

RMI Renew threads will stick around for a while, but also those will be gone for those specific clients eventually completely…

swingman · December 21, 2017, 10:19am

The problem is with clients left open. Once clients are quit, it looks to me like the threads are cleared up.
If I start a Smart client in the evening, it will have built up say like 400 RMI threads by the next afternoon, and if I quit the Smart Client, these server threads get cleaned out.

The users sometimes leave their Smart Client open for days – and I have not had any problems with this in the past on Servoy 7.x.

jcompagner · December 21, 2017, 10:32am

what is the rmi tunnel that you use? 2way socket? or http or http&socket?

swingman · December 21, 2017, 10:55am

http&socket. I tried http when we had the network issues earlier, but it did not help under the difficult networking conditions.

jcompagner · December 21, 2017, 11:15am

if you want to try really something else then you have to use 2waysocket (the rest are the same thing in a bit different configurations)

the thing is that rmi connections that are idle, so not used anymore should be removed by the “rmi.connection.timeout” setting we have on the admin page.
thats default i think 120 seconds but even if i lower that to 12 seconds it doesn’t seem to help for me locally

In 7.4 you did use also Http&Socket or just Socket?

swingman · December 21, 2017, 11:38am

From the 7.4.x servoy.properties.

SocketFactory.tunnelConnectionMode=http&socket
rmi.connection.timeout=120

Same settings as for 8.2.2. I tried shortening the RMI timeout to 15 and even set it to 0 during our network hell a few weeks back. It had no effect.
I have not tried 2waysocket, I was a bit concerned reading the documentation about having to manually switch all the clients back if it did not work…

jcompagner · December 21, 2017, 4:48pm

ok i think i finally found it…

Can you update/overwrite the rmitnl.jar from you 822 install?

It seems that the rmi sockets didn’t close correctly and because of that keep hanging around.

rmitnl.jar (135 KB)

swingman · December 21, 2017, 6:32pm

Thanks Johan, I will upload the JAR later tonight and leave a client open overnight…

swingman · December 21, 2017, 10:51pm

I have uploaded the jar and I’m pretty sure it has fixed the problem:
I now only have one RMI TCP Connection thread relating to my inactive client.

Thread: RMI TCP Connection(5)-XXX.XXX.XXX.164, state: WAITING, total cpu time: 11.254ms, total user time: 10.039ms

and a few minutes later the old thread is gone and I have a new one!

Thread: RMI TCP Connection(6)-XXX.XXX.XXX.164, state: WAITING, total cpu time: 1.311ms, total user time: 1.099ms

Topic		Replies	Views
Network errors, Smart Client Classic Servoy	8	6799	December 7, 2017
Server stops Classic Servoy	41	26243	January 16, 2012
Tomcat Server Rejecting RMI Connections Classic Servoy	7	6315	February 6, 2014
Server Intermittent Stalls Classic Servoy	25	14516	August 5, 2011
Windows Client Disconnecting Classic Servoy	16	14049	April 26, 2011

How many threads does Servoy 8.2.x Server need?

Related topics