Tomcat Server Rejecting RMI Connections

[david.pearce]

I am experiencing this more frequently.

The Servoy server (6.0.9) works perfectly, but new users cannot get in. They get the grey screen of death.

The specific RMI port rejects telnet connections, so clearly although everything else in Servoy is working fine, the RMI port on tomcat seizes up or stops listening.

Today this happened early on,after a DOS attack on our ISP. My theory is that some Servoy connections dont release the port and therefore maybe tomcat gets the maximum number of users on the RMI port.

Q. Is there a way to find or set the maximum number of users on the tomcat RMI port? i cannot see a setting easily in the server directory.
Q. Is there a way of reducing the timeout on this port, as maybe it is not terminating aborted connections for some reason.

The only method I have at present to allow new users in is to boot everyone out and restart the servoy server!

Thanks

David

[jcompagner]

What kind of tunnel configuration do you have?

2waysocket, Http&socket or just Socket?

for 2 way or socket the maximum number would be i guess the operating system maximum number of sockets that can be opened (most of the time called open file descriptors i think)

for http connections you need enough http/worker threads in tomcat to handle the all the clients (1 or 2 thread per client)
thats the maxThreads="500" property in the connecter configuration found in the server.xml file

How many users are we talking about here?

[david.pearce]

Http&Socket

We have up to 150 users on, but when this happened last there were only 60.

Those users who were on were fine.

No one could get into the solution, and just got the gray screen.

I telnetet to the RMI port 1101 on this installation and it was closed.

So it looks like Servoy is fine, except tomcat is no longer accepting connections on the RMI port when this happens.

It occurs nearly once per day (we do restart at 4am), it may be annecdotal, but when it happened on monday it was immediately after a DOS attack on our hosting company, so I have a hunch that maybe tomcat isnt properly releasing connections on the RMI port.

There is no setting for threads to the RMI port to increase them. I have reduced the timeout from 120 to 30 in the servoy settings, but it still happened yesterday morning.

One last thing, is i have seen this before where you cant get in for a minute or two and then it goes from gray to the solution chooser. That i suspect is the RMI threads filling up but then being released at 120 seconds.

My hunch is we need to somehow monitor, measure or increase the number of threads on the RMI port, but there is no setting in servoy to do this!

[jcompagner]

tomcat is not doing anything for port 1101 thats just an rmi server connection port of the Socket tunnel. (its not something tomcat handles at all)

Besides that it shouldn't really matter except if your os really runs out of native resources for that.

RMI just makes more threads i think, i can't even see a property for that at all:

http://docs.oracle.com/javase/7/docs/te ... rties.html

besides that if thread creation was a problem then the java vm would report that.

i do wonder if telnet to that port really goes into java, or if the os is already just giving it up.

[jcompagner]

don't you have anything in the log? Like that there is an accept thread failure or exception?
Because that would mean that the accept thread is just gone and yes then nothing can connect to it anymore.

also if you really just then close a few clients then still no other clients can connect? So freeing up connections doesn't make it suddenly accept a few again?

[david.pearce]

We will get a snapshot of the log next time. I dont think there is anything in it, but will double check when it next falls over.

Closing clients doesnt appear to help. Also i have tried to forceably kick poeple out and sometimes they appear to hang and not go, but thats not specific to this problem.

Next time, I will get a copy of the log and also try kicking everyone out, before i restart and seeing if that then opens the port again, before i actually restart.

David

[jcompagner]

also if that happens make a thread dump on the admin page.
Maybe we can see something there.

[david.pearce]

Thanks,

the dump from the the client (i think!!!) is on this post

viewtopic.php?f=5&t=20109

interestingly i did o a netsat which showed 102 thread some open some close on the 1101 port at that time.

Next time I will definitely dump from the servoy admin page.

David