Application server keeps freezing

Questions and Answers on installation, deployment, management, locking, tranasactions of Servoy Application Server

Application server keeps freezing

Postby Janssenjos » Tue Nov 14, 2017 10:26 am

We have mayor problems of an live server at one of our clients keep freezing.
These problems kinda started out of the blue. We didn't have any updates recently, so we can't pinpoint it down to something we changed that would be causing this.

It looks like Clients are not being cleaned up properly. We see Clients keep increasing during time. (But we have no idea if this is the problem)
We use webclients and headless client as rest service. (Both stateless rest as with the itech plugin with state)
When it freezes, the Servoy Server Home isn't accessible anymore. (Other pages still are.)

What we see at the times it happens, the active connections of the repository database server is full. (We have increased it a couple of times, now it is 50/50 for example) And eventually other databaseserver connections also get exhausted.
From that point on, no new users can login.
But we have no idea what is going on at that time that it freezes.


Part of dumpstack, rest of it is in the attachment.
Code: Select all
Current time: Mon Nov 13 23:30:09 CET 2017

Server Information
Servoy version 6.0.9 -build 1239
Port used by RMI Registry: 1099
Repository version 40

Current time: Mon Nov 13 23:30:09 CET 2017

Uptime:  39 minutes 24 seconds

JDK Information
java.vm.name=Java HotSpot(TM) 64-Bit Server VM
java.version=1.7.0_17
java.vm.info=mixed mode
java.vm.vendor=Oracle Corporation

Operating System Information
os.name=Windows Server 2008 R2
os.version=6.1
os.arch=amd64

System Information
Heap Space Memory: allocated=3920320K, used=2554018K, max=13048960K
Non-Heap Space Memory: allocated=143168K, used=90559K, max=180224K
Attachments
Dumpstack 13-11 2300.txt
Dumpstack 2330_13-11
(403.42 KiB) Downloaded 263 times
Jos Janssen
Software Developer
Axerrio
http://www.axerrio.com
Janssenjos
 
Posts: 148
Joined: Thu Aug 13, 2009 3:55 pm
Location: Bergen op Zoom

Re: Application server keeps freezing

Postby jcompagner » Tue Nov 14, 2017 11:31 am

yes it is a problem of the connection pooling, there are many threads waiting to borrow a new connection from the pool but the pool is waiting for a connection to be released because it reaches the max.

There are only 2 clients really active and doing a query to the database (those have a connection)
And there are a few threads waiting for those 2 clients.

But the majority is just waiting for the a connection to be released,

So something is eating up repository server connections, the question is why is that suddenly happening? Or did the repository connections always grow slowly?
And now the server is way more busy?

Problem is this is quite an older servoy release, so purely for that we can't really fix anything. The question is more who is keeping a reference to all those connections. Why are they not returned to the pool
Maybe in a memory dump when the repository servers connections are going up again (so the active) we can see where they are hold on to.
Can you make a memory dump on that system for example with JVisualVM (should be installed right inside the JDK installation)

It is a bit weird that also the other database connections are hanging, because that would mean some nested calls even, are long transactions used?
Johan Compagner
Servoy
User avatar
jcompagner
 
Posts: 8828
Joined: Tue May 27, 2003 7:26 pm
Location: The Internet

Re: Application server keeps freezing

Postby Janssenjos » Tue Nov 14, 2017 12:53 pm

Can you make a memory dump on that system for example with JVisualVM (should be installed right inside the JDK installation)


We are running VisaulVm now, we will make one when it happens again.
I can make a heapdump and a threaddump. Are both needed?
Jos Janssen
Software Developer
Axerrio
http://www.axerrio.com
Janssenjos
 
Posts: 148
Joined: Thu Aug 13, 2009 3:55 pm
Location: Bergen op Zoom

Re: Application server keeps freezing

Postby Andrei Costescu » Tue Nov 14, 2017 1:38 pm

Make sure to create a private case containing the new dump(s) rather then attaching them here.
Hmm actually the heap dump will be quite large; I guess Johan will say how to send it over.
Andrei Costescu
Servoy
Andrei Costescu
 
Posts: 1018
Joined: Tue Jun 26, 2007 3:14 pm

Re: Application server keeps freezing

Postby jcompagner » Tue Nov 14, 2017 3:14 pm

heapdump is the thing that can hopefully tells us where those connections are hold on to (that could be tricky because that could be just the pool but it is just that close is not called on it...)

If we can't find that then it is really tricky to fix, because everywhere the connection could be used should be checked then if a close() call did happen on it.
I still don't understand why this suddenly happens looking at the servoy version that is a quite old server that is already running for years right?

a heap dump can be 7zipped and if it is then still to big, place it on a cloud drive and attach that url to a private case.
Johan Compagner
Servoy
User avatar
jcompagner
 
Posts: 8828
Joined: Tue May 27, 2003 7:26 pm
Location: The Internet

Re: Application server keeps freezing

Postby Janssenjos » Wed Nov 15, 2017 9:45 am

Apparently when you are expecting/waiting it to happen again, the server appears to runs smoothly, and survives the day and night. (While the last days it wouldn't even survive half a workday anymore)

Thanks for the help/thought so far.
We will add a support case with the dump when it happens again.
Jos Janssen
Software Developer
Axerrio
http://www.axerrio.com
Janssenjos
 
Posts: 148
Joined: Thu Aug 13, 2009 3:55 pm
Location: Bergen op Zoom

Re: Application server keeps freezing

Postby jcompagner » Wed Nov 15, 2017 10:43 am

did it happen right after a restart?
Johan Compagner
Servoy
User avatar
jcompagner
 
Posts: 8828
Joined: Tue May 27, 2003 7:26 pm
Location: The Internet


Return to Servoy Server

Who is online

Users browsing this forum: No registered users and 5 guests