java.lang.OutOfMemoryError: unable to create new native thre

[aliqazi01]

Hi All,

some of our customers some times experience this error "java.lang.OutOfMemoryError: unable to create new native thread".
this is the service log and their service is getting restarted as a result of this error and when the restart happens, it causes issues to them as the database might not be avalable at that time or can't load a specific library:

Code: Select all

   Time   Thread   Level   Category   Message   ClientId   Solution Name
2016-08-25 14:59   pool-2-thread-2   ERROR   com.servoy.j2db.util.Debug   client_stats table missing in log server, cannot save client statistics       
2016-08-25 14:59   ContainerBackgroundProcessor[StandardEngine[Catalina]]   ERROR   com.servoy.j2db.util.Debug   Error occured unloading client plugin: UserManager       
java.lang.OutOfMemoryError: unable to create new native thread
       at java.lang.Thread.start0(Native Method)
       at java.lang.Thread.start(Unknown Source)
       at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source)
       at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
       at com.servoy.j2db.util.ThreadingRemoteInvocationHandler.invoke(ThreadingRemoteInvocationHandler.java:97)
       at com.sun.proxy.$Proxy3.unregisterClientInfo(Unknown Source)
       at com.drmaison.plugin.usermanager.UserManagerProvider.unregisterClientOnServer(UserManagerProvider.java:2413)
       at com.drmaison.plugin.usermanager.UserManager.unload(UserManager.java:68)
       at com.servoy.j2db.plugins.PluginManager.flushCachedItems(PluginManager.java:137)
       at com.servoy.j2db.ClientState.shutDown(ClientState.java:1084)
       at com.servoy.j2db.server.headlessclient.SessionClient.shutDown(SessionClient.java:390)
       at com.servoy.j2db.server.headlessclient.WebClient.shutDown(WebClient.java:681)
       at com.servoy.j2db.server.headlessclient.SessionClient.valueUnbound(SessionClient.java:549)
       at org.apache.catalina.session.StandardSession.removeAttributeInternal(StandardSession.java:1710)
       at org.apache.catalina.session.StandardSession.expire(StandardSession.java:812)
       at org.apache.catalina.session.StandardSession.isValid(StandardSession.java:587)
       at org.apache.catalina.session.ManagerBase.processExpires(ManagerBase.java:737)
       at org.apache.catalina.session.ManagerBase.backgroundProcess(ManagerBase.java:722)
       at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1368)
       at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1653)
       at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1662)
       at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1662)
       at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1642)
       at java.lang.Thread.run(Unknown Source)


Has anyone else seen this and are there any tips on what might be causing this and how to resolve it?

[Andrei Costescu]

You are probably hitting the OS limit for native threads. That is usually a lot of threads.
Why do you have that many threads in the first place? Who creates them? The app. server, some plugin, ...?

If you don't know the answer to that and want to find out - there are 3 ways I can think of now:
- if you are on Linux add -XX:OnOutOfMemoryError="kill -3 %p" to the start arguments; that should generate a stack dump in the console output (if you can access/see it) when you get that OOME; alternatively you could use I think something like -XX:OnOutOfMemoryError="jstack %p > outfile" where you can specify a file where to dump the stack (replace that 'outfile' with what the file you want);
- you could add -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=../logs to get a full heap dump when it crashes and then analyse the threads in that using jvisualvm or jconsole (it seems a bit of overkill though)
- something might be doable in onSolutionOpen using Rhino to attach a default uncaught exception handler for OOM to the JVM - if it's really needed but then we'd have to convert some java code into Rhino-java syntax.

[Andrei Costescu]

About the stack trace above... So a web client is shut down (session expires), plugin User Manager from drmaison tries to unregister the client and that ends up trying to start a thread...
But that doesn't mean much. That call is not necessarily the one leaking threads. It's the one that reaches the limit though.