We recently moved our SaaS solution to 6.0.4 from 4.1.7 and for the most part, it’s gone ok.
The one exception has been the smartclient ( although we’ve been able to also see this in the webclient… but all of our customers are still on the smartclient) hanging periodically.
Basically, the solution will hang / freeze and not fire any errors on the Server or in the console. Sometimes it will hang for up to 5 minutes, which obviously has our customers crying bloody murder.
After searching up and down and all around, we came across a few posts about the client hanging and a recommendation was to provide a stack dump.
Here’s one we got off a client after it hung and then broke free.
Dump thread stack ...
----------------------------------------------------
2012-01-30 11:19:20
Full thread dump Java HotSpot(TM) Client VM (17.0-b17 mixed mode, sharing):
"pool-1-thread-30" prio=6 tid=0x0812f400 nid=0x9c8 waiting on condition [0x0898f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x294c1e28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"pool-1-thread-29" prio=6 tid=0x08130800 nid=0x13b8 waiting on condition [0x06bff000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x294c1e28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Thread-121" prio=6 tid=0x0812d000 nid=0x141c runnable [0x05d3f000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked <0x2dced4b8> (a java.lang.Object)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked <0x2dced4d0> (a com.sun.net.ssl.internal.ssl.AppInputStream)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked <0x2dced4d0> (a com.sun.net.ssl.internal.ssl.AppInputStream)
at com.servoy.j2db.rmi.compressing.CompressingInputStream.readIn(CompressingInputStream.java:99)
at com.servoy.j2db.rmi.compressing.CompressingInputStream.read(CompressingInputStream.java:85)
- locked <0x2dced4e8> (a com.servoy.j2db.rmi.compressing.CompressingInputStream)
at com.servoy.j2db.rmi.SocketAdapter$StreamThread.run(SocketAdapter.java:37)
"Thread-120" prio=6 tid=0x0812e800 nid=0xb24 runnable [0x057cf000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at com.servoy.j2db.rmi.SocketAdapter$StreamThread.run(SocketAdapter.java:37)
"RMI TCP Connection(53)-192.168.1.43" daemon prio=6 tid=0x0812e400 nid=0xc84 runnable [0x05cef000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked <0x2dcef678> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"Lookup ValueList Timer" daemon prio=6 tid=0x05addc00 nid=0x17ec in Object.wait() [0x065ef000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x2a135d70> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:485)
at java.util.TimerThread.mainLoop(Unknown Source)
- locked <0x2a135d70> (a java.util.TaskQueue)
at java.util.TimerThread.run(Unknown Source)
"TimerQueue" daemon prio=6 tid=0x05adf800 nid=0x478 in Object.wait() [0x05d8f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at javax.swing.TimerQueue.run(Unknown Source)
- locked <0x294c1e18> (a javax.swing.TimerQueue)
at java.lang.Thread.run(Unknown Source)
"RMI Reaper" prio=6 tid=0x05ae3c00 nid=0x113c in Object.wait() [0x06baf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x294c1f88> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
- locked <0x294c1f88> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at sun.rmi.transport.ObjectTable$Reaper.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"RMI TCP Accept-3037" daemon prio=6 tid=0x05ae3400 nid=0x123c runnable [0x06a5f000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(Unknown Source)
- locked <0x294c2048> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(Unknown Source)
at java.net.ServerSocket.accept(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"GC Daemon" daemon prio=2 tid=0x05ae3000 nid=0x1d80 in Object.wait() [0x06a0f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at sun.misc.GC$Daemon.run(Unknown Source)
- locked <0x28eeaae0> (a sun.misc.GC$LatencyLock)
"RMI RenewClean-[ch.us1.paygosaas.com:1099]" daemon prio=6 tid=0x05ae2800 nid=0x1428 in Object.wait() [0x066af000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
- locked <0x28eea7d8> (a java.lang.ref.ReferenceQueue$Lock)
at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"RMI Scheduler(0)" daemon prio=6 tid=0x05ae2400 nid=0x1698 waiting on condition [0x0665f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x28eda2c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.DelayQueue.take(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
"SignallingChannel" prio=6 tid=0x05ae1000 nid=0x1f58 runnable [0x05ecf000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked <0x28eef0c8> (a java.lang.Object)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked <0x28eef158> (a com.sun.net.ssl.internal.ssl.AppInputStream)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked <0x28eef158> (a com.sun.net.ssl.internal.ssl.AppInputStream)
at com.servoy.j2db.rmi.compressing.CompressingInputStream.readIn(CompressingInputStream.java:99)
at com.servoy.j2db.rmi.compressing.CompressingInputStream.read(CompressingInputStream.java:85)
- locked <0x28eef180> (a com.servoy.j2db.rmi.compressing.CompressingInputStream)
at java.io.DataInputStream.readInt(Unknown Source)
at com.servoy.j2db.rmi.SignallingChannel.run(SignallingChannel.java:219)
"PingThread" prio=6 tid=0x05ae0400 nid=0xf30 in Object.wait() [0x05a0f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at com.servoy.j2db.rmi.SignallingChannel$1.run(SignallingChannel.java:67)
- locked <0x28eef230> (a com.servoy.j2db.rmi.SignallingChannel$1)
at java.lang.Thread.run(Unknown Source)
"AWT-EventQueue-0" prio=6 tid=0x05ae0000 nid=0x43c in Object.wait() [0x0619f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.awt.EventQueue.getNextEvent(Unknown Source)
- locked <0x28de2460> (a java.awt.EventQueue)
at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)
"D3D Screen Updater" daemon prio=8 tid=0x05adf400 nid=0xf14 in Object.wait() [0x05f1f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at sun.java2d.d3d.D3DScreenUpdateManager.run(Unknown Source)
- locked <0x28eefa98> (a java.lang.Object)
at java.lang.Thread.run(Unknown Source)
"AWT-EventQueue-1" prio=6 tid=0x05adec00 nid=0xb6c waiting on condition [0x05e2f000]
java.lang.Thread.State: RUNNABLE
at com.sun.deploy.util.ConsoleHelper.dumpAllStacksImpl(Native Met
Trace message truncated for length over 10K
I can only think the TIMED WAITING is some sort of clue, but how do we figure out what this means?
Our 6.0.4 Server has the same configuration we had with our 4.1.7 server including memory settings. Everything is the same but the solution was cruising in 4.1.7 and now we’re limping along.
Thanks!