I'm trying out purely for databroadcasting between two instances of servoy (not for load balancing), on a Debian Linux server with 12 GB of RAM
It works just fine for days at a time, but then the databroadcasting dies. Both servoy clients are alive and well, I can see them writing to MySQL, but each is unaware of the datachanges made by the other. Restarting servoy server without restarting Terracotta will often fix it.
What might cause this? I'm seeing some warnings in terracotta-server.log but these are happening many times a day without the databroadcasting falling over. e.g.
WARN tc.operator.event - NODE : localhost:9510 Subsystem: MEMORY_MANAGER Message: Current Memory usage(92%) crossed critical threshold(90%).
Databroadcasting died sometime between 16:50 and 17:00 in the logs below
terracotta-server.log
- Code: Select all
2012-11-12 16:23:49,973 [Statistics Logger] INFO com.terracottatech.dso - memory free : 221.216827 MB
2012-11-12 16:23:49,973 [Statistics Logger] INFO com.terracottatech.dso - memory used : 275.720673 MB
2012-11-12 16:23:49,973 [Statistics Logger] INFO com.terracottatech.dso - memory max : 496.937500 MB
2012-11-12 16:23:58,431 [Server Map Periodic Evictor] INFO tc.operator.event - NODE : localhost:9510 Subsystem: DCV2 Message: DCV2 Eviction - Time taken (msecs)=0, Number of entries evicted=0, Number of segments over threshold=0, Total Overshoot=0
2012-11-12 16:23:58,431 [Server Map Periodic Evictor] INFO com.tc.objectserver.impl.ServerMapEvictionStatsManager - Server Map Periodic eviction - Time taken (msecs): 0, Number of segments under threshold: 0, Number of segments over threshold: 0, Total overshoot: 0, Total number of samples requested: 0, Number of segments where eviction happened: 0, Total number of evicted entries: 0
2012-11-12 16:26:46,505 [WorkerThread(managed_object_request_stage, 0, 0)] INFO com.tc.objectserver.handler.ManagedObjectRequestHandler - Number of repeated/wasted lookups : 1600000
2012-11-12 16:26:46,507 [WorkerThread(managed_object_request_stage, 3, 3)] INFO com.tc.objectserver.handler.ManagedObjectRequestHandler - Number of repeated/wasted lookups : 1600000
2012-11-12 16:26:46,511 [WorkerThread(managed_object_request_stage, 3, 3)] INFO com.tc.objectserver.handler.ManagedObjectRequestHandler - Number of repeated/wasted lookups : 1600000
2012-11-12 16:38:49,972 [Statistics Logger] INFO com.terracottatech.dso - memory free : 164.553955 MB
2012-11-12 16:38:49,972 [Statistics Logger] INFO com.terracottatech.dso - memory used : 325.446045 MB
2012-11-12 16:38:49,972 [Statistics Logger] INFO com.terracottatech.dso - memory max : 490.000000 MB
2012-11-12 16:41:28,577 [TC Memory Monitor] WARN tc.operator.event - NODE : localhost:9510 Subsystem: MEMORY_MANAGER Message: Current Memory usage(92%) crossed critical threshold(90%).
2012-11-12 16:53:49,973 [Statistics Logger] INFO com.terracottatech.dso - memory free : 133.155716 MB
2012-11-12 16:53:49,973 [Statistics Logger] INFO com.terracottatech.dso - memory used : 355.031784 MB
2012-11-12 16:53:49,973 [Statistics Logger] INFO com.terracottatech.dso - memory max : 488.187500 MB
2012-11-12 16:53:58,430 [Server Map Periodic Evictor] INFO tc.operator.event - NODE : localhost:9510 Subsystem: DCV2 Message: DCV2 Eviction - Time taken (msecs)=0, Number of entries evicted=0, Number of segments over threshold=0, Total Overshoot=0
2012-11-12 16:53:58,431 [Server Map Periodic Evictor] INFO com.tc.objectserver.impl.ServerMapEvictionStatsManager - Server Map Periodic eviction - Time taken (msecs): 0, Number of segments under threshold: 0, Number of segments over threshold: 0, Total overshoot: 0, Total number of samples requested: 0, Number of segments where eviction happened: 0, Total number of evicted entries: 0
2012-11-12 17:08:37,952 [WorkerThread(garbage_collect_stage, 0)] INFO tc.operator.event - NODE : localhost:9510 Subsystem: DGC Message: DGC[ 102 ] started.
2012-11-12 17:08:49,972 [Statistics Logger] INFO com.terracottatech.dso - memory free : 117.682968 MB
2012-11-12 17:08:49,973 [Statistics Logger] INFO com.terracottatech.dso - memory used : 366.129532 MB
2012-11-12 17:08:49,973 [Statistics Logger] INFO com.terracottatech.dso - memory max : 483.812500 MB
2012-11-12 17:09:08,487 [WorkerThread(garbage_collect_stage, 0)] INFO com.tc.objectserver.dgc.impl.MarkAndSweepGarbageCollector - DGC[ 102 ] complete : GarbageCollectionInfo [ Iteration = 102 ] = type = FULL_GC startTime = 1352700517952 begin object count = 3916043 end object count = 307309 markStageTime = 28328 pauseStageTime = 242 totalMarkCycleTime = 30535 candidate garabage count = 3627675 actual garbage count = 3627675 pre rescue count = 3914713 rescue1Time = 27967 rescue 1 Count = 3627675 rescue2Time = 242
2012-11-12 17:09:14,200 [WorkerThread(gc_delete_from_disk_stage, 0)] INFO com.tc.objectserver.persistence.db.DBPersistorImpl - Removed 5000 objects in 771 ms.
2012-11-12 17:09:15,473 [WorkerThread(gc_delete_from_disk_stage, 0)] INFO com.tc.objectserver.persistence.db.DBPersistorImpl - Removed 5000 objects in 305 ms.
2012-11-12 17:09:21,551 [WorkerThread(gc_delete_from_disk_stage, 0)] INFO com.tc.objectserver.persistence.db.DBPersistorImpl - Removed 5000 objects in 314 ms.
2012-11-12 17:09:23,143 [WorkerThread(gc_delete_from_disk_stage, 0)] INFO com.tc.objectserver.persistence.db.DBPersistorImpl - Removed 5000 objects in 435 ms.
2012-11-12 17:09:38,316 [WorkerThread(gc_delete_from_disk_stage, 0)] INFO com.tc.objectserver.persistence.db.DBPersistorImpl - Removed 5000 objects in 325 ms.
2012-11-12 17:09:38,713 [WorkerThread(gc_delete_from_disk_stage, 0)] INFO com.tc.objectserver.persistence.db.DBPersistorImpl - Removed 5000 objects in 397 ms.
2012-11-12 17:09:49,340 [WorkerThread(gc_delete_from_disk_stage, 0)] INFO tc.operator.event - NODE : localhost:9510 Subsystem: DGC Message: DGC[ 102 ] finished. Begin Count : 3,916,043 Collected : 3,627,675 Time Taken : 71,388 ms Live Objects : 325,045
2012-11-12 17:11:28,578 [TC Memory Monitor] WARN tc.operator.event - NODE : localhost:9510 Subsystem: MEMORY_MANAGER Message: Current Memory usage(92%) crossed critical threshold(90%).
2012-11-12 17:23:49,972 [Statistics Logger] INFO com.terracottatech.dso - memory free : 248.999077 MB
2012-11-12 17:23:49,972 [Statistics Logger] INFO com.terracottatech.dso - memory used : 236.125923 MB
2012-11-12 17:23:49,972 [Statistics Logger] INFO com.terracottatech.dso - memory max : 485.125000 MB
2012-11-12 17:23:58,430 [Server Map Periodic Evictor] INFO tc.operator.event - NODE : localhost:9510 Subsystem: DCV2 Message: DCV2 Eviction - Time taken (msecs)=0, Number of entries evicted=0, Number of segments over threshold=0, Total Overshoot=0
2012-11-12 17:23:58,430 [Server Map Periodic Evictor] INFO com.tc.objectserver.impl.ServerMapEvictionStatsManager - Server Map Periodic eviction - Time taken (msecs): 0, Number of segments under threshold: 0, Number of segments over threshold: 0, Total overshoot: 0, Total number of samples requested: 0, Number of segments where eviction happened: 0, Total number of evicted entries: 0
Servoy32 terracotta-client.log
- Code: Select all
2012-11-12 16:38:59,577 [Statistics Logger] INFO com.terracottatech.dso - memory free : 190.859726 MB
2012-11-12 16:38:59,578 [Statistics Logger] INFO com.terracottatech.dso - memory used : 264.265274 MB
2012-11-12 16:38:59,578 [Statistics Logger] INFO com.terracottatech.dso - memory max : 455.125000 MB
2012-11-12 16:39:17,709 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 5 garbage locks
2012-11-12 16:40:17,709 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 2 garbage locks
2012-11-12 16:41:30,302 [pool-1699-thread-1] INFO com.tc.object.locks.ClientLockImpl - Retrying flush on DsoLockID(ObjectID=[87530579]) as flush level moved from READ to WRITE during flush operation
2012-11-12 16:41:30,302 [TaskExecuter[11]] INFO com.tc.object.locks.ClientLockImpl - Retrying flush on DsoLockID(ObjectID=[87530579]) as flush level moved from READ to WRITE during flush operation
2012-11-12 16:42:17,709 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 6 garbage locks
2012-11-12 16:45:17,710 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 6 garbage locks
2012-11-12 16:46:17,709 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 1 garbage locks
2012-11-12 16:47:17,710 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 6 garbage locks
2012-11-12 16:48:17,709 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 1 garbage locks
2012-11-12 16:50:17,709 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 3 garbage locks
2012-11-12 16:51:17,709 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 1 garbage locks
2012-11-12 16:53:37,217 [pool-1-thread-2] INFO com.tc.object.tx.RemoteTransactionManagerImpl - ClientID[0]: TransactionID=[-1] : Took more than 1000ms to add to sequencer : 1540 ms
2012-11-12 16:53:59,577 [Statistics Logger] INFO com.terracottatech.dso - memory free : 159.455223 MB
2012-11-12 16:53:59,577 [Statistics Logger] INFO com.terracottatech.dso - memory used : 315.857277 MB
2012-11-12 16:53:59,577 [Statistics Logger] INFO com.terracottatech.dso - memory max : 475.312500 MB
2012-11-12 16:55:17,710 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 3 garbage locks
2012-11-12 16:56:17,711 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 2 garbage locks
2012-11-12 16:57:17,711 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 1 garbage locks
2012-11-12 16:59:17,712 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 4 garbage locks
2012-11-12 17:02:00,757 [TaskExecuter[2]] INFO com.tc.object.locks.ClientLockImpl - Retrying flush on DsoLockID(ObjectID=[1001]) as flush level moved from READ to WRITE during flush operation
2012-11-12 17:02:00,758 [pool-1-thread-2] INFO com.tc.object.locks.ClientLockImpl - Retrying flush on DsoLockID(ObjectID=[1001]) as flush level moved from READ to WRITE during flush operation
2012-11-12 17:03:17,715 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 2 garbage locks
2012-11-12 17:04:17,713 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 12 garbage locks
2012-11-12 17:05:17,713 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 6 garbage locks
2012-11-12 17:06:17,713 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 1 garbage locks
2012-11-12 17:07:17,713 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 1 garbage locks
2012-11-12 17:08:17,714 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 1 garbage locks
2012-11-12 17:08:59,578 [Statistics Logger] INFO com.terracottatech.dso - memory free : 86.226685 MB
2012-11-12 17:08:59,578 [Statistics Logger] INFO com.terracottatech.dso - memory used : 401.648315 MB
2012-11-12 17:08:59,578 [Statistics Logger] INFO com.terracottatech.dso - memory max : 487.875000 MB
2012-11-12 17:10:17,713 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 1 garbage locks
2012-11-12 17:11:18,386 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 1 garbage locks
2012-11-12 17:12:08,301 [MessagePreHandler] INFO com.tc.object.tx.RemoteTransactionManagerImpl - ClientID[0]: TransactionID=[6574362] : Took more than 1000ms to add to sequencer : 1275 ms
2012-11-12 17:12:18,385 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 2 garbage locks
2012-11-12 17:13:18,386 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 7 garbage locks
2012-11-12 17:14:18,385 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 1 garbage locks
2012-11-12 17:15:18,386 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 4 garbage locks
2012-11-12 17:16:18,385 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 2 garbage locks
2012-11-12 17:17:18,386 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 6 garbage locks
2012-11-12 17:18:18,386 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 1 garbage locks
2012-11-12 17:19:18,386 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 1 garbage locks
2012-11-12 17:20:18,386 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 3 garbage locks
2012-11-12 17:21:18,386 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 4 garbage locks
2012-11-12 17:22:18,386 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 2 garbage locks
2012-11-12 17:23:18,387 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 4 garbage locks
2012-11-12 17:23:59,578 [Statistics Logger] INFO com.terracottatech.dso - memory free : 80.608398 MB
2012-11-12 17:23:59,578 [Statistics Logger] INFO com.terracottatech.dso - memory used : 392.141602 MB
2012-11-12 17:23:59,578 [Statistics Logger] INFO com.terracottatech.dso - memory max : 472.750000 MB
2012-11-12 17:24:18,387 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 3 garbage locks
2012-11-12 17:25:18,388 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 1 garbage locks
2012-11-12 17:26:18,389 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[0]: Lock GC collected 9 garbage locks
Servoy64 terracotta-client.log
- Code: Select all
2012-11-12 16:31:06,421 [Statistics Logger] INFO com.terracottatech.dso - memory free : 1585.806114 MB
2012-11-12 16:31:06,421 [Statistics Logger] INFO com.terracottatech.dso - memory used : 1466.756386 MB
2012-11-12 16:31:06,421 [Statistics Logger] INFO com.terracottatech.dso - memory max : 3052.562500 MB
2012-11-12 16:31:24,211 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 2 garbage locks
2012-11-12 16:32:24,211 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 2 garbage locks
2012-11-12 16:33:24,212 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 2 garbage locks
2012-11-12 16:34:24,213 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 4 garbage locks
2012-11-12 16:35:24,212 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 8 garbage locks
2012-11-12 16:36:24,213 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 2 garbage locks
2012-11-12 16:37:24,213 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 2 garbage locks
2012-11-12 16:38:24,214 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 3 garbage locks
2012-11-12 16:39:24,214 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 6 garbage locks
2012-11-12 16:40:24,213 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 3 garbage locks
2012-11-12 16:41:24,214 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 7 garbage locks
2012-11-12 16:41:30,335 [TaskExecuter[12]] INFO com.tc.object.locks.ClientLockImpl - Retrying flush on DsoLockID(ObjectID=[1001]) as flush level moved from READ to WRITE during flush operation
2012-11-12 16:41:30,335 [pool-1-thread-1] INFO com.tc.object.locks.ClientLockImpl - Retrying flush on DsoLockID(ObjectID=[1001]) as flush level moved from READ to WRITE during flush operation
2012-11-12 16:41:30,335 [pool-1-thread-2] INFO com.tc.object.locks.ClientLockImpl - Retrying flush on DsoLockID(ObjectID=[1001]) as flush level moved from READ to WRITE during flush operation
2012-11-12 16:41:30,337 [http-8080-12] INFO com.tc.object.locks.ClientLockImpl - Retrying flush on DsoLockID(ObjectID=[1001]) as flush level moved from READ to WRITE during flush operation
2012-11-12 16:42:24,213 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 4 garbage locks
2012-11-12 16:43:24,213 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 2 garbage locks
2012-11-12 16:44:24,214 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 4 garbage locks
2012-11-12 16:45:24,213 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 5 garbage locks
2012-11-12 16:46:06,422 [Statistics Logger] INFO com.terracottatech.dso - memory free : 1441.976227 MB
2012-11-12 16:46:06,422 [Statistics Logger] INFO com.terracottatech.dso - memory used : 1620.523773 MB
2012-11-12 16:46:06,422 [Statistics Logger] INFO com.terracottatech.dso - memory max : 3062.500000 MB
2012-11-12 16:46:24,214 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 2 garbage locks
2012-11-12 16:47:24,213 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 5 garbage locks
2012-11-12 16:48:24,213 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 5 garbage locks
2012-11-12 16:49:24,214 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 7 garbage locks
2012-11-12 16:50:24,213 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 1 garbage locks
2012-11-12 16:51:23,642 [http-8080-27] INFO com.tc.object.locks.ClientLockImpl - Retrying flush on DsoLockID(ObjectID=[85691698]) as flush level moved from READ to WRITE during flush operation
2012-11-12 16:51:24,214 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 3 garbage locks
2012-11-12 16:51:55,108 [http-8080-19] INFO com.tc.object.locks.ClientLockImpl - Retrying flush on DsoLockID(ObjectID=[83076985]) as flush level moved from READ to WRITE during flush operation
2012-11-12 16:51:55,164 [http-8080-19] INFO com.tc.object.locks.ClientLockImpl - Retrying flush on DsoLockID(ObjectID=[83076985]) as flush level moved from READ to WRITE during flush operation
2012-11-12 16:52:24,215 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 4 garbage locks
2012-11-12 16:53:24,216 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 2 garbage locks
2012-11-12 16:54:24,215 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 4 garbage locks
2012-11-12 16:55:24,216 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 3 garbage locks
2012-11-12 16:55:52,606 [MessagePreHandler] INFO com.tc.object.locks.ClientLockImpl - Retrying flush on DsoLockID(ObjectID=[72638634]) as flush level moved from READ to WRITE during flush operation
2012-11-12 16:56:24,215 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 2 garbage locks
2012-11-12 16:57:24,216 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 4 garbage locks
2012-11-12 16:58:24,215 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 4 garbage locks
2012-11-12 16:59:24,216 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 4 garbage locks
2012-11-12 17:00:24,217 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 7 garbage locks
2012-11-12 17:00:44,611 [http-8080-1] INFO com.tc.object.locks.ClientLockImpl - Retrying flush on DsoLockID(ObjectID=[88353517]) as flush level moved from READ to WRITE during flush operation
2012-11-12 17:01:06,421 [Statistics Logger] INFO com.terracottatech.dso - memory free : 1870.187050 MB
2012-11-12 17:01:06,421 [Statistics Logger] INFO com.terracottatech.dso - memory used : 1190.812950 MB
2012-11-12 17:01:06,421 [Statistics Logger] INFO com.terracottatech.dso - memory max : 3061.000000 MB
2012-11-12 17:01:24,168 [http-8080-7] INFO com.tc.object.locks.ClientLockImpl - Retrying flush on DsoLockID(ObjectID=[88248877]) as flush level moved from READ to WRITE during flush operation
2012-11-12 17:01:24,216 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 7 garbage locks
2012-11-12 17:02:24,217 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 3 garbage locks
2012-11-12 17:03:24,216 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 4 garbage locks
2012-11-12 17:04:24,216 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 5 garbage locks
2012-11-12 17:05:24,217 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 4 garbage locks
2012-11-12 17:06:24,217 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 2 garbage locks
2012-11-12 17:07:24,216 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 2 garbage locks
2012-11-12 17:07:37,022 [http-8080-29] INFO com.tc.object.locks.ClientLockImpl - Retrying flush on DsoLockID(ObjectID=[86962721]) as flush level moved from READ to WRITE during flush operation
2012-11-12 17:08:24,217 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 3 garbage locks
2012-11-12 17:09:24,217 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 6 garbage locks
2012-11-12 17:10:24,216 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 2 garbage locks
2012-11-12 17:11:24,217 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 3 garbage locks
2012-11-12 17:12:24,218 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 5 garbage locks
2012-11-12 17:13:24,217 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 4 garbage locks
2012-11-12 17:14:24,218 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 5 garbage locks
2012-11-12 17:15:24,217 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 4 garbage locks
2012-11-12 17:15:55,061 [http-8080-26] INFO com.tc.object.locks.ClientLockImpl - Retrying flush on DsoLockID(ObjectID=[88727199]) as flush level moved from READ to WRITE during flush operation
2012-11-12 17:16:06,422 [Statistics Logger] INFO com.terracottatech.dso - memory free : 1339.809471 MB
2012-11-12 17:16:06,422 [Statistics Logger] INFO com.terracottatech.dso - memory used : 1689.878029 MB
2012-11-12 17:16:06,422 [Statistics Logger] INFO com.terracottatech.dso - memory max : 3029.687500 MB
2012-11-12 17:16:24,223 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 5 garbage locks
2012-11-12 17:18:24,223 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 3 garbage locks
2012-11-12 17:19:24,224 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 5 garbage locks
2012-11-12 17:20:24,223 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 4 garbage locks
2012-11-12 17:21:24,224 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 3 garbage locks
2012-11-12 17:22:24,223 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 4 garbage locks
2012-11-12 17:23:24,224 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 4 garbage locks
2012-11-12 17:24:24,225 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 7 garbage locks
2012-11-12 17:24:44,837 [http-8080-4] INFO com.tc.object.locks.ClientLockImpl - Retrying flush on DsoLockID(ObjectID=[88805220]) as flush level moved from READ to WRITE during flush operation
2012-11-12 17:25:24,224 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 2 garbage locks
2012-11-12 17:26:24,224 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 3 garbage locks
2012-11-12 17:27:24,224 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 2 garbage locks
2012-11-12 17:28:24,224 [ClientLockManager LockGC] INFO com.tc.object.locks.ClientLockManager - ClientID[3]: Lock GC collected 2 garbage locks