Servoy memory usage question

[antonio]

Here's a fresh lot of data, all taken at ~7:23 definitely all from the same server (I only have 1 linux server running at the moment)

Code: Select all

             total       used       free     shared    buffers     cached
Mem:      12334848   12198620     136228          0     295340    2308332
-/+ buffers/cache:    9594948    2739900
Swap:      1949688      58576    1891112


htop

Screen Shot 2012-11-14 at 7.22.56 AM.png (286 KiB) Viewed 5153 times

[antonio]

VisualVM graphs at the same time, low load on the server, low memory use reported within Servoy, but almost no free RAM reported with free or htop.

Servoy32 PermGen Memory
Size: 111,017,984 B
Used: 90,002,912 B
Max: 268,435,456 B

Servoy64 PermGen Memory
Size: 108,199,936 B
Used: 108,195,728 B
Max: 268,435,456 B

TC PermGen Memory

Size: 38,928,384 B
Used: 37,193,592 B
Max: 85,983,232 B

and

JVM Information (Servoy64)
java.vm.name=Java HotSpot(TM) 64-Bit Server VM
java.version=1.6.0_26
java.vm.info=mixed mode
java.vm.vendor=Sun Microsystems Inc.

JVM Information (Servoy32)
java.vm.name=Java HotSpot(TM) Server VM
java.version=1.6.0_26
java.vm.info=mixed mode
java.vm.vendor=Sun Microsystems Inc.

[Andrei Costescu]

So it's basically the same. Used Java heap low, used OS memory very high. I'll need to run some stuff locally and see if I get the same but it really seems to be JVM (or Terracotta) related. It's like the JVM forgets to return memory back to the OS. I'm not sure that this causes the crashes though - if the Java heap can use this strange already allocated space when it needs to.

What version of Terracotta do you use? Any crash logs so that we can see which one crashed exactly and why?

It can be seen in the Terracotta server heap (that seems to be slowly but constantly increasing towards the max) that it didn't yet run the distributed garbage collector. I think by default, without any tuning it will run once a day. I mentioned in the other thread how this can be improved with the 2 settings.

[antonio]

Hello again Andrei,

From the log, its Terracotta 3.7.0

We haven't had any servoy application crashes since increasing the RAM to 12GB, only at 8GB a few weeks ago. You'd think I should be able to allocate more than 3G out of the 12G to Servoy64. I'll increase the Xmx setting for Servoy64 tonight and see what that does.

Looking at TC RAM usage in Visual VM, it appear to increase until it hits 90% eg around 11:27 then does a cleanup, which seems to take about 10 minutes...

Code: Select all

2012-11-14 11:21:28,374 [Statistics Logger] INFO com.terracottatech.dso - memory free : 177.960243 MB
2012-11-14 11:21:28,380 [Statistics Logger] INFO com.terracottatech.dso - memory used : 833.039757 MB
2012-11-14 11:21:28,381 [Statistics Logger] INFO com.terracottatech.dso - memory max : 1011.000000 MB
2012-11-14 11:27:42,026 [TC Memory Monitor] WARN tc.operator.event - NODE : localhost:9510  Subsystem: MEMORY_MANAGER Message: Current Memory usage(94%) crossed critical threshold(90%).
2012-11-14 11:36:28,374 [Statistics Logger] INFO com.terracottatech.dso - memory free : 653.333107 MB
2012-11-14 11:36:28,374 [Statistics Logger] INFO com.terracottatech.dso - memory used : 332.104393 MB
2012-11-14 11:36:28,374 [Statistics Logger] INFO com.terracottatech.dso - memory max : 985.437500 MB
2012-11-14 11:36:36,582 [Server Map Periodic Evictor] INFO tc.operator.event - NODE : localhost:9510  Subsystem: DCV2 Message: DCV2 Eviction - Time taken (msecs)=0, Number of entries evicted=0, Number of segments over threshold=0, Total Overshoot=0
2012-11-14 11:36:36,582 [Server Map Periodic Evictor] INFO com.tc.objectserver.impl.ServerMapEvictionStatsManager - Server Map Periodic eviction - Time taken (msecs): 0, Number of segments under threshold: 0, Number of segments over threshold: 0, Total overshoot: 0, Total number of samples requested: 0, Number of segments where eviction happened: 0, Total number of evicted entries: 0


I can try forcing the GC more often with your tips, but if it is happening anyway will it help?

Screen Shot 2012-11-14 at 12.16.58 PM.png (189.66 KiB) Viewed 5150 times

[Andrei Costescu]

The DGC tuning could help avoid TC server disk caching - which might slow down the TC server a bit (if it tends to go over Xmx). I don't think that it takes 10 min for that DGC to run, it's just around 10 min between the memory logger triggers. You can enable DGC logging as well - see the link for modifying the DGC interval on the other thread. Btw. the DGC only collects shared objects that have already been collected by GCs in the JVMs that used them, so DGC and JVM GCs should be tuned together. No use to have DGC clean every 10 min but JVM GCs clean much less often.

I thought you got crashes with 12 GB or RAM as well. If I remember correctly from the other discussion the 8 GB RAM crash was when you had configured Xmx (5G + 512M + 512M) + max permGen (~400M), so around 6 GB for the whole cluster and you already had up to 2 GB used. So then you were really at the limit with Java heap and permGen only, without taking into account other memory needs of the JVMs.

Yes, you should be able to increase the Xmx setting for Servoy 64. Let us know how it goes.

[Andrei Costescu]

A nice way to play with these GC and DGC settings is monitoring JVM heap via jvisualvm and Terracotta shared heap via terracotta's Developer Console.

[antonio]

Thanks for all the excellent tips.
At 8GB it crashed with 3G + 0.5G + 0.5G as the total used kept increasing to 8G, so I increased the RAM to 12G to see what would happen and found it does plateau at around 10G. This got me thinking about what is using up all the extra RAM, and what needs to be done if I want to increase RAM allocated to Servoy64.
I'll change one thing at a time and give it a reasonable window so we can see what makes the most difference.

[Andrei Costescu]

Ah ok. You can also try other JVM versions to see if the non-heap memory usage is the same.