Hi guys,
Sometimes our server just happened to stop running, I mean the process is gone. Right now I was checking on the admin page when suddenly it didn’t respond. When I check on the computer it runs the process was missing. Last few days it happened every day, but usually, once per week.
I have nothing in the log. Any reasons why this might happen?
Thanks,
Bogdan.
Hi Bogdan,
Nothing in the system logs of the server ?
I found this in the bottom server.log just after the crash:
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1367576 bytes for Chunk::new
# An error report file with more information is saved as:
# /Software/servoy_aeroplus/application_server/hs_err_pid30670.log
BUT:
Now, this is my memory usage on the vps. Maybe some of you might have a clue why the java memory is so high.
Tasks: 77 total, 2 running, 75 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.7%us, 0.1%sy, 0.0%ni, 99.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 4209504k total, 3755144k used, 454360k free, 62936k buffers
Swap: 488272k total, 21912k used, 466360k free, 1670040k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16388 root 20 0 2762m 1.7g 11m S 2 43.0 3:36.39 java
1536 root 20 0 1926m 7840 664 S 0 0.2 8:57.05 dbsrv11
1 root 20 0 2116 416 388 S 0 0.0 0:00.29 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.12 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:04.65 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/1
7 root 15 -5 0 0 0 S 0 0.0 0:00.04 ksoftirqd/1
8 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/1
9 root 15 -5 0 0 0 S 0 0.0 0:00.00 cpuset
10 root 15 -5 0 0 0 S 0 0.0 0:05.40 events/0
11 root 15 -5 0 0 0 S 0 0.0 0:00.03 events/1
12 root 15 -5 0 0 0 S 0 0.0 0:00.00 work_on_cpu/0
13 root 15 -5 0 0 0 S 0 0.0 0:00.00 work_on_cpu/1
14 root 15 -5 0 0 0 S 0 0.0 0:00.00 khelper
17 root 15 -5 0 0 0 S 0 0.0 0:00.00 netns
24 root 15 -5 0 0 0 S 0 0.0 0:00.00 xenwatch
25 root 15 -5 0 0 0 S 0 0.0 0:00.14 xenbus
68 root 15 -5 0 0 0 S 0 0.0 0:00.00 kintegrityd/0
69 root 15 -5 0 0 0 S 0 0.0 0:00.00 kintegrityd/1
71 root 15 -5 0 0 0 S 0 0.0 0:00.02 kblockd/0
72 root 15 -5 0 0 0 S 0 0.0 0:00.00 kblockd/1
79 root 15 -5 0 0 0 S 0 0.0 0:00.00 kseriod
98 root 15 -5 0 0 0 S 0 0.0 0:00.00 kondemand/0
99 root 15 -5 0 0 0 S 0 0.0 0:00.00 kondemand/1
124 root 15 -5 0 0 0 S 0 0.0 0:04.28 kswapd0
125 root 15 -5 0 0 0 S 0 0.0 0:00.00 aio/0
126 root 15 -5 0 0 0 S 0 0.0 0:00.00 aio/1
207 root 15 -5 0 0 0 S 0 0.0 0:00.00 khvcd
555 root 15 -5 0 0 0 S 0 0.0 0:06.87 kjournald
627 root 16 -4 2300 368 364 S 0 0.0 0:00.15 udevd
1071 root 20 0 3040 992 784 S 0 0.0 0:05.80 xe-daemon
1314 root 20 0 28308 1100 812 S 0 0.0 0:00.06 rsyslogd
1327 messageb 20 0 2632 364 360 S 0 0.0 0:00.00 dbus-daemon
1339 avahi 20 0 3144 1068 740 S 0 0.0 0:40.31 avahi-daemon
1340 avahi 20 0 2896 220 212 S 0 0.0 0:00.00 avahi-daemon
1351 root 20 0 5428 620 516 S 0 0.0 0:02.33 sshd
1362 zabbix 25 5 5148 384 364 S 0 0.0 0:00.00 zabbix_agentd
1364 zabbix 25 5 5148 520 464 S 0 0.0 2:06.27 zabbix_agentd
1365 zabbix 25 5 5164 560 472 S 0 0.0 0:00.34 zabbix_agentd
1366 zabbix 25 5 5164 560 472 S 0 0.0 0:00.48 zabbix_agentd
1367 zabbix 25 5 5156 560 472 S 0 0.0 0:00.44 zabbix_agentd
1368 zabbix 25 5 5148 636 536 S 0 0.0 1:00.89 zabbix_agentd
1383 root 20 0 3440 492 428 S 0 0.0 0:00.02 cron
1504 root 20 0 1780 316 312 S 0 0.0 0:00.00 getty
1505 root 20 0 1780 316 312 S 0 0.0 0:00.00 getty
1506 root 20 0 1780 316 312 S 0 0.0 0:00.00 getty
Have a look also at this. This is servoy process. It consumes 1.8GB, if I’m not wrong…
root 16388 2.5 43.0 2829220 1811772 ? Sl 20:11 3:43 java -Djava.awt.headless=true -Xmx1048m -Xms512m -XX:MaxPermSize=128m -classpath .:lib/ohj-jewt.jar:lib/MRJAdapter.jar:lib/compat141.ja
, while in the admin page of servoy it state this:
System Information
Heap memory: allocated=514432K, used=89877K, max=953920K
None Heap memory: allocated=90272K, used=55336K, max=180224K
which java version is this?
It seems to collapse with an out of mem, but not a “normal” java one, but something deep in the vm and the vm is then crashing…
This is the java version:
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) Server VM (build 20.1-b02, mixed mode)
Do you think reinstalling it might solve the problem?
you could try but i don’t know if that will help
Because there won’t be much of a change then
As i said this is a hard crash, what you could do is try an slightly older or newer version of java
also we could look a bit more into the file: hs_err_pid30670.log
This is it:
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1367576 bytes for Chunk::new
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (allocation.cpp:317), pid=30670, tid=1775238032
#
# JRE version: 6.0_26-b03
# Java VM: Java HotSpot(TM) Server VM (20.1-b02 mixed mode linux-x86 )
--------------- T H R E A D ---------------
Current thread (0x0869dc00): JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=30678, stack(0x69c7f000,0x69d00000)]
Stack: [0x69c7f000,0x69d00000], sp=0x69cfd030, free space=504k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x7248b0]
that doesn’t tell me much more, but it seems that it is really internal to java (the compiler thread) that wants to have more memory
But the system can’t give it i guess because for a 32 bit vm it is already maxed out
But that is a bit weird because you only give it a heap of 1GB and some none heap. So why it seems to be almost on the max of the 2GB 32bit vm space i dont know
Please try a different java install then
Thank you!
Hello Johan,
I’m back with some other comments on the server memory issue. I also discussed with Nicola Gervasi and he gave me some more tips. They were really helpful, so right now it seems that the server is more stable concerning the memory, I hope it will keep this way. He told me about 2 java parameters that can force java to get rid of unused memory so now I always have at most 1.3GB consumed, instead of more then 2GB:
root 6207 5.6 29.1 2329884 1228580 ? Sl 03:55 1:52 java -Djava.awt.headless=true -Xmx1048m -Xms256m -XX:MaxPermSize=128m -XX:MaxHeapFreeRatio=10 -XX:MinHeapFreeRatio=10 -classpath .:lib/
I just used these params, as you can see :
-XX:MaxHeapFreeRatio=10
-XX:MinHeapFreeRatio=10
I didn’t reinstall java, because this is the only online server running so far, so we wanted to make first some tests on other computes and then decide.
On another linux server the memory at startup without those params is this:
root 3775 0.2 45.7 1382552 939508 ? Sl Oct20 2:48 java -Djava.awt.headless=true -Xmx256m -Xms64m -XX:MaxPermSize=128m -classpath .:lib/ohj-jewt.jar:lib/MRJAdapter.jar:lib/compat141.jar:lib/commons-
Heap memory: allocated=67672K, used=42568K, max=253440K
None Heap memory: allocated=67392K, used=66661K, max=180224K
Still too high, I think, without any clients connected. Java using 1GB while the heap used is 42MB.
On a mac it looks better, though:
bogdan 10640 0.0 8.8 4197316 367556 s000 S 2:37PM 1:25.29 /usr/bin/java -Djava.awt.headless=true -Xmx1024m -Xms256m -XX:MaxPermSize=128m -XX:MaxHeapFreeRatio=10 -
This is the memory after connecting some client. Btw, all clients we have are web clients. On this mac, java starts with a little more then 200MB. When connecting with first webclient the memory reach almost 1GB, or 700MB sometimes, then after few minutes it decrease. Then when connecting another client it won’t jump to 1GB again, but stabilize somewhere between 350MB and 500MB.
We also thought that the errors we have in the log might cause this, although it doesn’t seems to relevant. If they would leak memory, the leaks would be in the heap. But I’ll also try to fix as many using your tips and then I’ll wrote back on the other topic.
Kind Regards,
Bogdan.
setting this one: MaxHeapFreeRatio to 10 is a bit low in my eyes
that is not a good idea for everybody
Because that would mean if there is more then 10% free heap after a gc it will reduce the heap to be used + 10%
That will mean a lot of growing and shrinking…
I do think that the default valule (70) is way to high, i would personally do something like 30 or 40
Yes, for a server instance maybe 30-40 is more appropriate but from my testing I noticed that on Smart Client setting both params to 10 helps a lot and doesn’t have any impact on performance.
My issue was that smart client were using too much memory (due to a Jasper Report memory leak) and they never released it to the OS, since this customer is using Citrix it was quite a problem because the high memory usage was limiting the number of citrix sessions per server.
problem with 10 even on the client is that maybe you don’t directly notice it
But if the latency is a bit higher between the server i think the constant reduction of the smart clients heap can result in a lot less caching of related foundset and rows
That would mean way more queries in the long run then you would get if it is allowed a bit more free memory
So 10 is really also on the client not directly the best thing to do i think 30 or 40 is way better
Then for example a client that consumes on average 100MB it will not shrink to 110 (clear way more caches caches and grows/shrinks quite a lot)
but it will keep 140 free which is way more playing field.
Of course this depends a bit on the solution, but i would give the client a bit more space to play in.
After almost one day with -XX:MaxHeapFreeRatio=10 -XX:MinHeapFreeRatio=10 params set the server is much stable. I still didn’t changed them to 30. But I will and then see how it behaves. The heap is this. I never saw it so low after one day until now:
Heap memory: allocated=343488K, used=289895K, max=953920K
None Heap memory: allocated=94816K, used=88255K, max=180224K
But the free memory on the linux decrees during this evening from 1GB to 500MB although the servoy memory remained to 1.3GB, and also the heap is in the same values, and I didn’t find any other processes to consumes the memory.
I have found this command to clean any cache from the memory:
sync; echo 3 > /proc/sys/vm/drop_caches
After using it I was amazed. Instead of 500MB free memory it went to 2.5GB. It decreased in few minutes to 2.1GB, but for now it seems to like this value.
you don’t have to do both to 30
only MaxHeapFreeRatio
that MinHeapFreeRatio can be on 10 just fine.
Hi guys,
We installed these days another vps with 64bit, same java version 6 u26. A fresh instance of Servoy 5.2.10 which seems to be more stable, even when I deploy a new solution it takes much less then before. But the memory consumed by java is the same 1.8GB.
root 11629 6.9 44.4 2105640 1829808 hvc0 Sl 11:42 14:18 java -Djava.awt.headless=true -Xmx256m -Xms64m -XX:MaxPermSize=128m -classpath .:lib/ohj-jewt.jar:lib/MRJAdapter.jar:lib/compat141.jar:lib/commons-codec.jar:lib/commons-httpclient.jar:lib/activation.jar:lib/antlr.jar:lib/commons-collections.jar:lib/commons-dbcp.jar:lib/commons-fileupload-1.2.1.jar:lib/commons-io-1.4.jar:lib/commons-logging.jar:lib/commons-pool.jar:lib/dom4j.jar:lib/help.jar:lib/jabsorb.jar:lib/hibernate3.jar:lib/j2db.jar:lib/j2dbdev.jar:lib/jdbc2_0-stdext.jar:lib/jmx.jar:lib/jndi.jar:lib/js.jar:lib/jta.jar:lib/BrowserLauncher2.jar:lib/jug.jar:lib/log4j.jar:lib/mail.jar:lib/ohj-jewt.jar:lib/oracle_ice.jar:lib/server-bootstrap.jar:lib/servlet-api.jar:lib/wicket-extentions.jar:lib/wicket.jar:lib/wicket-calendar.jar:lib/slf4j-api.jar:lib/slf4j-log4j.jar:lib/joda-time.jar:lib/rmitnl.jar:lib/networktnl.jar com.servoy.j2db.server.ApplicationServer
We had a crash after about one day, then we increase the xmx to 512.
SEVERE: An exception or error occurred in the container during the request processing
java.lang.OutOfMemoryError: Java heap space
Nov 15, 2011 12:36:05 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error finishing response
Since then it was ok for few days, but this morning I restarted it again because the used memory was almost to maximum, and I didn’t want to risk a crash during the day.
Now, the old vps which has been put on paused I using it for testing. I didn’t want to made and changes before, when the solution was online there, not to damage anything, as I’m a beginner with linux.
I just installed java7 there and I compared. Only one webclient for each test:
java7:
root 17487 46.5 16.7 1364272 354304 pts/1 Sl 17:33 0:39 ./jre1.7.0_01/bin/java -Djava.awt.headless=true -Xmx1024m -Xms128m -XX:MaxPermSize=128m -XX:MaxHeapFreeRatio=20 -XX:MinHeapFreeRatio=10 -classpath .:lib/ohj-jewt.jar:lib/MRJAdapter.jar:lib/compat141.jar:lib/commons-codec.jar:lib/commons-httpclient.jar:lib/activation.jar:lib/antlr.jar:lib/commons-collections.jar:lib/commons-dbcp.jar:lib/commons-fileupload-1.2.1.jar:lib/commons-io-1.4.jar:lib/commons-logging.jar:lib/commons-pool.jar:lib/dom4j.jar:lib/help.jar:lib/jabsorb.jar:lib/hibernate3.jar:lib/j2db.jar:lib/j2dbdev.jar:lib/jdbc2_0-stdext.jar:lib/jmx.jar:lib/jndi.jar:lib/js.jar:lib/jta.jar:lib/BrowserLauncher2.jar:lib/jug.jar:lib/log4j.jar:lib/mail.jar:lib/ohj-jewt.jar:lib/oracle_ice.jar:lib/server-bootstrap.jar:lib/servlet-api.jar:lib/wicket-extentions.jar:lib/wicket.jar:lib/wicket-calendar.jar:lib/slf4j-api.jar:lib/slf4j-log4j.jar:lib/joda-time.jar:lib/rmitnl.jar:lib/networktnl.jar com.servoy.j2db.server.ApplicationServer
java6 u26:
root 17703 61.7 59.6 2371572 1259004 pts/1 Sl 17:36 0:43 java -Djava.awt.headless=true -Xmx1024m -Xms128m -XX:MaxPermSize=128m -XX:MaxHeapFreeRatio=20 -XX:MinHeapFreeRatio=10 -classpath .:lib/ohj-jewt.jar:lib/MRJAdapter.jar:lib/compat141.jar:lib/commons-codec.jar:lib/commons-httpclient.jar:lib/activation.jar:lib/antlr.jar:lib/commons-collections.jar:lib/commons-dbcp.jar:lib/commons-fileupload-1.2.1.jar:lib/commons-io-1.4.jar:lib/commons-logging.jar:lib/commons-pool.jar:lib/dom4j.jar:lib/help.jar:lib/jabsorb.jar:lib/hibernate3.jar:lib/j2db.jar:lib/j2dbdev.jar:lib/jdbc2_0-stdext.jar:lib/jmx.jar:lib/jndi.jar:lib/js.jar:lib/jta.jar:lib/BrowserLauncher2.jar:lib/jug.jar:lib/log4j.jar:lib/mail.jar:lib/ohj-jewt.jar:lib/oracle_ice.jar:lib/server-bootstrap.jar:lib/servlet-api.jar:lib/wicket-extentions.jar:lib/wicket.jar:lib/wicket-calendar.jar:lib/slf4j-api.jar:lib/slf4j-log4j.jar:lib/joda-time.jar:lib/rmitnl.jar:lib/networktnl.jar com.servoy.j2db.server.ApplicationServer
So, as you can see, there is a big difference in the memory used by java. The heap was almost the same. I’ll do more testing, and I’ll update the post with any weird things I might see.
Back again!
Java6 update29:
root 16018 28.8 12.4 1349500 262760 pts/1 Sl 06:14 0:51 ./jre1.6.0_29/bin/java -Djava.awt.headless=true -Xmx1024m -Xms128m -XX:MaxPermSize=128m -XX:MaxHeapFreeRatio=20 -XX:MinHeapFreeRatio=10 -classpath .:lib/ohj-jewt.jar:lib/MRJAdapter.jar:lib/compat141.jar:lib/commons-
Same as before, only one webclient opened. System memory is less though, that’s good. Heap memory looks the same, it’s around 100MB used. It went to almost 200MB, then dropped to 40MB. WOW. I never saw that on my server.
System Information
Heap memory: allocated=186112K, used=42130K, max=932096K
None Heap memory: allocated=46528K, used=46273K, max=180224K
While I was writing, I checked again the system. It went up to 300MB now.
root 16018 14.5 14.4 1381300 305676 pts/1 Sl 06:14 1:06 ./jre1.6.0_29/bin/java -Djava.awt.headless=true -Xmx1024m -Xms128m -XX:MaxPermSize=128m -XX:MaxHeapFreeRatio=20 -XX:MinHeapFreeRatio=10 -classpath .:lib/ohj-jewt.jar:lib/MRJAdapter.jar:lib/compat141.jar:lib/commons-
I got java from oracle: http://download.oracle.com/otn-pub/java … x-i586.bin , for 64bit the link is this: http://download.oracle.com/otn-pub/java … ux-x64.bin
I’ll start to test it now without the -XX:MaxHeapFreeRatio=20 -XX:MinHeapFreeRatio=10 params.
Guys, I’m starting to be happy I hope in the end I will keep my happiness and maybe increase it.
UPDATE: after couple of minutes. I’ve tested without the HeapFree params and this is what I have after running for about 5 minutes with one webclient and one batchprocess:
java6u29:
root 16538 8.6 21.8 1358748 461424 pts/1 Sl 06:24 1:17 ./jre1.6.0_29/bin/java -Djava.awt.headless=true -Xmx1024m -Xms128m -XX:MaxPermSize=128m -classpath .:lib/ohj-jewt.jar:lib/MRJAdapter.ja
java7:
root 17340 19.7 21.7 1351624 458516 pts/1 Sl 06:40 0:57 ./jre1.7.0_01/bin/java -Djava.awt.headless=true -Xmx1024m -Xms128m -XX:MaxPermSize=128m -classpath .:lib/ohj-jewt.jar:lib/MRJAdapter.ja
Another test with a new installed java6 update26, downloaded now from oracle, the one that cause all this problem. 1.1GB used at startup after I logged in with one webclient and a batch process.
root 25624 49.1 53.1 2276296 1121780 pts/0 Sl 10:01 0:47 ./jre1.6.0_26/bin/java -Djava.awt.headless=true -Xmx1024m -Xms128m -XX:MaxPermSize=128m -classpath .:lib/ohj-jewt.jar:lib/MRJAdapter.ja
System Information
Heap memory: allocated=347648K, used=119405K, max=932096K
None Heap memory: allocated=45312K, used=45172K, max=180224K
I really beginning to think that this version has a problem.
Hello again, coming up with another issue. I’m running now with java7 and servoy 5.2.9 which I’m using only for testing.
For few days I’m keeping an eye on the heap memory (the total process memory is OK now with java7, not perfect, but much better then with java6u26).
I’m having the server running for about 12 hours with only a batch process. Until now the memory was about 100MB plus or minus 10 - 20 MB, depending on what it calculate and the release of the memory from time to time. I just deployed a new solution. And immediately after the update was finished the memory increased to more then 200MB. I watched it some time and I saw the it moved between 200 - 300MB. Remember, I had only one batch process, no other clients what so ever.
I closed also the batch process, but the memory remain in the same range.
Minimum heap level (after a GC release I think)
System Information
Heap memory: allocated=343040K, used=205781K, max=932096K
None Heap memory: allocated=48064K, used=47568K, max=180224K
Maximum heap
System Information
Heap memory: allocated=363584K, used=306829K, max=932096K
None Heap memory: allocated=48064K, used=47567K, max=180224K
I think it is increasing from 200 to 300 MB because I refresh the admin page to see the memory usage. For each refresh I notice an increase with about 1MB or less.
So, from what I see here is that after a solution import, the memory went up to 100MB more then before. It doesn’t decrease when the GC comes up to free the unused memory.
I don’t know if this is the only case. But my questions are these:
1. In case a client is closed all the memory that was allocated within it, should be released, no matter what, right? It might be possible not to free it all in case of some exceptions/errors?
2. If the client’s memory is released when it is closed, but I still see unused memory, then can servoy leak it?
you are trying to see stuff that shouldn’t be looked at like that
Do you have a a problem? Is your server running out of memory at one point it time?
And then without really suddenly having loads of users?
Let java handle it, by just looking at the admin page values you dont know when it is fully cleared that is completely up to java
those things are just an indication what is currently used