OutOfMemory-Exception with Heap memory

Questions and Answers on installation, deployment, management, locking, tranasactions of Servoy Application Server

Re: OutOfMemory-Exception with Heap memory

Postby jcompagner » Wed Feb 09, 2022 5:34 pm

does killing the profiling (performace stuff) so set it to 0 in production make it better?

We can't do anything except when we get a memory dump before it started to crash to be able to see what is really in memory.
Johan Compagner
Servoy
User avatar
jcompagner
 
Posts: 8822
Joined: Tue May 27, 2003 7:26 pm
Location: The Internet

Re: OutOfMemory-Exception with Heap memory

Postby rmidata » Thu Feb 10, 2022 5:24 am

Hi
We have set the values to 0
As soon as the problem arises, make a memory dump and attach it here
Thanks
Roberto
Roberto Diez.
Interdata Software.
Santa Fe, Argentina.
User avatar
rmidata
 
Posts: 42
Joined: Thu Jul 09, 2009 10:26 pm
Location: Santa Fe, Argentina.

Re: OutOfMemory-Exception with Heap memory

Postby marazzi1539352432 » Thu Feb 10, 2022 10:10 am

Disabling the profiling hadn't solved the problem.
In the next few days we will try upgrading to 2021.12.2 and let you know.
marazzi1539352432
 
Posts: 7
Joined: Fri Oct 12, 2018 3:53 pm

Re: OutOfMemory-Exception with Heap memory

Postby jcompagner » Thu Feb 10, 2022 10:49 am

rmidata wrote:Hi
We have set the values to 0
As soon as the problem arises, make a memory dump and attach it here
Thanks
Roberto


if you can create memory dumps, do 7zip them (7zip.org) then almost nothing is left.
Johan Compagner
Servoy
User avatar
jcompagner
 
Posts: 8822
Joined: Tue May 27, 2003 7:26 pm
Location: The Internet

Re: OutOfMemory-Exception with Heap memory

Postby rmidata » Thu Feb 10, 2022 10:49 pm

Hi Johan
I think we were just able to do the crash dump
The server was working 5 days and the memory was full
Attached the memory dump
Thanks
Roberto
Attachments
memory_dump.txt
(663.77 KiB) Downloaded 124 times
Roberto Diez.
Interdata Software.
Santa Fe, Argentina.
User avatar
rmidata
 
Posts: 42
Joined: Thu Jul 09, 2009 10:26 pm
Location: Santa Fe, Argentina.

Re: OutOfMemory-Exception with Heap memory

Postby jcompagner » Fri Feb 11, 2022 10:25 am

thats not a memory dump
thats a stack dump, a memory dump generates a huge file (can be as big as the full used heap memory )
but your system is not out of mem:

Heap memory: allocated=15462400K, used=12843481K, max=15528448K

used is only 12G and it can go to 15.5G


A memory dump can be created with tools like JMap or YourKit
Johan Compagner
Servoy
User avatar
jcompagner
 
Posts: 8822
Joined: Tue May 27, 2003 7:26 pm
Location: The Internet

Re: OutOfMemory-Exception with Heap memory

Postby rmidata » Fri Feb 11, 2022 12:52 pm

Hi
Ah Ok, I had never done a memory dump, I thought it was that option, now I will investigate it and do a memory dump
The same comment that after 5 or 6 days the server gets slow and we have to restart, even though we see that it has memory
It is as if at some point the memory is not released, but it is an assumption
We have already reinstalled the linux server, tomcat, application server, etc.
When I have the memory dump I'll upload it here
Thanks
Roberto Diez.
Interdata Software.
Santa Fe, Argentina.
User avatar
rmidata
 
Posts: 42
Joined: Thu Jul 09, 2009 10:26 pm
Location: Santa Fe, Argentina.

Re: OutOfMemory-Exception with Heap memory

Postby desarrollo1534423715 » Wed Feb 23, 2022 6:33 pm

Hi guys, the same thing happens to me. I have two different server versions installed. In the latest version of servoy version 2021.12.2 -releaseNumber 3724 after a few days it crashes and you have to restart Tomcat version 9. With Servoy version 8.3.3 -releaseNumber 3209 we have no problems.
I send screenshots with differences in Tomcat sessions that I see:
Servoy8_3_3.png
Servoy version 8.3.3 -releaseNumber 3209
Servoy8_3_3.png (52.68 KiB) Viewed 3843 times


Servoy2021_12_2.png
Servoy version 2021.12.2 -releaseNumber 3724
Servoy2021_12_2.png (36.09 KiB) Viewed 3843 times



regards!
Juan Pablo
desarrollo1534423715
 
Posts: 4
Joined: Thu Aug 16, 2018 2:48 pm

Re: OutOfMemory-Exception with Heap memory

Postby jcompagner » Thu Feb 24, 2022 11:42 am

what kind of crash?

without any memory dumps (if it is an out of mem) we can't do much

are those ngclient ? because i think in 8.3 we don't create http sessions for ngclient (only a websocket session)

We do now in the later releases
Johan Compagner
Servoy
User avatar
jcompagner
 
Posts: 8822
Joined: Tue May 27, 2003 7:26 pm
Location: The Internet

Re: OutOfMemory-Exception with Heap memory

Postby desarrollo1534423715 » Thu Mar 03, 2022 8:00 pm

Hello Johan, yes they are NG clients. I sent the previous post because it is the only difference I found between the two servers. Which the latest version causes Tomcat to crash.
I don't know if it has something to do with those sessions that are now created on the tomcat server.

Greetings!
Juan Pablo
desarrollo1534423715
 
Posts: 4
Joined: Thu Aug 16, 2018 2:48 pm

Re: OutOfMemory-Exception with Heap memory

Postby darren » Sat Dec 10, 2022 1:34 am

Hi - we too are experiencing the java.lang.OutOfMemoryError: Java heap space problem.

We're upgrading from 2020.06 to 2022.09.2. I deployed the 2022.09 .war file to our Production environment last night. Before noon today we encountered the heap space problem (~16 hours). Note that our configuration for Tomcat 8.5.x --> -Xms1G -Xmx2G; so we could increase -Xmx a bit, but we never ran into this issue with a 2020.06 .war. I have set the database and client performance values to 0 (from 500) in hopes that it will delay the next OOM.

We have been testing the 2022.09 .war file on our UAT server for over a month and haven't noticed the heap space problem. Of course we've only got 6 to 10 users on the UAT, compared to > 300 at times in Production. And Tomcat on UAT is restarted more often (with each .war deployment).

Unfortunately we didn't configure Tomcat to create the memory dump file on out of memory. Are there any other suggestions to avoid this problem? I've already looked at this https://support.servoy.com/browse/SVY-17370, but no solution is suggested or resolution noted.

Waiting for the next OutOfMemoryError: Java heap space error to occur is not a desirable option, so I'll be rolling back to 2020.06 until we have some other options.

Has anyone else encountering this problem on this thread successfully resolved the issue?

Darren
darren
 
Posts: 19
Joined: Mon Nov 01, 2021 11:10 pm

Re: OutOfMemory-Exception with Heap memory

Postby jcompagner » Sat Dec 10, 2022 3:08 pm

to see what is eating up, we need to have a memory dump

Also are all clients being cleaned up, so on the admin page are you seeing more and more clients that are in a "websocket disconnected" state but they are not cleaned up after 60 seconds (that the session time out)

then you could slowly grow and grow because the clients are not cleaned up.

this i can also see a bit if you make a stackdump from the admin page, to see what the clean up thread is doing.
Johan Compagner
Servoy
User avatar
jcompagner
 
Posts: 8822
Joined: Tue May 27, 2003 7:26 pm
Location: The Internet

Re: OutOfMemory-Exception with Heap memory

Postby darren » Mon Dec 12, 2022 8:02 am

Thanks for the response Johan. Unfortunately I didn't know to check for "websocket disconnected" clients at the time, or to do a stackdump from the admin page.

As I mentioned in my other post, I've had to revert back to the 2020.06 .war file to avoid having our application go down intermittently. So currently I won't be able to answer those questions, and I'm hoping I won't have to deploy a .war file (2022.09) with a known issue, without at least trying something to fix it...

Is there a setting somewhere that would cause the disconnected websocket to not get cleaned up? Or something else I can do, other than set the database and client performance values to 0 (from 500)? Can the "disconnected websockets" be manually cleaned up on the admin page? Of course, that's assuming we were getting any clients with "websocket disconnected".

Thanks.
darren
 
Posts: 19
Joined: Mon Nov 01, 2021 11:10 pm

Re: OutOfMemory-Exception with Heap memory

Postby jcompagner » Mon Dec 12, 2022 10:25 am

if you get that that can mean multiply things.
for some we have fixed in the coming 2022.12 release (rc1 already has fixes for this) or the coming 2022.03 LTS release (released very likely early next year)

but it could also be other things like that the solution does stuff so that one client is in a never ending loop, or some kind of deadlock happens, (long transactions on the database) and so on.
If such a thing happens Servoy can't really clean up the clients because the client blocks its own clean up..
And this kind of stuff needs to be fixed in the solution code itself. (those are kind of programming errors in the servoy solution code)

But you can always try to do some test on the 2022.12.0_RC1 release that was released friday to see how it works in that one.
Johan Compagner
Servoy
User avatar
jcompagner
 
Posts: 8822
Joined: Tue May 27, 2003 7:26 pm
Location: The Internet

Re: OutOfMemory-Exception with Heap memory

Postby darren » Mon Dec 12, 2022 6:30 pm

Thanks for the additional information. We're not ready to move to 2022.12, due to the Java 17 dependency. We're just trying to get through this upgrade from 2020.06 to 2022.09 and are having enough trouble. Adding another change point (Java 11 to 17) before the other problems are fixed doesn't seem like a good idea...

You say there are multiple things that could be causing the java.lang.OutOfMemoryError: Java heap space error. Do they all eventually lead to "websocket disconnected" clients? which in turn cause the Java heap space error? If so, I can monitor our UAT environment and see if there are any "websocket disconnected" clients.

I'm not convinced that it is programming errors in our solution code, only because we weren't encountering the error with a 2020.06 .war file... unless there are changes within Servoy between 2020.06 and 2022.09 that would 'expose' such solution code and cause the error. Perhaps that's what you're referring to with the fixes in 2022.12?
Last edited by darren on Wed Dec 14, 2022 1:19 am, edited 1 time in total.
darren
 
Posts: 19
Joined: Mon Nov 01, 2021 11:10 pm

PreviousNext

Return to Servoy Server

Who is online

Users browsing this forum: No registered users and 1 guest