Servoy Forum

does killing the profiling (performace stuff) so set it to 0 in production make it better?

We can't do anything except when we get a memory dump before it started to crash to be able to see what is really in memory.

Hi
We have set the values to 0
As soon as the problem arises, make a memory dump and attach it here
Thanks
Roberto

Disabling the profiling hadn't solved the problem.
In the next few days we will try upgrading to 2021.12.2 and let you know.

rmidata wrote:Hi
We have set the values to 0
As soon as the problem arises, make a memory dump and attach it here
Thanks
Roberto

if you can create memory dumps, do 7zip them (7zip.org) then almost nothing is left.

Hi Johan
I think we were just able to do the crash dump
The server was working 5 days and the memory was full
Attached the memory dump
Thanks
Roberto

thats not a memory dump
thats a stack dump, a memory dump generates a huge file (can be as big as the full used heap memory )
but your system is not out of mem:

Heap memory: allocated=15462400K, used=12843481K, max=15528448K

used is only 12G and it can go to 15.5G


A memory dump can be created with tools like JMap or YourKit

Hi
Ah Ok, I had never done a memory dump, I thought it was that option, now I will investigate it and do a memory dump
The same comment that after 5 or 6 days the server gets slow and we have to restart, even though we see that it has memory
It is as if at some point the memory is not released, but it is an assumption
We have already reinstalled the linux server, tomcat, application server, etc.
When I have the memory dump I'll upload it here
Thanks

Hi guys, the same thing happens to me. I have two different server versions installed. In the latest version of servoy version 2021.12.2 -releaseNumber 3724 after a few days it crashes and you have to restart Tomcat version 9. With Servoy version 8.3.3 -releaseNumber 3209 we have no problems.
I send screenshots with differences in Tomcat sessions that I see:





regards!
Juan Pablo

what kind of crash?

without any memory dumps (if it is an out of mem) we can't do much

are those ngclient ? because i think in 8.3 we don't create http sessions for ngclient (only a websocket session)

We do now in the later releases

Hello Johan, yes they are NG clients. I sent the previous post because it is the only difference I found between the two servers. Which the latest version causes Tomcat to crash.
I don't know if it has something to do with those sessions that are now created on the tomcat server.

Greetings!
Juan Pablo

Hi - we too are experiencing the java.lang.OutOfMemoryError: Java heap space problem.

We're upgrading from 2020.06 to 2022.09.2. I deployed the 2022.09 .war file to our Production environment last night. Before noon today we encountered the heap space problem (~16 hours). Note that our configuration for Tomcat 8.5.x --> -Xms1G -Xmx2G; so we could increase -Xmx a bit, but we never ran into this issue with a 2020.06 .war. I have set the database and client performance values to 0 (from 500) in hopes that it will delay the next OOM.

We have been testing the 2022.09 .war file on our UAT server for over a month and haven't noticed the heap space problem. Of course we've only got 6 to 10 users on the UAT, compared to > 300 at times in Production. And Tomcat on UAT is restarted more often (with each .war deployment).

Unfortunately we didn't configure Tomcat to create the memory dump file on out of memory. Are there any other suggestions to avoid this problem? I've already looked at this https://support.servoy.com/browse/SVY-17370, but no solution is suggested or resolution noted.

Waiting for the next OutOfMemoryError: Java heap space error to occur is not a desirable option, so I'll be rolling back to 2020.06 until we have some other options.

Has anyone else encountering this problem on this thread successfully resolved the issue?

Darren

to see what is eating up, we need to have a memory dump

Also are all clients being cleaned up, so on the admin page are you seeing more and more clients that are in a "websocket disconnected" state but they are not cleaned up after 60 seconds (that the session time out)

then you could slowly grow and grow because the clients are not cleaned up.

this i can also see a bit if you make a stackdump from the admin page, to see what the clean up thread is doing.

Thanks for the response Johan. Unfortunately I didn't know to check for "websocket disconnected" clients at the time, or to do a stackdump from the admin page.

As I mentioned in my other post, I've had to revert back to the 2020.06 .war file to avoid having our application go down intermittently. So currently I won't be able to answer those questions, and I'm hoping I won't have to deploy a .war file (2022.09) with a known issue, without at least trying something to fix it...

Is there a setting somewhere that would cause the disconnected websocket to not get cleaned up? Or something else I can do, other than set the database and client performance values to 0 (from 500)? Can the "disconnected websockets" be manually cleaned up on the admin page? Of course, that's assuming we were getting any clients with "websocket disconnected".

Thanks.

if you get that that can mean multiply things.
for some we have fixed in the coming 2022.12 release (rc1 already has fixes for this) or the coming 2022.03 LTS release (released very likely early next year)

but it could also be other things like that the solution does stuff so that one client is in a never ending loop, or some kind of deadlock happens, (long transactions on the database) and so on.
If such a thing happens Servoy can't really clean up the clients because the client blocks its own clean up..
And this kind of stuff needs to be fixed in the solution code itself. (those are kind of programming errors in the servoy solution code)

But you can always try to do some test on the 2022.12.0_RC1 release that was released friday to see how it works in that one.

Thanks for the additional information. We're not ready to move to 2022.12, due to the Java 17 dependency. We're just trying to get through this upgrade from 2020.06 to 2022.09 and are having enough trouble. Adding another change point (Java 11 to 17) before the other problems are fixed doesn't seem like a good idea...

You say there are multiple things that could be causing the java.lang.OutOfMemoryError: Java heap space error. Do they all eventually lead to "websocket disconnected" clients? which in turn cause the Java heap space error? If so, I can monitor our UAT environment and see if there are any "websocket disconnected" clients.

I'm not convinced that it is programming errors in our solution code, only because we weren't encountering the error with a 2020.06 .war file... unless there are changes within Servoy between 2020.06 and 2022.09 that would 'expose' such solution code and cause the error. Perhaps that's what you're referring to with the fixes in 2022.12?