thats not a memory dump
thats a stack dump, a memory dump generates a huge file (can be as big as the full used heap memory )
but your system is not out of mem:
Heap memory: allocated=15462400K, used=12843481K, max=15528448K
used is only 12G and it can go to 15.5G
A memory dump can be created with tools like JMap or YourKit
Hi
Ah Ok, I had never done a memory dump, I thought it was that option, now I will investigate it and do a memory dump
The same comment that after 5 or 6 days the server gets slow and we have to restart, even though we see that it has memory
It is as if at some point the memory is not released, but it is an assumption
We have already reinstalled the linux server, tomcat, application server, etc.
When I have the memory dump I’ll upload it here
Thanks
Hi guys, the same thing happens to me. I have two different server versions installed. In the latest version of servoy version 2021.12.2 -releaseNumber 3724 after a few days it crashes and you have to restart Tomcat version 9. With Servoy version 8.3.3 -releaseNumber 3209 we have no problems.
I send screenshots with differences in Tomcat sessions that I see:
[attachment=1]Servoy8_3_3.png[/attachment]
[attachment=0]Servoy2021_12_2.png[/attachment]
regards!
Juan Pablo
what kind of crash?
without any memory dumps (if it is an out of mem) we can’t do much
are those ngclient ? because i think in 8.3 we don’t create http sessions for ngclient (only a websocket session)
We do now in the later releases
Hello Johan, yes they are NG clients. I sent the previous post because it is the only difference I found between the two servers. Which the latest version causes Tomcat to crash.
I don’t know if it has something to do with those sessions that are now created on the tomcat server.
Greetings!
Juan Pablo
Hi - we too are experiencing the java.lang.OutOfMemoryError: Java heap space problem.
We’re upgrading from 2020.06 to 2022.09.2. I deployed the 2022.09 .war file to our Production environment last night. Before noon today we encountered the heap space problem (~16 hours). Note that our configuration for Tomcat 8.5.x → -Xms1G -Xmx2G; so we could increase -Xmx a bit, but we never ran into this issue with a 2020.06 .war. I have set the database and client performance values to 0 (from 500) in hopes that it will delay the next OOM.
We have been testing the 2022.09 .war file on our UAT server for over a month and haven’t noticed the heap space problem. Of course we’ve only got 6 to 10 users on the UAT, compared to > 300 at times in Production. And Tomcat on UAT is restarted more often (with each .war deployment).
Unfortunately we didn’t configure Tomcat to create the memory dump file on out of memory. Are there any other suggestions to avoid this problem? I’ve already looked at this https://support.servoy.com/browse/SVY-17370, but no solution is suggested or resolution noted.
Waiting for the next OutOfMemoryError: Java heap space error to occur is not a desirable option, so I’ll be rolling back to 2020.06 until we have some other options.
Has anyone else encountering this problem on this thread successfully resolved the issue?
Darren
to see what is eating up, we need to have a memory dump
Also are all clients being cleaned up, so on the admin page are you seeing more and more clients that are in a “websocket disconnected” state but they are not cleaned up after 60 seconds (that the session time out)
then you could slowly grow and grow because the clients are not cleaned up.
this i can also see a bit if you make a stackdump from the admin page, to see what the clean up thread is doing.
Thanks for the response Johan. Unfortunately I didn’t know to check for “websocket disconnected” clients at the time, or to do a stackdump from the admin page.
As I mentioned in my other post, I’ve had to revert back to the 2020.06 .war file to avoid having our application go down intermittently. So currently I won’t be able to answer those questions, and I’m hoping I won’t have to deploy a .war file (2022.09) with a known issue, without at least trying something to fix it…
Is there a setting somewhere that would cause the disconnected websocket to not get cleaned up? Or something else I can do, other than set the database and client performance values to 0 (from 500)? Can the “disconnected websockets” be manually cleaned up on the admin page? Of course, that’s assuming we were getting any clients with “websocket disconnected”.
Thanks.
if you get that that can mean multiply things.
for some we have fixed in the coming 2022.12 release (rc1 already has fixes for this) or the coming 2022.03 LTS release (released very likely early next year)
but it could also be other things like that the solution does stuff so that one client is in a never ending loop, or some kind of deadlock happens, (long transactions on the database) and so on.
If such a thing happens Servoy can’t really clean up the clients because the client blocks its own clean up…
And this kind of stuff needs to be fixed in the solution code itself. (those are kind of programming errors in the servoy solution code)
But you can always try to do some test on the 2022.12.0_RC1 release that was released friday to see how it works in that one.
Thanks for the additional information. We’re not ready to move to 2022.12, due to the Java 17 dependency. We’re just trying to get through this upgrade from 2020.06 to 2022.09 and are having enough trouble. Adding another change point (Java 11 to 17) before the other problems are fixed doesn’t seem like a good idea…
You say there are multiple things that could be causing the java.lang.OutOfMemoryError: Java heap space error. Do they all eventually lead to “websocket disconnected” clients? which in turn cause the Java heap space error? If so, I can monitor our UAT environment and see if there are any “websocket disconnected” clients.
I’m not convinced that it is programming errors in our solution code, only because we weren’t encountering the error with a 2020.06 .war file… unless there are changes within Servoy between 2020.06 and 2022.09 that would ‘expose’ such solution code and cause the error. Perhaps that’s what you’re referring to with the fixes in 2022.12?
but you should not move to 2022.09 if you don’t plan to move to .12 after that very soon
because 09 is already end of live and will not get any more updates
If you don’t plan to update often (so 06 to 09 to 12) then it is way better to just install the lts, so move to 2022.03.x LTS
That will get the most important fixes, the next release of that will have fixes if you would get websocket disconnected and clients are not removed, it a certain scenario.
I understand your reasoning on moving to new versions. I shouldn’t say we’re not willing to move to 2022.12, it is just unfortunate that doing so requires this other big change point (Java 17), when there is this other large problem (OOM Java heap space) we’re trying to solve. We could move to 2022.03 LTS right now, but from earlier posts in this thread, it appears as though we’ll still have the OOM problem. So perhaps we’ll wait until the new 2022.03 LTS with the fixes is released.
Are you able to comment on my other questions:
You say there are multiple things that could be causing the java.lang.OutOfMemoryError: Java heap space error. Do they all eventually lead to “websocket disconnected” clients? which in turn cause the Java heap space error? If so, I can monitor our UAT environment and see if there are any “websocket disconnected” clients.
I’m not convinced that it is programming errors in our solution code, only because we weren’t encountering the error with a 2020.06 .war file… unless there are changes within Servoy between 2020.06 and 2022.09 that would ‘expose’ such solution code and cause the error. Perhaps that’s what you’re referring to with the fixes in 2022.12?
Additionally, I’ve uploaded the relevant logging from our servoy application log file in the lead up to the OOM - Java heap space error, in case there are some clues. There are certainly some log messages that I haven’t seen before. Perhaps you could take a quick look.
Thanks again.
[attachment=0]servoy-application-log.txt[/attachment]
servoy-application-log.txt (1.28 MB)
there can be hunderds of causes that will end up in OOM…
i have seen many, like big report generation, huge files loaded in to memory and then that intermediate result of “websocket disconnected” can also be stuff like databasetransactions in deadlock blocking stuff, never ending loops in code, code deadlocks (this last thing is something servoy must fix), websocket/browser connections errors that results in a very weird state at the server (this is a fix that is in .12 and in the next 03.LTS) and then in the end you OOM out because the clients are never shutdown.
there are some very weird exceptions in that log:
2022-12-09 11:47:46,018 ERROR [Executor,uuid:F252A3E:2, clientid: 3ED40EB9-9CF4-4D29-8DAA-EF03F92E5C65] org.sablo.websocket.utils.JSONUtils - unsupported value type:null for value: QBSelect(SELECT columns FROM db:/qfi/v_deal_progress where XXXX)
at org.sablo.websocket.utils.JSONUtils$FullValueToJSONConverter.toJSONValue(JSONUtils.java:479) ~[sablo_2022.9.2.3805.jar:?]
at org.sablo.websocket.utils.JSONUtils$FullValueToJSONConverter.toJSONValue(JSONUtils.java:1) ~[sablo_2022.9.2.3805.jar:?]
at org.sablo.websocket.utils.JSONUtils.writeData(JSONUtils.java:101) ~[sablo_2022.9.2.3805.jar:?]
at org.sablo.websocket.BaseWindow.sendMessageInternal(BaseWindow.java:611) ~[sablo_2022.9.2.3805.jar:?]
at org.sablo.websocket.BaseWindow.sendAsyncMessage(BaseWindow.java:509) ~[sablo_2022.9.2.3805.jar:?]
for some reason it tries to push of a query to the client… i have really no idea how that is possible. that should never happen in the components and services we ship
the rest really seems to be a lot of users that are pressing a button, changing data, showing form in a dialog, so really interacting with the server, but then just close the browser or tab, a lot of exceptions that the websocket is disconnected in the middle of calls…
The QBSelect exception is something we’ve had for awhile and tried to work through with Servoy support, but never got anywhere and since it the form was working correctly, we chose to ignore it. Your comment on it today prompted us to revisit the exception, and we found two ways to avoid the exception… there’s something weird going on there. I’ll get around to writing it up one day.
I’m guessing the websocket disconnect exceptions at the end of the file were causes by the imminent OOM.
We’ve decided to go ahead and deploy the problem 2022.09 .war file sometime soon and try to capture the required memory dump. Thanks.
