OutOfMemory-Exception with Heap memory

[jcompagner]

but you should not move to 2022.09 if you don't plan to move to .12 after that very soon
because 09 is already end of live and will not get any more updates

If you don't plan to update often (so 06 to 09 to 12) then it is way better to just install the lts, so move to 2022.03.x LTS
That will get the most important fixes, the next release of that will have fixes if you would get websocket disconnected and clients are not removed, it a certain scenario.

[darren]

I understand your reasoning on moving to new versions. I shouldn't say we're not willing to move to 2022.12, it is just unfortunate that doing so requires this other big change point (Java 17), when there is this other large problem (OOM Java heap space) we're trying to solve. We could move to 2022.03 LTS right now, but from earlier posts in this thread, it appears as though we'll still have the OOM problem. So perhaps we'll wait until the new 2022.03 LTS with the fixes is released.

Are you able to comment on my other questions:

You say there are multiple things that could be causing the java.lang.OutOfMemoryError: Java heap space error. Do they all eventually lead to "websocket disconnected" clients? which in turn cause the Java heap space error? If so, I can monitor our UAT environment and see if there are any "websocket disconnected" clients.

I'm not convinced that it is programming errors in our solution code, only because we weren't encountering the error with a 2020.06 .war file... unless there are changes within Servoy between 2020.06 and 2022.09 that would 'expose' such solution code and cause the error. Perhaps that's what you're referring to with the fixes in 2022.12?

Additionally, I've uploaded the relevant logging from our servoy application log file in the lead up to the OOM - Java heap space error, in case there are some clues. There are certainly some log messages that I haven't seen before. Perhaps you could take a quick look.
Thanks again.

servoy-application-log.txt

(1.28 MiB) Downloaded 48 times

[jcompagner]

there can be hunderds of causes that will end up in OOM..
i have seen many, like big report generation, huge files loaded in to memory and then that intermediate result of "websocket disconnected" can also be stuff like databasetransactions in deadlock blocking stuff, never ending loops in code, code deadlocks (this last thing is something servoy must fix), websocket/browser connections errors that results in a very weird state at the server (this is a fix that is in .12 and in the next 03.LTS) and then in the end you OOM out because the clients are never shutdown.

there are some very weird exceptions in that log:

2022-12-09 11:47:46,018 ERROR [Executor,uuid:F252A3E:2, clientid: 3ED40EB9-9CF4-4D29-8DAA-EF03F92E5C65] org.sablo.websocket.utils.JSONUtils - unsupported value type:null for value: QBSelect(SELECT columns FROM db:/qfi/v_deal_progress where XXXX)
at org.sablo.websocket.utils.JSONUtils$FullValueToJSONConverter.toJSONValue(JSONUtils.java:479) ~[sablo_2022.9.2.3805.jar:?]
at org.sablo.websocket.utils.JSONUtils$FullValueToJSONConverter.toJSONValue(JSONUtils.java:1) ~[sablo_2022.9.2.3805.jar:?]
at org.sablo.websocket.utils.JSONUtils.writeData(JSONUtils.java:101) ~[sablo_2022.9.2.3805.jar:?]
at org.sablo.websocket.BaseWindow.sendMessageInternal(BaseWindow.java:611) ~[sablo_2022.9.2.3805.jar:?]
at org.sablo.websocket.BaseWindow.sendAsyncMessage(BaseWindow.java:509) ~[sablo_2022.9.2.3805.jar:?]

for some reason it tries to push of a query to the client.. i have really no idea how that is possible. that should never happen in the components and services we ship

the rest really seems to be a lot of users that are pressing a button, changing data, showing form in a dialog, so really interacting with the server, but then just close the browser or tab, a lot of exceptions that the websocket is disconnected in the middle of calls..

[darren]

The QBSelect exception is something we've had for awhile and tried to work through with Servoy support, but never got anywhere and since it the form was working correctly, we chose to ignore it. Your comment on it today prompted us to revisit the exception, and we found two ways to avoid the exception... there's something weird going on there. I'll get around to writing it up one day.

I'm guessing the websocket disconnect exceptions at the end of the file were causes by the imminent OOM.

We've decided to go ahead and deploy the problem 2022.09 .war file sometime soon and try to capture the required memory dump. Thanks.