OutOfMemory-Exception with Heap memory

Hi

We’ve got memory issues with one of our application.

It’s a NG application, that is used from public users (no authentication needed). Usually there are about 10 to 50 clients running. An average session runs about 5 minutes and then often the session remains another 10 minutes because the user doesn’t exit the application. So the Servoy “plugin.svyIdle”-Plugins will kick out the user after 10 minutes of inactivity. This all works very well, except, that the usage of “heap memory” increases until we get “OutOfMemoryError: Java heap space”. Even if we remove all clients, the memory usage remains. The only solution then is to restart the Tomcat-server.

I don’t know what leaks. My assumptions:

  • InMemory-Foundset (we use just two quit small InMemory-Foundsets with less then 50 Records!)
  • Scheduler-Plugin
  • ???

Our Environment:

  • Windows-Server 2019 (Windows 10)
  • OpenJDK 64-Bit 14.0.2
  • Servoy 2020.03 (Built 3562)
  • max. JavaHeap space configured 12GB!!! (we lose about 2 GB per day, so the server must be rebooted after at least one week)

I will add a dump from the servoy admin page.

Any ideas?
Thanks in advance for any suggestions
Roland

dump.txt (54.8 KB)

to understand this you need to make a memory dump (with yourkit or another profile tool)
and it would be nice if the dump could be made after a few days (when it did grow) and at that time almost had no clients then we really can see what keeps hanging around

Hi Johan

Thanks for your reply. I’ll do that and send the dump in the next few days, as soon as I got one.

Hi Johan

I’ll send you a PM with the link to download the memory dump.

Just for the sake of completeness: as Johan confirmed the reason is an issue with the InMemory database (HSQLDB) in Servoy versions prior 2021.03 LTS. The memory is not cleaned up propperly at the end of the session.

So the solution to our problem would be:
a) update to Servoy 2021.03 LTS
OR
b) replace InMemory foundset with datasets

Hi, everybody for us same problem but the update to Servoy 2021.03 and beyond 2021.09 it does not work.
I can know what command you ran to dump the memory from linux?

I added these parameters when launching the tomcat application but it does not create the dump file for me
Environment=‘CATALINA_OPTS=-Xms16G -Xmx24G -server -XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/dump_memory

what do you mean it doesn’t not work (.12.2 or the latest 03 LTS)?
You mean you upgraded it and you still have out of memory after a few days? so the memory slowly keeps on growing and you get an out of mem exception in the log ?
those flags (XX) seems fine (if tomcat process can write in that dir)

Else you need to manually trigger a memory dump with the various tools like yourkit or jmap

yes now we have several server with this version
Servoy version 2021.12.1 -releaseNumber 3723 the memory continues to grow up to the classic out of memory.

the same server are VM cloned with Servoy version 2020.6.0 -releaseNumber 3582 has a regular and constant memory.

if the parameters I enter to get the dump go veins it could then be that tomcat cannot write to that folder in fact I could try to change path.
if I don’t put any path parameter where does the dump write by default?
thanks a lot this problem is consuming us :)

if nothing is specified i think it just starts to dump it in the “user.dir” so thats the start dir of the application.

ah but you are using a 12.1 and i guess you have profiling enabled on the admin page? there was a memory leak fixed in 12.2: https://wiki.servoy.com/display/DOCS/2021.12
upgrade to 12.2 or make sure you disable in production all your profiling (client and database)

I don’t remember ever activating a profiles in servoy admin … this is this page here attached. I don’t think there is anything active

not profiles but profiling, so the 2 latest items in the list “Database performance” and “Client performance”

Are you talking about these parameters? This is our actual configuration. Should we change this to “disabled”?
[attachment=1]Screenshot_1.png[/attachment]
[attachment=0]Screenshot_2.png[/attachment]

yes or update to 2021.12.2

We’ll try and let you know if it will solve the problem.

Hello
We are having the same issue with Servoy 2021.12.2 with the WAR and tomcat 9.x and with the MariaDB 10.x database engine.
The memory is not being freed and it causes the application server to crash, this happens after a few days and we have to restart the server.
We are following this thread of conversation, we await news.
Greetings from Argentina.

does killing the profiling (performace stuff) so set it to 0 in production make it better?

We can’t do anything except when we get a memory dump before it started to crash to be able to see what is really in memory.

Hi
We have set the values to 0
As soon as the problem arises, make a memory dump and attach it here
Thanks
Roberto

Disabling the profiling hadn’t solved the problem.
In the next few days we will try upgrading to 2021.12.2 and let you know.

rmidata:
Hi
We have set the values to 0
As soon as the problem arises, make a memory dump and attach it here
Thanks
Roberto

if you can create memory dumps, do 7zip them (7zip.org) then almost nothing is left.

Hi Johan
I think we were just able to do the crash dump
The server was working 5 days and the memory was full
Attached the memory dump
Thanks
Roberto

memory_dump.txt (664 KB)