Servoy upgrade and RAM requirements

[darren]

Hi,

We're in the process of upgrading from 2020.06 to 2023.03-LTS. For Production, we run a load-balanced environment with two application (virtual) servers, each with their own database (virtual) server. I'm currently running a test in our Production environment to compare the two versions. Specifically for the application servers, here is our current application deployment details:

Node 1:
Tomcat 8.5
Java 11
Servoy 2020.06
Basic_Servoy_JasperReports_Plugin_v.6.12.2_2020.03

Node 2:
Tomcat 8.5
Java 17
Servoy 2023.03-LTS
Basic_Servoy_JasperReports_Plugin_v.2022.3_6.19.1

But the same application code (Servoy forms, Jasper Report .jrxml's and Java based business logic code) is currently being run on these two nodes. A difference would be the Java code on 2023.03-LTS is compiled against Java 17.

We've noticed a significant increase in RAM required to run the 2023.03-LTS version of Servoy and we're wondering if anyone else has noticed the same?

After deployment of a .war file, which occurs in the evening when we have maybe one or two users online, we always restart Tomcat. After the Tomcat restart, the 2023.03-LTS server is immediately using way more memory than 2020.06, with no clients connected.

Once the workday begins, the 2020.06 server memory usage climbs steadily and will gain around 8% over the workday, with no risk of memory troubles. 2023.03-LTS will climb quickly and drop quickly, with an overall climb that sometimes reaches 98% of the virtual server's total memory. Now, we've not yet had an outage due to memory issues, as the GC appears to be keeping things in check (somewhat), but when approaching the high 90's, I think we're asking for trouble. Of course we can always throw more memory at it, but I'd like to hear what others have to say. More RAM = more $ and it looks like our costs will increase substantially.

Above, I've mentioned the use of Jasper Reports here as well, since we've also noticed the generation of a Jasper Report PDF from within the 2023.03-LTS application appears to use a lot more memory than in 2020.06. Generating a single PDF can make the MEM% (as viewed in htop on our Linux server) jump 0.5 to 5% on the 2023.03-LTS application. Eventually some of that memory is reclaimed, but I get the sense that if too many users try to generate a PDF around the same time our RAM usage % is in the 90's, we're risking an outage.

By comparison, we rarely see any jump in memory on 2020.06 when generating the same report (exact same .jrxml files). If there is a change, it is 0.1%

Hoping some of you can share what you've experienced with both the upgrade to the newer versions of Servoy and also with Jasper Reports.

Thanks.
Darren

[mboegem]

Hi Darren,

to get the complete picture: what type of client are you running (SC/WC/(Ti)NG)?
Also did you switch client type from 2020 to 2023?

[darren]

Hi Marc - sorry for not mentioning. NG Client running for both 2020.06 and 2023.03-LTS --> no change in client type yet. Just trying to get the first step stable, then we'll look at the move to Titanium.

[mboegem]

Hi Darren,

I had an issue using http clients which we initialised over and over again without ever closing a client.
The footprint for these clients has become way bigger as a result of the new http client (not sure in which version it was introduced, but definitely part of 2023.3)
Is this something you might be using?

[darren]

We have some newer code that is not yet widely used (if at all, tbh) that isn't being closed:

var oListHTTPClient = plugins.http.createNewHttpClient( );

So thanks for that reminder, it will be fixed. But that code is form specific, initiated by users, so it wouldn't explain the significantly higher initial memory usage right after starting Tomcat. Plus the memory jumps that we see due to Jasper Reports.

[sbutler]

My hunch is this probably isn't something you shouod worry about unless you're actually seeing performance problems. Java version changed deim 11 to 17, so there is likely also a difference in how it handles garbage collection. Java tends to be lazy about reclaiming memory and varies based on hoe the jvm is tuned. So, I'd suggest hammering the server over and over and seeing if it handles it. I'd guess memory probably peaks and then it starts garbage collection and continues to handle the load.

[darren]

Thanks for your thoughts, Scott.

Yes, we've certainly changed the garbage collector moving to Java 17. Tomcat's been running for a little over a week now and we haven't had a crash, so the GC seems to be doing the job. I've logged in a few times during the high memory usage periods and the application seems a little slower in general, but that could be my perception, my home internet... or who knows.

We've had some users report that the only get a blank screen in their browser when trying to hit the login screen. In most cases, restarting the browser fixes the problem. In a few cases we've had to have them clear their browser cache before they're able to get to the login. This problem only arose after we deployed 2023.03-LTS to Node 2, however we've not been able to determine if Node 2 is the culprit in these instances. Perhaps we could find out if we dig deeper (get their IP address, ask our hosting company to check which Node that IP was sent to through the load balancer...).

Mainly I was hoping to hear from others if they also needed to increase RAM after making the switch to the newer versions of Servoy. And are the memory usage jumps (0.5 to 5%) from Jasper Report generation to be expected with the newer version?

We're a small company trying to keep costs down and the increase in RAM has raised our hosting costs > 10%, and we haven't even moved to TiNG yet. Should we expect to have to increase RAM even more with that transition?

[sean]

Hi Darren, I would not expect to see any increase in RAM usage by moving from NG1 to NG Titanium when all else stays the same.
That is largely a change in the code base which is sent to the browser client.

Best,
Sean

[robert.edelmann]

Do you have insights into the internal management of the ram for tomcat?

I was under the impression that tomcat allocates much of the memory upfront and G1GC handles from there. (see https://wiki.servoy.com/display/DOCS/Memory+Management).

This looks like this in our environment:

CleanShot 2024-03-19 at 12.54.20.png (135.88 KiB) Viewed 973 times

Each cycle represents about 25 Minutes.

[darren]

We'll need to dive in further when we have time, especially for the suspected Jasper Report memory usage. For now, we've been running on Production for a few weeks without any crashes. According to our resource monitor, RAM utilization percentage does sometimes peak in the mid to upper 90's. Today's plot is attached (RAM is the blue line). So the garbage collector (ZGC in our case) seems to be doing the job. The overall goal is to see if we can reduce the RAM we're allocating to our virtual server, to reduce costs - is it even worth the time (rhetorical question, that). Thanks to all for the replies.