Batch processor java.lang.OutOfMemoryError

[swingman]

Hi all,

Servoy version 2022.12.1 -releaseNumber 3824 NG Client. Ubuntu

We have a batch processor mostly pushing and pulling data via web services. It grids to a halt after a few hours.

Code: Select all

09:42 scheduler_Worker-1 ERROR org.quartz.core.ErrorLogger Job (A4FC88E3-E7B4-4CA8-94F5-C34370600933.ticker threw an exception.

java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
at java.lang.Thread.start0(Native Method) ~[?:?]
at java.lang.Thread.start(Thread.java:802) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1353) ~[?:?]
at com.servoy.j2db.plugins.ClientPluginAccessProvider.executeMethod(ClientPluginAccessProvider.java:466) ~[j2db.jar:2022.12.1.3824]
at com.servoy.extensions.plugins.scheduler.ExecuteScriptMethodJob.execute(ExecuteScriptMethodJob.java:74) ~[scheduler.jar:?]
at org.quartz.core.JobRunShell.run(JobRunShell.java:202) ~[quartz.jar:?]
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) ~[quartz.jar:?]


Running

Code: Select all

ps -eLf

Reveals lots of lines like these

Code: Select all

tomcat   2144037       1 2147067  0  267 21:02 ?        00:00:00 /usr/lib/jvm/java-17-openjdk-amd64/bin/java -Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoader
tomcat   2144037       1 2147068  0  267 21:02 ?        00:00:00 /usr/lib/jvm/java-17-openjdk-amd64/bin/java -Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoader
tomcat   2144037       1 2147157  0  267 21:03 ?        00:00:00 /usr/lib/jvm/java-17-openjdk-amd64/bin/java -Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoader
tomcat   2144037       1 2147158  0  267 21:03 ?        00:00:00 /usr/lib/jvm/java-17-openjdk-amd64/bin/java -Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoader
tomcat   2144037       1 2147159  0  267 21:03 ?        00:00:00 /usr/lib/jvm/java-17-openjdk-amd64/bin/java -Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoader
tomcat   2144037       1 2147286  0  267 21:06 ?        00:00:00 /usr/lib/jvm/java-17-openjdk-amd64/bin/java -Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoader

Any ideas what may cause this?

[jcompagner]

no you can only know that if you see what is eating up the memory. so we need a heap dump when it happens to analyze such a thing.

"unable to create native thread" is sometimes also the problem when running 32 bits java, but you don't do that i see because i see amd64 so that is not it.

it could be that it just needs a bit more memory, so what is the max you give it?

[swingman]

Here is what I see in Servoy Admin:

[swingman]

How do I create a heap-dump?
There is a stack-dump option on the server-console. Is that it?
And is it safe to do while clients are logged in?

[jcompagner]

to get it when an out of mem happens you need to pass 1 or 2 arguments:

-XX:+HeapDumpOnOutOfMemoryError and then optionally: -XX:HeapDumpPath=/path/to/ to point to a path where it can write

you can also use "jmap" tools or something like that to trigger a heap dump when running (when you see "used is going more to the max)

the stack dump option on admin page is not a heap dump that is just dumping the current thread stacks, not the heap

[swingman]

Thanks for this, I just discovered that this may be caused by trying to push data to a web-service that is being throttled.
If you hit the other server too often it responds slower and slower until it does not respond at all.

I had forgotten to whitelist our new cloud-based Servoy server ;-(