We still have the same messages, but fewer of them. About 30 Smart Clients connected.
There is another issue, the Server claims to run out of heap memory and restarts about once a day.
- Code: Select all
INFO | jvm 1 | 2017/12/06 09:24:33 | Dec 06, 2017 9:24:33 AM org.apache.coyote.http11.Http11Processor service
INFO | jvm 1 | 2017/12/06 09:24:33 | INFO: Error parsing HTTP request header
INFO | jvm 1 | 2017/12/06 09:24:33 | Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
INFO | jvm 1 | 2017/12/06 09:24:33 | java.lang.IllegalArgumentException: Invalid character found in method name. HTTP method names must be tokens
INFO | jvm 1 | 2017/12/06 09:24:33 | at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:417)
INFO | jvm 1 | 2017/12/06 09:24:33 | at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:667)
INFO | jvm 1 | 2017/12/06 09:24:33 | at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
INFO | jvm 1 | 2017/12/06 09:24:33 | at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:789)
INFO | jvm 1 | 2017/12/06 09:24:33 | at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1437)
INFO | jvm 1 | 2017/12/06 09:24:33 | at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
INFO | jvm 1 | 2017/12/06 09:24:33 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
INFO | jvm 1 | 2017/12/06 09:24:33 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
INFO | jvm 1 | 2017/12/06 09:24:33 | at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
INFO | jvm 1 | 2017/12/06 09:24:33 | at java.lang.Thread.run(Thread.java:748)
INFO | jvm 1 | 2017/12/06 09:24:33 |
INFO | jvm 1 | 2017/12/06 10:26:06 | Dec 06, 2017 10:26:06 AM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
INFO | jvm 1 | 2017/12/06 10:26:06 | WARNING: RMI TCP Accept-1099: accept loop for ServerSocket[addr=null,localport=0] throws
INFO | jvm 1 | 2017/12/06 10:26:06 | java.lang.OutOfMemoryError: unable to create new native thread
STATUS | wrapper | 2017/12/06 10:26:06 | Filter trigger matched. Restarting JVM.
INFO | jvm 1 | 2017/12/06 10:26:06 | at java.lang.Thread.start0(Native Method)
INFO | jvm 1 | 2017/12/06 10:26:06 | at java.lang.Thread.start(Thread.java:717)
INFO | jvm 1 | 2017/12/06 10:26:06 | at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
INFO | jvm 1 | 2017/12/06 10:26:06 | at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
INFO | jvm 1 | 2017/12/06 10:26:06 | at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:415)
INFO | jvm 1 | 2017/12/06 10:26:06 | at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
INFO | jvm 1 | 2017/12/06 10:26:06 | at java.lang.Thread.run(Thread.java:748)
INFO | jvm 1 | 2017/12/06 10:26:06 |
INFO | jvm 1 | 2017/12/06 10:26:06 | java.lang.reflect.InvocationTargetException
INFO | jvm 1 | 2017/12/06 10:26:06 | at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.continueAfterAcceptFailure(TCPTransport.java:499)
INFO | jvm 1 | 2017/12/06 10:26:06 | at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:474)
INFO | jvm 1 | 2017/12/06 10:26:06 | at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
INFO | jvm 1 | 2017/12/06 10:26:06 | at java.lang.Thread.run(Thread.java:748)
INFO | jvm 1 | 2017/12/06 10:26:06 | Caused by: java.lang.OutOfMemoryError: unable to create new native thread
STATUS | wrapper | 2017/12/06 10:26:06 | Filter trigger matched. Restarting JVM.
INFO | jvm 1 | 2017/12/06 10:26:06 | at java.lang.Thread.start0(Native Method)
INFO | jvm 1 | 2017/12/06 10:26:06 | at java.lang.Thread.start(Thread.java:717)
INFO | jvm 1 | 2017/12/06 10:26:06 | at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
INFO | jvm 1 | 2017/12/06 10:26:06 | at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
INFO | jvm 1 | 2017/12/06 10:26:06 | at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:415)
INFO | jvm 1 | 2017/12/06 10:26:06 | ... 2 more
INFO | jvm 1 | 2017/12/06 10:26:06 | Exception in thread "MessageScheduler"
This would never happen to the 7.4.x server and has not happened to a 8.2.1 server I have running in Windows 2008. I allocated 3GB, but it does not help, we never see the heap go that high.
Then I read you would get the exactly same error if the computer runs out of threads. And allocating lots of heap memory makes the problem slightly worse, so reduced heap memory to 2.5GB last night. Right now the network settings are:
- Code: Select all
java.rmi.server.hostname: 127.0.0.1
servoy.rmiStartPort: 1099
rmi.connection.timeout: 60
ApplicationServer.pingDelay: 60
SocketFactory.tunnelConnectionMode: http&socket
SocketFactory.compress: Yes
SocketFactory.useSSL: No
SocketFactory.tunnelUseSSLForHttp: Yes
On a Mac you can run
- Code: Select all
sysctl kern.num_threads
sysctl kern.num_taskthreads
to see the maximum number of threads and threads per task.
On our Mac Mini it comes back as 10240 and 2048. My MacBook Pro has 20480 and 4096.
I'm trying to work out what is happening on the Servoy server just before it falls over...