Page 1 of 2

War Deployment suddenly failing 2020.03.01 LTS

PostPosted: Sat Sep 05, 2020 1:30 pm
by swingman
Hi all,

I upgraded a system to 2020.03.01 LTS and now my solution refuses to deploy on Tomcat 9 Ubuntu. I upload the WAR file via the Tomcat manager, but the manager page never refreshes after deployment. I can see the war has been unpacked when I look in the web apps directory.
The syslog shows:
Code: Select all
05-Sep-2020 11:10:01.160 INFO [https-openssl-nio-8443-exec-5] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/var/lib/tomcat9/webapps/myappname.war]
05-Sep-2020 11:10:03.324 INFO [https-openssl-nio-8443-exec-5] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.


The same happens on 2020.06.01...

Re: War Deployment suddenly failing 2020.03.01 LTS

PostPosted: Sat Sep 05, 2020 3:20 pm
by swingman
I have re-deployed my 2020.03 for now while investigating the issue...

Re: War Deployment suddenly failing 2020.03.01 LTS

PostPosted: Tue Sep 08, 2020 12:53 pm
by Andrei Costescu
Check both tomcat logs and servoy logs if they were created (by default in [user_home_dir]/.servoy/server/[your_context_name]/...).

Re: War Deployment suddenly failing 2020.03.01 LTS

PostPosted: Tue Sep 08, 2020 7:23 pm
by robert.edelmann
I have similar problems with 2020.6.1

  • deploying war without solution works
  • importing the solution seems to work, but can't be opened
  • tomcat-manager won't respond

This is only the case with the ng-client-version of our app, the old web client works without problems.

I attached both log files.

Re: War Deployment suddenly failing 2020.03.01 LTS

PostPosted: Tue Sep 08, 2020 7:37 pm
by jcompagner
we need if possible stackdumps of that (if the admin page is accessible do it from there, else you need a tool like jstack or jvisualvm)

the only thing is see in the servoy log that index creation are going wrong (of the i18n message table we try to make an index called: i18n_messages_m_l)
(and that shouldn't really matter, can be ignored if its already there)

Re: War Deployment suddenly failing 2020.03.01 LTS

PostPosted: Tue Sep 08, 2020 9:52 pm
by robert.edelmann
I can't get jstack to connect to the jvm on the ubuntu-system, but luckily I have a windows machine with the same symptoms. I hope that works as well...

Re: War Deployment suddenly failing 2020.03.01 LTS

PostPosted: Tue Sep 08, 2020 10:08 pm
by jcompagner
that looks like this: https://support.servoy.com/browse/SVY-15236

is fixed for 2020.9_rc, please test this release to see if this helped.

Re: War Deployment suddenly failing 2020.03.01 LTS

PostPosted: Tue Sep 08, 2020 10:23 pm
by jcompagner
for one that are on the lts release, the same patch is also applied to that branch, so who is on the latest lts branch installation can use : https://build.servoy.com/latest/servoy_ ... update.zip

Re: War Deployment suddenly failing 2020.03.01 LTS

PostPosted: Tue Sep 08, 2020 11:39 pm
by swingman
Thank you for looking into this, I tried patching my 2020.03.1 LTS by dropping in the files from https://build.servoy.com/latest/servoy_ ... update.zip, but deploy failed again.
This may be my patching going wrong, I did not see anything like this in the catalina logs when my deploy was failing.

Code: Select all
08-Sep-2020 21:13:20.880 INFO [https-openssl-nio-8443-exec-14] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/var/lib/tomcat9/webapps/mysolution.war]
08-Sep-2020 21:13:20.884 WARNING [https-openssl-nio-8443-exec-14] org.apache.catalina.startup.SetContextPropertiesRule.begin [SetContextPropertiesRule]{Context} Setting property 'clearReferencesStatic' to 'true' did not find a matching property.
08-Sep-2020 21:13:22.051 WARNING [https-openssl-nio-8443-exec-14] org.apache.tomcat.util.descriptor.web.WebXml.orderWebFragments Used a wrong fragment name [log4j] at web.xml absolute-ordering tag!
08-Sep-2020 21:13:22.800 SEVERE [https-openssl-nio-8443-exec-14] org.apache.catalina.startup.HostConfig.deployWAR Error deploying web application archive [/var/lib/tomcat9/webapps/mysolution.war]
java.lang.IllegalStateException: Error starting child
   at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:716)
   at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690)
   at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:695)
   at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:978)
   at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:493)
   at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1643)
   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   at org.apache.tomcat.util.modeler.BaseModelMBean.invoke(BaseModelMBean.java:289)
   at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:809)
   at java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
   at org.apache.catalina.manager.ManagerServlet.check(ManagerServlet.java:1584)
   at org.apache.catalina.manager.HTMLManagerServlet.upload(HTMLManagerServlet.java:294)
   at org.apache.catalina.manager.HTMLManagerServlet.doPost(HTMLManagerServlet.java:212)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
   at org.apache.catalina.filters.CsrfPreventionFilter.doFilter(CsrfPreventionFilter.java:136)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
   at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:607)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
   at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
   at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
   at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
   at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
   at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415)
   at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
   at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/mysolution]]
   at org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:441)
   at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:198)
   at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:713)
   ... 42 more
Caused by: java.lang.NoClassDefFoundError: org/json/JSONObject
   at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
   at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3166)
   at java.base/java.lang.Class.getDeclaredMethods(Class.java:2309)
   at org.apache.catalina.util.Introspection.getDeclaredMethods(Introspection.java:133)
   at org.apache.catalina.startup.WebAnnotationSet.loadMethodsAnnotation(WebAnnotationSet.java:285)
   at org.apache.catalina.startup.WebAnnotationSet.loadApplicationFilterAnnotations(WebAnnotationSet.java:110)
   at org.apache.catalina.startup.WebAnnotationSet.loadApplicationAnnotations(WebAnnotationSet.java:68)
   at org.apache.catalina.startup.ContextConfig.applicationAnnotationsConfig(ContextConfig.java:330)
   at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:771)
   at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:301)
   at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
   at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5034)
   at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
   ... 43 more
Caused by: java.lang.ClassNotFoundException: org.json.JSONObject
   at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1363)
   at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1186)
   ... 56 more

08-Sep-2020 21:13:22.807 INFO [https-openssl-nio-8443-exec-14] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/var/lib/tomcat9/webapps/mysolution.war] has finished in [1,927] ms
08-Sep-2020 21:13:36.980 SEVERE [https-openssl-nio-8443-exec-23] org.apache.catalina.startup.ExpandWar.copy Error copying [/tmp/0-mysolution] to [/tmp/1-mysolution]
java.io.FileNotFoundException: /tmp/0-mysolution (No such file or directory)
   at java.base/java.io.FileInputStream.open0(Native Method)
   at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
   at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
   at org.apache.catalina.startup.ExpandWar.copy(ExpandWar.java:276)
   at org.apache.catalina.startup.ContextConfig.antiLocking(ContextConfig.java:709)
   at org.apache.catalina.startup.ContextConfig.beforeStart(ContextConfig.java:747)
   at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:303)
   at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
   at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:424)
   at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:182)
   at org.apache.catalina.manager.ManagerServlet.start(ManagerServlet.java:1409)
   at org.apache.catalina.manager.HTMLManagerServlet.start(HTMLManagerServlet.java:698)
   at org.apache.catalina.manager.HTMLManagerServlet.doPost(HTMLManagerServlet.java:223)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
   at org.apache.catalina.filters.CsrfPreventionFilter.doFilter(CsrfPreventionFilter.java:136)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
   at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:607)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
   at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
   at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
   at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
   at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
   at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415)
   at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
   at java.base/java.lang.Thread.run(Thread.java:834)

Re: War Deployment suddenly failing 2020.03.01 LTS

PostPosted: Wed Sep 09, 2020 11:30 am
by Andrei Costescu
Is your Servoy installation on Win, Linux or Mac? (applying update zip on Mac is a bit special)
When you dropped the update zip on top of existing installation you did it so that it overwrites existing files right?

After you do that you have to re-export the war from developer.

Also try to clean/undeploy failed war deployments on that (Ubuntu) Tomcat before deploying the new war.

Re: War Deployment suddenly failing 2020.03.01 LTS

PostPosted: Wed Sep 09, 2020 12:04 pm
by swingman
Andrei Costescu wrote:Is your Servoy installation on Win, Linux or Mac? (applying update zip on Mac is a bit special)
When you dropped the update zip on top of existing installation you did it so that it overwrites existing files right?


I unpacked and replaced the application_server beans, lib and plugins directories with the ones in the zip (Thinking now that maybe I have accidentally deleted some files which were not updated by doing it this way).
Then I dropped the developer/plugins into Eclipse.app/Contents/Eclipse/plugins.
I will try it again now in a way which does not overwrite directories.

Andrei Costescu wrote:After you do that you have to re-export the war from developer.


Yes, I did.

Andrei Costescu wrote:Also try to clean/undeploy failed war deployments on that (Ubuntu) Tomcat before deploying the new war.


I un-deployed, but did not check the actual directory or .war had been cleaned out.

Re: War Deployment suddenly failing 2020.03.01 LTS

PostPosted: Wed Sep 09, 2020 12:41 pm
by swingman
My developer now says "2020.3.1.3563_LTS" after my second attempt at patching.
The solutions runs fine via Developer, but when deploying on Tomcat I get the same
Code: Select all
09-Sep-2020 10:29:24.544 INFO [https-openssl-nio-8443-exec-28] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/var/lib/tomcat9/webapps/mysolution.war]
09-Sep-2020 10:29:26.841 INFO [https-openssl-nio-8443-exec-28] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.


in the Catalina logs.

I'm thinking of restoring the 2020.03 version to get an update to my solution deployed.

Re: War Deployment suddenly failing 2020.03.01 LTS

PostPosted: Wed Sep 09, 2020 12:48 pm
by jcompagner
We need a stack dump like the one of above

Re: War Deployment suddenly failing 2020.03.01 LTS

PostPosted: Wed Sep 09, 2020 12:55 pm
by Andrei Costescu
I have to check something - one more fix might be needed for that case. But a new stack dump can confirm where the problem is yes.

Re: War Deployment suddenly failing 2020.03.01 LTS

PostPosted: Wed Sep 09, 2020 1:56 pm
by robert.edelmann
I tried upgrading to 2020.09rc, and my problems persisted.

I'm in contact with Patrick Ruhsert, he got the attached stackdumps + a backup of the repository database, he sad someone is looking into it, but I assume you are in contact?