Server doesn't start - 2023.03 LTS war file

[darren]

Hi. We're still in the middle of the (long) process of moving from 2020.06 to 2023.03 LTS. We're updating our virtual servers (DEV, UAT and production) environments at the same time, to accommodate the requirements of this Servoy upgrade (Java 11 to Java 17) and to also move forward our operating systems, tomcat (8.5 to 9) and MySQL versions.

Our DEV environment move is complete and seems to be working well. Initially deployed a 2020.06 .war file for our application and that worked. Then, after some trouble exporting a .war file in 2023.03 LTS, we successfully exported and deployed it in DEV.

Next, on to our UAT environment. New virtual server built with all the software upgrades. Deployed the 2020.06 .war file and the application worked well. Now, the 2023.03 LTS .war file won't deploy.

I've checked out https://wiki.servoy.com/display/DOCS/Tr ... esn'tStart and the only cause that stands out might be:

Invalid configuration of the database server connection named 'repository_server' in Servoy

I say this, because we initially get this error:

2024-01-09 16:45:12,098 WARN [ServoyServerScheduler-1-thread-2] servoy.DBIProvider - server dbi file '/WEB-INF/db/repository_server.dbi' not found

However, after comparing the UAT problems to the DEV environment, I see that same warning message there, and the application works fine in DEV. Plus, the repository_server is configured like this:

server.1.URL=jdbc\:hsqldb\:mem\:repository_server

Maybe we don't need the repository_server.dbi file anyways?

So, on to the next log messages (trimmed here, but fully included in attached file):

2024-01-09 16:45:12,403 ERROR [ServoyServerScheduler-1-thread-2] persistence.Server - Final get connection failure for server qfi in 1 times [ ]
2024-01-09 16:45:13,189 ERROR [main] com.servoy.j2db.util.Debug - Server 'qfi' does not exist. Import failed. [ ]
2024-01-09 16:45:13,922 ERROR [main] com.servoy.j2db.util.Debug - Could not load solutions [ ]
com.servoy.j2db.persistence.RepositoryException: Cannot find database server "qfi", import failed.
2024-01-09 16:45:13,943 ERROR [main] com.servoy.j2db.util.Debug - Servoy Application server context /qfi didn't start up correctly, very likely because of an import problem [ ]
com.servoy.j2db.persistence.RepositoryException: Cannot find database server "qfi", import failed.
2024-01-09 16:45:13,944 ERROR [main] com.servoy.j2db.util.Debug - Can't start the application server [ ]
javax.servlet.ServletException: Can't start the application server

Going back to the possible causes in the troubleshooting page, the MySQL instance is running on the same virtual server and is confirmed to be running. The database 'qfi' is certainly available to be found. I don't see the JDBC connection being an issue, as it is unchanged from our 2020.06 version of the application (which worked in UAT), and it's working in DEV for the 2023.03 LTS version of the application.

I've attached relevant log file snippets from catalina.out, localhost.log and our application log (named servoy.log) in a single file.

Any help, as always, is much appreciated.

[darren]

A couple other notes as I continue investigating...

I see that there is no solution.servoy file in the /home/servoy/.servoy/server/qfi/ folder. During the unpacking of the .war file, solution.servoy is placed there briefly, but then removed immediately. I'm guessing that's a symptom of the underlying problem, not the cause.

As well, on the DEV server, I'm able to switch back and forth between the 2020.06 and 2023.03 .war file (for the same qfi context) without issue. However on UAT, now that the 2023.03 .war file has failed in deployment, the 2020.06 .war file will not deploy either. Something in the 2023.03 deployment messed up the access to the database server.

[darren]

still trying to resolve this. Looks like a jdbc connection/SSL issue...

Originally using this in servoy.properties/servoy_server.properties (which worked in 2020.06 for 3+ years):

server.0.URL=jdbc\:mysql\://localhost\:3306/qfi

Then changed it to this and the application works:

server.0.URL=jdbc\:mysql\://localhost\:3306/qfi?useSSL\=false

But, want to use SSL, so I've tried numerous variations of this definition:

server.0.URL=jdbc\:mysql\://localhost\:3306/qfi?useSSL\=true&sslMode\=VERIFY_IDENTITY&trustCertificateKeyStoreUrl\=file\:/opt/tomcat9/conf/demo2.oursoftware.com&trustCertificateKeyStorePassword\=XXXXX&enabledTLSProtocols\=TLSv1.2

But no luck so far. Does anyone think this is the right path to a solution?

Logs are similar to those posted originally.

catalina.out:
18-Jan-2024 17:29:11.646 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
18-Jan-2024 17:29:11.647 SEVERE [main] org.apache.catalina.core.StandardContext.startInternal Context [/qfi] startup failed due to previous errors

localhost.log:
18-Jan-2024 17:29:11.383 SEVERE [main] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.servoy.j2db.server.servlets.WarClientServletContextListener]
java.lang.RuntimeException: Error starting application server
at com.servoy.j2db.server.servlets.WarClientServletContextListener.contextInitialized(WarClientServletContextListener.java:190)
Caused by: javax.servlet.ServletException: Can't start the application server
at com.servoy.j2db.server.servlets.WarClientServletContextListener.startAppServer(WarClientServletContextListener.java:91)
Caused by: com.servoy.j2db.persistence.RepositoryException: Cannot find database server "qfi", import failed.

application.log:
2024-01-18 17:29:09,678 WARN [ServoyServerScheduler-1-thread-2] servoy.DBIProvider - server dbi file '/WEB-INF/db/repository_server.dbi' not found [ ]
2024-01-18 17:29:09,943 ERROR [ServoyServerScheduler-1-thread-2] persistence.Server - Final get connection failure for server qfi in 1 times [ ]
2024-01-18 17:29:10,645 ERROR [main] com.servoy.j2db.util.Debug - Server 'qfi' does not exist. Import failed. [ ]
2024-01-18 17:29:11,359 ERROR [main] com.servoy.j2db.util.Debug - Could not load solutions [ ]
com.servoy.j2db.persistence.RepositoryException: Cannot find database server "qfi", import failed.
2024-01-18 17:29:11,381 ERROR [main] com.servoy.j2db.util.Debug - Servoy Application server context /qfi didn't start up correctly, very likely because of an import problem [ ]
com.servoy.j2db.persistence.RepositoryException: Cannot find database server "qfi", import failed.
2024-01-18 17:29:11,382 ERROR [main] com.servoy.j2db.util.Debug - Can't start the application server [ ]
javax.servlet.ServletException: Can't start the application server
Caused by: com.servoy.j2db.persistence.RepositoryException: Cannot find database server "qfi", import failed.

[omar]

Hi Darren,

I have had a similar issue with SQL Server when I updated to the Servoy version that uses the latest jdbc driver. The solution was to add the following to the database connection:

Code: Select all

TrustServerCertificate=true

Hope this helps.

[darren]

Hi Omar. I appreciate your suggestion. It does look like that parameter is specific to SQL Server (or at least not used for MySQL) and didn't make a difference for me. But that did prompt me to look further at the MySQL connection parameters and I've removed the ones that were introduced in a newer version of the mysql connector (we're still using mysql-connector-java-5.1.41-bin.jar)

So my current attempt is

server.0.URL=jdbc\:mysql\://localhost\:3306/qfi?useSSL\=true&trustCertificateKeyStoreUrl\=file\:/opt/tomcat9/conf/demo2.oursoftware.com&trustCertificateKeyStorePassword\=XXXXX&verifyServerCertificate=true

but I'm still getting the same errors in the log files. In fact, I purposefully made a typo in the filename value for trustCertificateKeyStoreUrl and continued getting the same errors... which I would maybe expect a different error saying the file wasn't found... anyways, still haven't found the magic formula.

Anyone else using MySQL and have a sample server.x.URL they could share?

[darren]

As a test, I've upgraded to mysql-connector-j-8.0.33.jar, but that didn't help either. Still getting the exact same error messages... full logs found in attached file. Thanks.

[tnguyen]

Hi Darren,

There's some documentation about this for the driver, depending on which version you are using. Have you already gone over this portion?

https://dev.mysql.com/doc/connector-j/e ... g-ssl.html

For 8.0.12 and earlier: As long as the server is correctly configured to use SSL, there is no need to configure anything on the Connector/J client to use encrypted connections (the exception is when Connector/J is connecting to very old server versions like 5.6.25 and earlier or 5.7.5 and earlier, in which case the client must set the connection property useSSL=true in order to use encrypted connections). The client can demand SSL to be used by setting the connection property requireSSL=true; the connection then fails if the server is not configured to use SSL. Without requireSSL=true, the connection just falls back to non-encrypted mode if the server is not configured to use SSL.

For 8.0.13 and later: As long as the server is correctly configured to use SSL, there is no need to configure anything on the Connector/J client to use encrypted connections. The client can demand SSL to be used by setting the connection property sslMode=REQUIRED, VERIFY_CA, or VERIFY_IDENTITY; the connection then fails if the server is not configured to use SSL. With sslMode=PREFERRED, the connection just falls back to non-encrypted mode if the server is not configured to use SSL. For X-Protocol connections, the connection property xdevapi.ssl-mode specifies the SSL Mode setting, just like sslMode does for MySQL-protocol connections (except that PREFERRED is not supported by X Protocol); if not explicitly set, xdevapi.ssl-mode takes up the value of sslMode ( if xdevapi.ssl-mode is not set and sslMode is set to PREFERRED, xdevapi.ssl-mode is set to REQUIRED).

[darren]

Thanks for the reference, Tuan. Ultimately we're not looking to change the MySQL Connector/J driver at this point, as it looks like that might require some change to our business logic code. I had just tried the driver upgrade to see if that would help w/ the problem.