Page 1 of 1

Trouble-shooting Centos installation of Server 7.0.0 SOLVED

PostPosted: Sat Apr 06, 2013 11:45 pm
by richh
Summary
I am investigating a move from a central HQ Windows server to a VM on Centos on the web. A test setup using Servoy Server 6.1.3 worked fine, but 7.0.0 has problems for us. This post describes progress and invites suggestions for further trouble-shooting.

Scenario:
Servoy Application Server hosted on a Centos virtual server in cloud. Servoy 6.1.3 runs OK, with 2waysocket selected. Clients connect easily. The server has a separate postgreSQL service. Nice and stable. This is a test regime to check out cloud services prior to migration from a Windows server in an HQ office.
The server has Java VM as follows:
java.vm.name=Java HotSpot(TM) 64-Bit Server VM
java.version=1.7.0_09
java.vm.info=mixed mode
java.vm.vendor=Oracle Corporation

On a Windows VM in Parallels we install Developer 7.0.1 to make a few improvements to our main solution. The solution works fine in Developer and Smart Client.

Next step is to check out the App Server on our local area network. We install 7.0.0 on a spare OS X box. We create a new repository on that machine, and upgrade it according to the Servoy wiki. With solution imported, the Application Server checks out fine. It is set with Network Settings 2waysocket, SocketFactory.compress:ticked, SocketFactory.useSSL:ticked, SocketFactory.tunnelUseSSLForHttp:NOT ticked. We checkout two solutions: one is the upgraded main solution; the other is a simple one form solution called 'names' which connects to a single table and enables simple queries to be fired at the database. 'names' has no variables, no javascript. It simply displays a first name and last name for all of the rows. Both solutions check out OK on our Servoy App Server 7.0.0 test box.

Next step is to uninstall Servoy 6.1.3 from the Centos VM; and to install 7.0.0 application server in its place. We configure servoy.properties as before, but following the Servoy configuration web page rather than starting with our old servoy properties file. We start with a new repository, and upgrade it. We then add database servers, our users and our two test solutions. Our network settings are set as above.

On downloading a new client from the Centos VM port 8080, we are offered the two solutions. On login to either of the solutions, we see a momentary 'You are disconnected etc' - the error message disappears too quickly to see the whole of it. Neither solution loads a form, though the client window remains with a grey background. The Application Server clients page shows the client as connected under solution <none>, and a message can be sent from the server page to the client, where it is presented immediately. The server log reads: ClientExportNotifyListner[3] ERROR com.servoy.j2db.util.Debug Signalling channel lost when reading pings or client export notifies, removing ports: [3001].

Changing the network settings to http&socket does not change the observed behaviour of logging in, but the comms error is different. It changes to: Date/Time multiplexer-reader-Socket[addr=[obscured]localport=1099] ERROR com.sebster.tunnel.impl.od multiplexer failed for client [uuid type identifier].

With this second set of network settings, a check of webclient shows that 'names' can be loaded, and our main solution gets challenged for login, but then shows 'Internal Error'. We see in the server log: java.lang.IllegalStateException: invalid command: 22
at com.sebster.tunnel.impl.lc.run(lc.java:60)
at java.lang.Thread.run(Thread.java:722)

Diagnosis
The observed communications error appears to be associated with the change from v6 to v7 of the application server, but I am not sure.

Any suggestions for further tests? I am working towards a setup which works in smart client.

Re: Trouble-shooting Centos installation of Servoy Server 7.

PostPosted: Tue Apr 09, 2013 12:19 pm
by jcompagner
this sounds to me really like a network problem.
Because the connection is constantly dropping out or something like that.

Re: Trouble-shooting Centos installation of Servoy Server 7.

PostPosted: Tue Apr 09, 2013 7:47 pm
by richh
Yes Johan: you and I have been thinking along the same lines. However my further testing today gives new information which raises questions about authentication and login on Server 7.0.0. It is not simply a matter of network settings.

Today I used new installs (to eliminate histories). On a development machine (Windows 7 VM in Parallels) we have Servoy 7.0.1 Developer. Our solution is associated with an authentication solution and a login form solution as per wiki. The authenticator uses Servoy security. Smart client in Developer behaves as expected: we log in, getting all the right responses for authentication, the solution loads and we can use it.

To eliminate network problems external to the development machine we investigate running the server and client on the same Windows 7 virtual machine. We close Developer down, and run up Servoy Server 7.0.0, loading main, login and authenticator solutions exported from Developer.

Downloading a new client from 8080, we log in to our main solution. Authentication works as expected. But the solution fails to load. In Windows Task Manager, we see javaw.exe *32 at 99% CPU, memory 69MB. In Servoy-admin/clients we see the client logged in with the correct username. We can send messages to the client, reported as a success by the server, but these are not displayed by the client. We cannot close the client using window controls; no menu is showing. We attempt to shut down the client from the servoy-admin/clients web page. We get green text saying ...signalled to shutdown successfully. Refreshing the clients web page shows the client is still present.

We then kill the stalled javaw.exe *32 using the Windows Task Manager. It disappears as expected. But refreshing the clients web page shows the client still listed. We try sending a message to this client, and get an amber message: 'Client [id string] not registered with server.'. Refreshing the clients page once again, we see no clients are connected.

I attach an extract from the servoy_log file.

In a separate test, I have added a simple solution which displays one form connected to a single table. Its solution property 'mustAuthenticate' is ticked. Selecting this solution in a new client, we are able to get through authentication and display the form and table data with no problem. But the main solution still cannot load.

In summary: working under a single operating system, the combination of authentication, login solution and main solution work well in Developer 7.0.0, but exhibit connectivity problems when run under Server 7.0.0. A simple solution which does not use the authentication and login solutions works OK in both Developer and Server.

Re: Trouble-shooting Centos installation of Servoy Server 7.

PostPosted: Wed Apr 10, 2013 10:26 am
by jcompagner
do you have logging of the client, not the server the logging you show here is that the server tries to connect directly to the client because the normal way (what we call signalling channel) did collapse
And a direct connection will normally never work because of blocked ports or internal ip's of clients.

So open the java console on the client and see what the log says there.

Re: Trouble-shooting Centos installation of Servoy Server 7.

PostPosted: Wed Apr 10, 2013 1:30 pm
by richh
Johan: good thinking.

Console log attached, showing several launches of the client, including one where the main solution authenticated but then failed to load and CPU on javaws *32 went to 97+%. Maybe it is on such an occasion that the signalling channel collapses?

I observed today that there is a difference in behaviour if a user has already authenticated using the simple 'names' solution. The login process does not use separate login and authentication solutions, but rather Servoy Server authentication only. In this case, the user is authenticated and is able to use 'names'. When the user closes 'names' and opens the main solution, s/he is presented with the login form as expected. Authentication occurs and the main solution loads. The user is then able to use the main solution. The attached log covers this use case.

On the other hand, if the user launches a new client and logs into the main solution first, then the login form appears and but the solution then fails to load and javaws *32 races. There is nothing added to the java console log at this point. The log also covers this use case.

It seems to me that the communications problem is caused by a fault in the 'advanced authentication' process which shows up only if the user has not logged into another solution earlier with the same client.

Re: Trouble-shooting Centos installation of Servoy Server 7.

PostPosted: Wed Apr 10, 2013 4:57 pm
by jcompagner
that log has nothing
is that of a client where it did go wrong? Only that log works those logs are from a single run

Re: Trouble-shooting Centos installation of Servoy Server 7.

PostPosted: Wed Apr 10, 2013 10:36 pm
by richh
Yes Johan: the java log contains entries from a client where it went OK and from one where it went wrong.

Re: Trouble-shooting Centos installation of Servoy Server 7.

PostPosted: Thu Apr 11, 2013 8:49 am
by jcompagner
but then the client didn't report any error or connection problem..
because there is as i said nothing relevant in the log, there is not one log statement that servoy generates.

Re: Trouble-shooting Centos installation of Servoy Server 7.

PostPosted: Thu Apr 11, 2013 10:57 am
by richh
Given that the client hangs on authentication, this is a show-stopping event. The fact that neither Servoy nor the Java console log catch the fault shows that we have a situation outside the scope of the two logging mechanisms.
When I return to the office at the weekend, I shall attempt some more precise testing of the authentication and report back here.

Re: Trouble-shooting Centos installation of Servoy Server 7.

PostPosted: Thu Apr 11, 2013 11:21 am
by jcompagner
normally if there are exceptions or connection problems that will definitively show up the the webstart log of that application
Is there a server where i could connect to to see that?

Re: Trouble-shooting Centos installation of Servoy Server 7.

PostPosted: Thu Apr 11, 2013 10:49 pm
by richh
Not currently: but on my return after the weekend.

Re: Trouble-shooting Centos installation Server 7.0.0 SOLVED

PostPosted: Thu Apr 18, 2013 5:14 pm
by richh
Johan looked at the Webstart log and found evidence which pointed to an error in wrapper.conf that had been identified and corrected two weeks ago in post viewtopic.php?f=16&t=19425&start=30#p104940.

A straightforward correction of wrapper.conf as detailed in that post restored the server to full operation.

Note that this problem lay not in network settings, nor in authentication, nor in Java 6 versus Java 7, nor in Servoy 6 versus Servoy 7, but simply in an error in wrapper.conf. It was a show-stopper but simple to correct once diagnosed.

Well done Johan! Many thanks for your outstanding support.