Login Screen not showing up

[Bernd.N]

After our customer changed some network components, we see the issue that sometimes, a smart client does not show the login screen on startup.
Only the initial logo is shown.
It "hangs" on that logo for some minutes then.

When that is the case, also all other users who try to start the smart client, have the same experience, i.e. they can not log in.
So it seems something on the Servoy Server or the network temporarily blocks all smart clients from starting up.

After a while, everything works as before.

What could that probably be?
We never had that issue during the last 8 month.
Is a temporary blockade of the 1099 port a likely root cause?
(We run the tunnelConnectionMode http&socket)

[patrick]

That is a connection problem on the RMI port (1099) ...

[Bernd.N]

Ok, we will try to track who else is using RMI port 1099.
Do we need to track this on both Server side and Client side, or is one side sufficient when we run http&socket?

I found two threads about how to track such port usage:
http://serverfault.com/questions/414039/how-to-figure-out-what-is-blocking-my-connection-on-windows-7
http://superuser.com/questions/348661/how-do-i-check-which-process-is-using-port-1099

[patrick]

I wouldn't think that the port is used by something else, but the communication is "disturbed". Try to run this http only. That should fix this.

[Bernd.N]

I switched to "http only" this Monday, and saw a huge decrease in speed on the next day. That's why I do not want to use that for now.

I now called netstat on the server and got a quite interesting large list.
The Foreign Addresses with state "Established" seem to be all clients that are connected currently, because their number did match the number of active clients that Servoy AppServer showed in its clients tab.
And I could see that the Server is communicating with them over its own 1099 port:

Active Connections

Proto Local Address Foreign Address State
TCP 10.103.xxx:445 xxxxa2033:51891 ESTABLISHED
TCP 10.103.xxx:1099 xxxx.68.28:56773 ESTABLISHED
TCP 10.103.xxx:1099 xxxxA2034:56758 ESTABLISHED
TCP 10.103.xxx:1099 xxxxa0026:64420 CLOSE_WAIT
TCP 10.103.xxx:1099 xxxxl0017:23478 CLOSE_WAIT
TCP 10.103.xxx:1099 xxxxs002:50213 CLOSE_WAIT

However there were also a lot of connections in CLOSE_WAIT state.
There were about 90 lines with state CLOSE_WAIT compared to 30 lines with state ESTABLISHED.
Is that a normal situation?
Because I would expect that a connection should be completely terminated and disappear from this list, after a client logged out properly.

This is a link where all those states are described in detail:
https://blogs.technet.microsoft.com/jan ... lose_wait/

There I could read that the CLOSE_WAIT is not really a "usual one":

CLOSE_WAIT
Indicates that the server has received the first FIN signal from the client and the connection is in the process of being closed
So this essentially means that his is a state where socket is waiting for the application to execute close()
A socket can be in CLOSE_WAIT state indefinitely until the application closes it.
Faulty scenarios would be like filedescriptor leak, server not being execute close() on socket leading to pile up of close_wait sockets

[Bernd.N]

This happend again, I would like to describe the behaviour as exactly as possible:

In our smart client solution, users of our customer did not get the login screen after they started the solution. Only the initial branding logo screen appeared.
After a while (about 15 minutes), the login screen finally appeared.

We have 200 licences installed, at that time about 100 were used.

In the server log, around the time where it happend we can see a lot of messages telling this:
java.rmi.ConnectException: Connection refused to host: 10.119.10.127; nested exception is:
java.net.ConnectException: no multiplexer found for server with id=5d79d0d5-a597-4832-9ee3-1f31662bb7a2

Other error messages we see are:
- java.io.EOFException
- java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is ...

While users could not login, the already logged in users could work without any problem.

This happend last time about 3 month ago (see above), and now it happend again on Friday afternoon and Monday morning.

What could be a possible reason that the login screen did not appear?
Maybe something outside the Servoy server/solution itself, like network problems?

I also created a new case:
https://support.servoy.com/browse/SVP-841

[patrick]

This again sounds like connection problem on the RMI (default 1099) port... A number of things can go wrong here from firewalls to proxys to "internet security" packages installed on client machines.

[Bernd.N]

My current assumption is that there is some kind of external service/job running on the server or the network, that is causing the 15-minute-blockade of the RMI port.
Because this is only a very rare temporary problem.

So I asked the network admins of our client if they could think of such a service that uses and blocks 1099 temporary.

Thanks for the hints, Patrick, I will forward your answer to the network admins.

[jcompagner]

what does the logging say on the client?

So open a client with a webstart console and see what it reports there.
Because that EOF seems to be the cause to me, the client connects but then something in the middle just kills the connection again.

[Bernd.N]

Thanks, I will tell this to the client's admin.

[Bernd.N]

We could not see anything interesting on the client's webstart console.

There is an interesting time pattern for this issue:
When it appears for about 10 minutes, it usually appears also two days after the first appearance. Then we have some weeks or even month silence.

I meanwhile suspect that it could be an update service, like the Google Update Service. Updates could be automatically downloaded somehow every other month, and then as users detect bugs, another update will come about 2 days after the first update.
That would explain the time pattern and the short problem period of 10 minutes.

So I asked the IT guys now to check all running services on the server and to close down those which are not absolutely necessary.
McAfee virus scan runs itself about six different services on that server. Who knows what those services are doing with this port 1099.

I do not assume that a Servoy application server itself denies login to users every six weeks for about 10 minutes e.g. when it does some garbage collection or something like that.
But if someone knows about such a thing, I would be happy to get to know it.