Solved ssh immediately disconnecting across all jails

Farhan Khan

Farhan Khan

Hi all,

I ran freebsd-update across all my jails. Now, whenever I try to ssh(8) into one, the connection immediately dies. There are no messages in /var/log/messages.

Example below:

Code: 
$ telnet 192.168.100.22 22 
Trying 192.168.100.22...
Connected to 192.168.100.22.
Escape character is '^]'.
Connection closed by foreign host.

The jail is definitely listening on port 22, but immediately drops the connection. I have the same behavior on inetd(8). I noticed that packages I added later, such as nginx, work just fine and do not experience this problem.

Any ideas? How do I fix this?
 
Raise logging verbosity to check what's going on using the -vvv commandline option.

(this is indeed assuming you're using ssh to connect with SSH instead of telnet (obviously). Also: it might be more useful to check if the Jail is actually listening on that port. Try using sockstat -4l and look for port 22).
 
OP's test appears valid, SSH should present a banner and wait for input.

Bash: 
[root@CentralServices ~]# telnet 10.1.10.175 22
Trying 10.1.10.175...
Connected to 10.1.10.175.
Escape character is '^]'.
SSH-2.0-OpenSSH_7.2 FreeBSD-20161230
Bash: 
[root@CentralServices ~]# nc 10.1.10.175 22
SSH-2.0-OpenSSH_7.2 FreeBSD-20161230
 
ShelLuser said:
Raise logging verbosity to check what's going on using the -vvv commandline option.

(this is indeed assuming you're using ssh to connect with SSH instead of telnet (obviously). Also: it might be more useful to check if the Jail is actually listening on that port. Try using sockstat -4l and look for port 22).
Click to expand...
It immediately disconnects, so the ssh client will not provide any additional information.
 
leebrown66 said:
OP's test appears valid, SSH should present a banner and wait for input.

Bash: 
[root@CentralServices ~]# telnet 10.1.10.175 22
Trying 10.1.10.175...
Connected to 10.1.10.175.
Escape character is '^]'.
SSH-2.0-OpenSSH_7.2 FreeBSD-20161230
Bash: 
[root@CentralServices ~]# nc 10.1.10.175 22
SSH-2.0-OpenSSH_7.2 FreeBSD-20161230
Click to expand...

I thought it might be, but better to remove any ambiguity.
 
Farhan Khan said:
It immediately disconnects, so the ssh client will not provide any additional information.
Click to expand...
Time to check the actual Jail then.

(edit) But you should still share the error. What error message does it give you? Because that can determine the actual cause. (edit2) If it doesn't give you any extra info at all then I'd also suspect your SSH client to be flakey.
 
ShelLuser said:
Time to check the actual Jail then.

(edit) But you should still share the error. What error message does it give you? Because that can determine the actual cause. (edit2) If it doesn't give you any extra info at all then I'd also suspect your SSH client to be flakey.
Click to expand...
I ran a tail -f /var/log/messages and nothing appeared there. No error messages :/
It might be the ssh client, but this happens even with telnet, suggesting it isn't a client issue.
 
If you ran ssh -vvv <host> it would have shown you several debug messages on the console (stdout), and those will give you a clear indication why the connection failed. As such my question about the error message.

/var/log/messages is a good place to look, but not on the client but within your Jail. Still, I'd start by checking the output of sockstat -4l (in the Jail) to ensure that sshd is actually running. Perhaps followed by ssh localhost to make sure. If all of that functions normally then the cause of your problem lies somewhere between the host and the Jail, most likely a firewall of some sort.

Just rule out possible causes and you'll soon find the actual problem.
 
Is the jail sshd the base one? You mentioned you updated all your jails.
If this is the base daemon it should have been updated.
If this is a port/pkg daemon it may have needed updating manually.

If you want debug from the jail side, I usually do this so I don't have to mess with the existing daemon.
Jail# sshd -D -d -p 2222
Then from outside put -p 2222 before the host, i.e.
ssh -vvv -p 2222 <host>

(The jail daemon can be run as an unprivileged user as the port is above 1024).
 
Hi all. Continuing to debug this issue, someone suggesting running ssh -deD. Here is the output:

Code: 
# /usr/sbin/sshd -deD
debug1: sshd version OpenSSH_7.2, OpenSSL 1.0.2k-freebsd  26 Jan 2017
debug1: private host key #0: ssh-rsa SHA256:zgwe8V/2354pHNzd6oeYf8fuN+3GF5NLk1qGOANOfJk
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:Ce4L9LPUBjt+OBf3W0FUNflazBvSPULk9QYdtnvRTLQ
debug1: private host key #2: ssh-ed25519 SHA256:D+ZrZd2LhKsNFHBxE1jNMcn6Jajye97GDiWz819PhEQ
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-deD'
debug1: madvise(): Operation not permitted
debug1: Bind to port 22 on ::.
debug1: Server TCP RWIN socket size: 65536
Server listening on :: port 22.
debug1: Bind to port 22 on 0.0.0.0.
debug1: Server TCP RWIN socket size: 65536
Server listening on 0.0.0.0 port 22.

When a connection occurs, I get this:

Code: 
debug1: fd 5 clearing O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: sshd version OpenSSH_7.2, OpenSSL 1.0.2k-freebsd  26 Jan 2017
debug1: private host key #0: ssh-rsa SHA256:zgwe8V/2354pHNzd6oeYf8fuN+3GF5NLk1qGOANOfJk
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:Ce4L9LPUBjt+OBf3W0FUNflazBvSPULk9QYdtnvRTLQ
debug1: private host key #2: ssh-ed25519 SHA256:D+ZrZd2LhKsNFHBxE1jNMcn6Jajye97GDiWz819PhEQ
debug1: inetd sockets after dupping: 3, 3
debug1: res_init()
/usr/sbin/sshd: Undefined symbol "Fssh_get_remote_port"

Per someone on IRC, I ran "freebsd-update IDS" and found a ton of SHA256 mismatches. However, "freebsd-update fetch install" said "No updates needed to update system to 11.1-RELEASE-p8", even with the -F flag.

Any ideas?
 
Farhan Khan said:
/usr/sbin/sshd: Undefined symbol "Fssh_get_remote_port"
Click to expand...
That smacks of an incomplete update, or a library mismatch.

What's the host OS version?
What was the jail OS version?
What is the new jail OS version?
How did you construct the jails?
How exactly did you update the jails?
 
I ran into this bug again and just found this unanswered question.
On the base and in the jail:
Code: 
freebsd-version -u
11.1-RELEASE-p10
Where is this library supposed to be located?
 
I resolved this problem by running cd /usr/src && sudo make installworld DESTDIR=/jails/base and it resolved the issue.
Overall I do not understand where the issue lays or how it began, but I highly suspect its a missing library somewhere.
Strange.
 
Last edited:
You must log in or register to reply here.
Back
Top