Solved SSH keeps disconnecting from Supermicro Server after few mins, then completely refuses to reconnect.

First_Law_of_Unix

First_Law_of_Unix

Hello.

I installed FreeBSD 13.1 on a Supermicro X8DT3 and also on an AMD desktop machine.

I installed OpenSSH server on the X8DT3.

Now when I try to connect to it by LAN using:
Code: 
# ssh -vvv -p 2222 user@192.168.0.104

Everything works. But then after some time it gets disconnected and then I can not reconnect to it and need to reboot to be able to reconnect to it.

This is what I get when it automatically kicks out the client:
Code: 
client_loop: send disconnect: Broken pipe

So I try to reconnect:
Code: 
$ ssh -p 2222 user@192.168.0.104

Then I get:
Code: 
ssh: connect to host 192.168.0.104 port 2222: No route to host

Here are the verbose details when I try to reconnect again:
Code: 
OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd  3 May 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolve_canonicalize: hostname 192.168.0.104 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/user/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/user/.ssh/known_hosts2'
debug3: ssh_connect_direct: entering
debug1: Connecting to 192.168.0.104 [192.168.0.104] port 2222.
debug3: Fssh_set_sock_tos: set socket 4 IP_TOS 0x48
debug1: connect to address 192.168.0.104 port 2222: Connection refused
ssh: connect to host 192.168.0.104 port 2222: Connection refused

This is what I get when I try to use the regular SSH client:
Code: 
$  ssh -vvv user@192.168.0.104

Code: 
OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd  3 May 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolve_canonicalize: hostname 192.168.0.104 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/user/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/user/.ssh/known_hosts2'
debug3: ssh_connect_direct: entering
debug1: Connecting to 192.168.0.104 [192.168.0.104] port 22.
debug3: Fssh_set_sock_tos: set socket 4 IP_TOS 0x48
debug1: Connection established.
debug1: identity file /home/user/.ssh/id_rsa type -1
debug1: identity file /home/user/.ssh/id_rsa-cert type -1
debug1: identity file /home/user/.ssh/id_dsa type -1
debug1: identity file /home/user/.ssh/id_dsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa type -1
debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/user/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/user/.ssh/id_ed25519 type -1
debug1: identity file /home/user/.ssh/id_ed25519-cert type -1
debug1: identity file /home/user/.ssh/id_ed25519_sk type -1
debug1: identity file /home/user/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/user/.ssh/id_xmss type -1
debug1: identity file /home/user/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.8 FreeBSD-20211221
debug1: Remote protocol version 2.0, remote software version dropbear_0.50
debug1: Fssh_compat_banner: no match: dropbear_0.50
debug2: fd 4 setting O_NONBLOCK
debug1: Authenticating to 192.168.0.104:22 as 'user'
debug3: Fssh_record_hostkey: found key type RSA in file /home/user/.ssh/known_hosts:1
debug3: Fssh_load_hostkeys_file: loaded 1 keys from 192.168.0.104
debug1: Fssh_load_hostkeys: fopen /home/user/.ssh/known_hosts2: No such file or directory
debug1: Fssh_load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: Fssh_load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: prefer hostkeyalgs: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa
debug2: ciphers ctos: 3des-cbc
debug2: ciphers stoc: 3des-cbc
debug2: MACs ctos: hmac-sha1
debug2: MACs stoc: hmac-sha1
debug2: compression ctos: zlib,none
debug2: compression stoc: zlib,none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: (no match)
Unable to negotiate with 192.168.0.104 port 22: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1

So I try to use the cipher syntax and also set it to the ssh_config file:
Code: 
$ ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 user@192.168.0.104

Code: 
$ user@192.168.0.104's password:
Permission denied, please try again.

I provide the correct FreeBSD login password for the username "user".

On the server machine, I set root login and Public key enabled and turned off Password Authenticate. This makes SSH to work but only after a reboot and only works for few mins then it kicks out the client and after that completely refuses to login.

I also set "ServerAliveInterval=60" to the client machine on /etc/ssh/ssh_config. To prevent ssh auto timeouts.

Also did the firewall /etc/pf.conf:

Code: 
pass in proto tcp from any to any port 2222
pass in proto tcp from any to any port 22

Then entered on terminal:
pfctl -f /etc/pf.conf

Thanks for any advice.
 
I spoke with ChatGPT.

Seems like the entire time on sshd_config file, the line which contains "port 2222" was commented the entire time. I then uncommented it out. (But why was ssh working after each reboot?)

Then did a service sshd restart
I was getting denied in the restart service for ssh. Seems like some process was using the 2222 port.

I then installed "lsof":
pkg install lsof

Then did the following:
lsof -i :2222

And terminated the process consuming the 2222 port:
kill <PID #>

Then did again:
service sshd restart

Now ssh is working in order again.
 
Wow, ChatGPT just taught me a whole lot about FreeBSD networking in concise manner and what each parameter precisely means in clean, comprehensible and simple english.
 
First_Law_of_Unix said:
Wow, ChatGPT just taught me a whole lot about FreeBSD networking in concise manner and what each parameter precisely means in clean, comprehensible and simple english.
Click to expand...
Have you had a look at the manpages? That's very likely what ChatGPT is citing anyways - but most likely from some random release...
Also keep in mind that ChatGPT will spew just about any crap at you while still sounding sophisticated just to give some answer. It has been proven again and again that the output of that data harvesting tool turned to chatbot (it's nothing else, there is no 'intelligence' behind it) is very often plain out wrong and can't be trusted. Best prove for that: There is absolutely no need to install lsof on FreeBSD - you could have used sockstat(1) for the task of finding the process using port 2222, or (as it was obvious that sshd is listening on that port) simply just use 'pkill(1) sshd', which doesn't require you to look up the PID.

FreeBSD has excellent documentation (handbook) and manpages which require absolute no additional effort like talking to some pseudo-intelligent chatbot and is most importantly of known origin/age. If you don't exactly know what you are looking for, you can also just use apropos(1)


RE your original problem:
- why are you using another port? "security by obscurity" = no security. Use key-based logins and prevent password-based ones. Also run something like blacklistd(8) or security/sshguard
- why is sshd crashing/freezing? anything in /var/log/messages or dmesg? That X8 platform is ANCIENT, so chances are very high that this thing (or any other component of the same age) is dying or just too old, so that drivers don't support it any more (e.g. the NIC...) which may cause problems even if that component seems to work.
 
sko said:
That X8 platform is ANCIENT, so chances are very high that this thing (or any other component of the same age) is dying or just too old, so that drivers don't support it any more (e.g. the NIC...) which may cause problems even if that component seems to work.
Click to expand...
My X8DT3-LN4F works just fine with 13.2-STABLE. Yes, it's old. But for a home-lab it'll do just fine.

First_Law_of_Unix said:
I installed OpenSSH server on the X8DT3.
Click to expand...
You know it's included with the base OS? No need to install security/openssh-portable unless you have some specific requirements that aren't enabled in sshd(8) of the base.
 
Yeah, also note the firewall state timing out. If you have an idle connection the firewall state will timeout, effectively killing the connection. Make sure to enable keep alive on the client, so it sends regular "noop" packets, that will keep the firewall state active.

You can add this to your ~/.ssh/config:
Code: 
Host *
        ServerAliveInterval 10
        ServerAliveCountMax 2
 
I just started up the server again, I'm getting the same issue. Really strange...

Client Machine:
ssh -vvv -p 2222 user@192.168.0.104
Code: 
OpenSSH_8.8p1, OpenSSL 1.1.1o-freebsd  3 May 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolve_canonicalize: hostname 192.168.0.104 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/user/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/user/.ssh/known_hosts2'
debug3: ssh_connect_direct: entering
debug1: Connecting to 192.168.0.104 [192.168.0.104] port 2222.
debug3: Fssh_set_sock_tos: set socket 4 IP_TOS 0x48
debug1: connect to address 192.168.0.104 port 2222: Connection refused
ssh: connect to host 192.168.0.104 port 2222: Connection refused

Server Machine:
sockstat -4 | grep :2222
Code: 
root    sshd    2110    4    tcp4    *:2222

pfctl -s all | grep 2222
Code: 
pass in proto tcp from any to any port = 2222 flags S/SA keep state

ls -l /etc/ssh/sshd_config
Code: 
-rw-r--r--  1 root  wheel  2348 Feb 27 13:27 /etc/ssh/sshd_config
 
I have no idea why, when I originally powered up the server today, the LAN cable was not plugged in.
I later plugged the LAN cable back in when I needed to use SSH.

I did immense troubleshooting and can not figure out why I couldn't get ssh to work pragmatically without doing a reboot.

So I finally did a reboot since nothing worked, SSH works now, I can connect to the server.
So it seems that I can never get SSH to work if FreeBSD boots without the LAN cable connected.
I need to boot FreeBSD with the LAN cable connected to the server.
 
sko said:
Have you had a look at the manpages? That's very likely what ChatGPT is citing anyways - but most likely from some random release...
Also keep in mind that ChatGPT will spew just about any crap at you while still sounding sophisticated just to give some answer. It has been proven again and again that the output of that data harvesting tool turned to chatbot (it's nothing else, there is no 'intelligence' behind it) is very often plain out wrong and can't be trusted.
Click to expand...

Honestly the term "intelligence" is highly irrelevant. Theres a lot of reasons to why this is.
For example, a PhD rocket scientist is more intelligent than the local farmer, but it can be also said that the local farmer is more intelligent than the PhD rocket scientist.
Why?
The rocket scientist knows how to make rockets but would be clueless on how to germinate seeds for a 100-acre farm.
I hope you get my point. "intelligence" is just being compared to another intelligent standard, everything is programmatic and was theorized by Charles boolean all the way back in his works of "The Laws of Thought (1854)".

ChatGPT has it's pros and cons. It does throw a lot of nonsense, but you need to teach it and it will learn. They programmed it to throw nonsense because saying "no" is worse than "lying".

I have learned so much low level fundamental programming thanks to ChatGPT.
It has made me extremely productive and able to learn quickly and fast of complex programming topics.

Such as creating back end program services in pure C code for web servers. Good luck for anyone in this universe to find someone who can tutor them one-to-one or give examples in great details on how to create a network socket program in pure C-code.

Many will laugh at you and think you're crazy using C-code and have no clue how to help. No content what soever on the Internets about it. You have to read large thick text books, the golden unix socket programming. Let alone would need to be an undergraduate to know many of the computer science terms used in these books.

ChatGPT is a great tool to help you guide the right path but need to be aware that it's answers are not always correct and need adjustment.

Yesterday I was able to precisely, carefully and surely learn about JavaScript AJAX's XMLHttpRequest API programming, I learned it in 1 hour, thanks to ChatGPT. Else. it would've taken me a day or two to learn it from scratch.

It's explanations and code examples for highly documented and popular topics like C/C++/HTML/CSS/JavaScript are just phenomenal.

It was trained to learn the book: "Unix Network Programming" and can help anyone to tutor trough the book.

sko said:
That X8 platform is ANCIENT, so chances are very high that this thing (or any other component of the same age) is dying or just too old, so that drivers don't support it any more (e.g. the NIC...) which may cause problems even if that component seems to work.
Click to expand...

Old hardware from quality vendors never die. Many server builders are buying old NIC tech from eBay and works flawlessly. I actually bought an old 4-port gigabit NIC card and also Raid cards to make JBOD file servers and they have rock solid driver open-source support. They are cheap and always works. But need to buy the correct cards.
 
Last edited:
First_Law_of_Unix said:
So it seems that I can never get SSH to work if FreeBSD boots without the LAN cable connected.
I need to boot FreeBSD with the LAN cable connected to the server.
Click to expand...
FreeBSD will start up and there will be no connection.

sshd will try and startup but there’s no network so I expect it will give up and fail to start - anything in logs? If you watch the boot process are there any messages when sshd starts? It might stick for a while before it times out.

When you plug the cable in FreeBSD will notice and networking will work but perhaps sshd won’t notice without a prod - did you try restarting sshd rather than the whole machine?

The above is a theory based on what you are saying not an expert opinion.
 
Alright, I finally figured out the issue. This is my first server machine and had no idea about some of it's useful server components, such as IPMI (Intelligent Platform Management Interface) which is also know as BMC (Baseboard Management Controller). Really cool stuff, basically a mini computer within a computer and it works independently monitoring the server.

I did read the manual earlier before starting up the Supermicro's X8DT3 server, it did mention that the BMC's controller chip which provides IPMI implementation is connected to both the dedicated IPMI LAN port and as well the onboard LAN port-1, meaning that admins are able to access the X8DT3 via both ports.

This raises many questions, many which are not explained and Supermicro has a dedicated manual just for it's BMC IPMI and couldn't find info on this and as well on how to reset the the BMC completely headless mode, since I have no info about the server's previous static IP settings which it was set to. I also found out that clearing the CMOS BIOS on the mobo did not erase the BMC at all, this made it even more difficult for me and had to really set up the server using a VGA monitor.

Anyhow, I always expect hardware engineers to make consumer (admins) lives easier, it seems the way how the BMC chip was programmed, is not implemented or designed in a proper "logical" manner.

Non the less, here is what caused the issue:

When I went to the BIOS settings, I changed the previous owner's IPMI static IP address to the correct IP address based on my LAN network.
I then connected the LAN cable to the onboard LAN port-1 and not to the dedicated IPMI LAN port.
I then installed FreeBSD and set the same static LAN IP address to FreeBSD.
After all this, SSH was giving me problems, I kept being kicked out and not able to log in. Never in my life had I ever had this much of an issue with SSH on FreeBSD.

After 1 day trying to figure out the issue, I finally realized what was causing the problem with SSH, I completely disabled BMC IPMI and after this, SSH was working perfectly, no more getting kicked out from SSH and not being prevented to log into the server via SSH. I seriously doubted that the BMC would cause the IP conflict.

The BMC IPMI needs to have it's own Static IP address and must not be the same Static IP address for the onboard LAN port-1. There was an obvious network IP conflict between the BMC controller and FreeBSD using the same static IP address, it seems like the BMC controller has more authority (which is good) and completely messes and crashes the LAN port 1, when both coexist at the same time using the same static IP address.
 
First_Law_of_Unix said:
I then connected the LAN cable to the onboard LAN port-1 and not to the dedicated IPMI LAN port.
I then installed FreeBSD and set the same static LAN IP address to FreeBSD.
Click to expand...
No, no. They need to be different IP addresses. That BMC is basically a completely separate builtin computer that can control the hardware of the server itself. As long as the power's connected this "outside" computer is turned on and running. Even if you have switched off the server. You can actually turn the server on or off through this BMC 'computer'. And honestly, it's a lot easier if you just stick a network cable in the specific IPMI network port. Sure you can essentially 'share' a physical interface with the server and IPMI but this kind of server management is called 'out-of-band' for a reason.

Install sysutils/ipmitool. You can use this to configure the IPMI network settings from FreeBSD's command line. Make sure you load the ipmi(4) kernel module. Useful command; ipmitool lan print 1 This will show you the current IP settings of IPMI.


Minimal IPMI configuration settings:
Code: 
ipmitool lan set 1 ipaddr 10.0.0.10
ipmitool lan set 1 netmask 255.255.255.0
ipmitool lan set 1 defgw ipaddr 10.0.0.1

Then point a browser at the IP of the IPMI settings. Default username/password is ADMIN/ADMIN (Username needs to be all caps too). On this old beast the actual remote console is done through a java web app. Newer IPMI firmware on more modern systems use HTML5. The Java app is going to refuse to connect due to old SSL/TLS versions. You will need to enable those in your JRE or else it's going to fail to connect.
 
You must log in or register to reply here.
Back
Top