Solved Can't ssh after Upgrading to 11.2

Lamia

Lamia

I have been keeping an eye on issues pertaining to 11.2 on this platform for the past four days. I, like many others, just upgraded my box [from FBSD 11.1-RELEASE-p9] to 11.2. And I ran a few other commands like portmaster -a, pkg upgrade and so on after the upgrade. After restarting the PC, I could not log in. I noticed 11.2 making changes to sshd_config and /etc/ttys [during the upgrade] and I have double checked to make sure the changes cause no problem.

Openssh-portable has been explored and it is now enabled with sshd in my rc.conf. I cannot login from a remote client neither can I ssh 'root@locallhost' on the machine. And for the openssh-portable, it [or any ssh service] does not run in the background/deamon [visible in ps aux] when it is enabled in rc.conf. Hence, I needed to enable it and sshd too though I noticed its service [crypto/openssh] could be restarted.

At the moment, I see 'portmaster -a' hanging at 2000/4388 when building llvm60. But ignoring that (with -x), all installed system ports/pkgs are up-to-date.

SSH terminates at "ssh_exchange_identification: Connection closed by remote host". I have attempted all possible suggestions on the Internet to fix it with no luck.
Code: 
debug1: key_load_public: No such file or directory
debug1: identity file /home/xbox/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/xbox/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2 FreeBSD-06102017

I read that there was a bug in that version of OpenSSH but a FBSD box that is running a current release or so already has it fixed in its base. That line 'Local version....' was copied from a Linux box. I couldn't easily fetch the exact line from sshing into localhost in the FBSD itself but I am sure it was that version with no 'p2 or so' indicating a patch.
 
I have also checked sshguard-pf. All other services seem to be fine but not sshd. And from what it worth, this is a problem with sshd & it dependencies. This problem is re-described here in the FreeBSD Diary. While I cannot easily copy the sshd library from another machine, could there be another way around it?
 
Seems 11.2 is not very successful release, because, it seems, there are
some issues with it, I also had problems with it, I couldn't boot after upgrading to 11.2,
but after switching to old kernel version (11.1) via boot menu it booted fine,
without any issues, personally I refrain to update to 11.2, at least for now.

EDIT: Problem solved, see https://forums.freebsd.org/threads/...ease-fails-to-boot-reboots.66431/#post-394308
 
Last edited by a moderator:
tingo said:
So, have you run ldd on sshd yet?
Click to expand...
Yes, I did awhile back. Thanks tingo.
Please see below.
ldd-sshd.png
 
ILUXA said:
Seems 11.2 is not very successful release, because, it seems, there are
some issues with it, I also had problems with it, I couldn't boot after upgrading
to 11.2, because my zfs pool didn't mount, it showed zfs related error
and rebooted, but after switching to old kernel version (11.1) via boot menu
it booted fine, without any issues, personally I refrain to update to 11.2,
at least for now.
Click to expand...
You reminded on of Celine Dion - 'if I could turn back the hands of time'.
I noticed that we are advised against a rollback. Sadly, switching to the old kernel won't help me in this case. It didn't!
Damn it, I have been on this matter for over three days (::).
 
So I take it you build SSH from ports instead of installing it as a package? If it continues to fail then this might be caused by one of its dependencies. What I usually do when a port fails on me is to rebuild the port and its direct dependencies to see if that can solve the problem. # portmaster `prdeps`, where prdeps is a small script I made which checks a ports run dependencies. I also have pbdeps which is used to check building dependencies...

Code: 
#!/bin/sh
make run-depends-list | sed -e 's/\/usr\/ports\///g'
I often prefer this method instead of # portmaster -f because that recurses through all dependencies of all ports and rebuilds the whole lot. This is a little less intrusive and usually gets the job done.

Either way; considering how the problem seems to be focused on the encryption part you might want to verify security/openssl as well.
 
ShelLuser said:
I also have pbdeps which is used to check building dependencies...
Click to expand...
Many thanks ShellLuser, please share your pbdeps script too. In the meantime, I would want to explore the above prdeps.

If you meant that I installed openssh via ports, yes. I know that sshd is part of the base; so I would not have installed it via ports/pkgs management. I also installed openssh-portable via ports, just like I did for the latest version of openssl. I may try installing them via 'pkg install' now.

My interaction with the machine is now very limited since I can't ssh. I am currently accessing it via the QEMU [virtual/serial] terminal. I can't even cut & paste into it. I can only type few characters. And the fingerprint notifications drive me nut. I have tried suppressing them following suggestions like this but no luck. That is an issue for another day. Until then, I need get access back to the machine via SSH ASAP.
 
Right, then I misunderstood a few things, I thought you were already using the ports version.

First thing to do is to rule out possible problems in your configuration. Maybe try # sshd -T | less to see if that shows anything unusual (probably not).

Then stop the service, make sure no other instances are running, and then on the console which you have use # sshd -d. Check what the outpost is telling you, then try to connect and pay close attention to the messages on the server side.

Worst case scenario... move /etc/ssh/sshd_config out of the way and replace it with a vanilla version. If you have the sourcecode installed you should be able to find this in /usr/src/crypto/openssh/sshd_config. Enable only the specific options which you need and then see if you can fire up sshd.
 
ShelLuser said:
Then stop the service, make sure no other instances are running, and then on the console which you have use [BGCOLOR=#dee3e7] # sshd -d[/BGCOLOR]. Check what the outpost is telling you, then try to connect and pay close attention to the messages on the server side.
Click to expand...
I am getting closer to fixing it. Thanks a million. The problem is an undefined symbol - 'Fssh_get_remote_port'. Someone's box had the same problem here.

My challenge now is how to buildworld in a machine that I have a restricted access to. This is not about the terminal or so. I have tried to build FreeBSD from src on it in the past with no luck. It is more or less a VM. I need a compiled sshd library to get over this matter once and for all.
 
Building world going fine... At least, I have not made changes necessary for to a custom kernel. Pretty minimalistic for now.
 
(re)building the world should help, but why not focus on the port for now? If that builds you might be able to do the step with the sshd -d option and see what that tells you. Thing is; (re)building a port is much faster than rebuilding the world.

Even so; I wouldn't interrupt the process if it's already running, that'd be a shame. But if that process somehow fails then I would definitely recommend focusing on the port for now, trying to get it running manually with debug output.
 
ShelLuser said:
(re)building the world should help, but why not focus on the port for now? If that builds you might be able to do the step with the [BGCOLOR=#dee3e7] sshd -d[/BGCOLOR] option and see what that tells you. Thing is; (re)building a port is much faster than rebuilding the world.
Click to expand...
SSH, which shows the undefined symbol, has never been a port but part of the base. The openssh-portable and openssl mentioned above produce no error. What port are you referring to?
 
"make -j4 buildworld" error:
makebuildworld.png


"make -j4 kernel" error:
makekernel.png



I got the src files from - ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/11.2-RELEASE/src.txz - and ran the above command.

To make matter worse, I got the "sshd" executable from another FBSD (11.1-RELEASE-p9) and replaced the one on the VM with it but the "Fssh_get_remote_port" undefined symbol error resurfaced.
@ShellLuser: you mind shedding more light on the prdeps and pbdeps? An How-to will be appreciated. I have tried running the script with portmaster but it threw an error - obviously, it's not a Makefile.

Are there other recommendations from the community - e.g. a fix for the above build src, the right port/pkg to build for ssh, etc?
 
Code: 
FreeBSD sentinel.sentry.org 11.1-RELEASE-p10 FreeBSD 11.1-RELEASE-p10 #2 r333388: Wed May  9 12:43:41 AEST 2018     xxxx@newyork:/usr/obj/usr/src/sys/VULT_11.1  amd64

root@sentinel [/usr/lib] $ grep -i Fssh_get_remote_port *
Binary file libprivatessh.a matches
Binary file libprivatessh.so matches
Binary file libprivatessh.so.5 matches
Binary file libprivatessh_p.a matches

1. Do you have this library?

2. Can you try rebuilding this library?
 
ShelLuser said:
So why not use those for the time being? At least to gain better access to your system.
Click to expand...
I did as mentioned above; but they won't help. I still need to enable sshd in rc.conf to connect; so, I get returned to the same error.

I have now successfully built 11.1-RELEASE kernel & world on a local machine with 11.1-RELEASE-p9 running on it. I wonder why 11.2 kernel & world won't successfully build on a machine recently upgraded to 11.2-RELEASE.

As things stand, I am now recompiling all the ports with "portmaster -af". I noticed a few ports proofing difficult - one of them was curl-7.6.x. I had to choose the option 'no gssapi support' after it failed with the gssapi_base and gssapi_mit on separate occasions. That required me removing OpenSSL so that curl could install the best-fit version of it.

trev: Thank you for your input. I am now watching that library.

/me: I am happy helping the community test the 11.2-RELEASE even though it has taken the better part of a whole week from me.
 
Lamia said:
I did as mentioned above; but they won't help. I still need to enable sshd in rc.conf to connect; so, I get returned to the same error.
Click to expand...
So what happens if you try to start the binary manually? As I mentioned earlier: # sshd -d? (of course now referring to the binary which got installed with the port). I wouldn't bother with all the ports right now but only focus on sshd so that you can get easier access asap.

Oh, by the way, I overlooked your earlier question... There's nothing special about my pbdeps script, it does nearly the exact same thing only using a different build target: make build-depends-list. See also ports(7).

These can often help to rule out weird behavior with ports yet without having to rebuild all their dependencies, only a specific few. Of course its not a full proof solution.
 
Note that /usr/bin/ and /usr/sbin are on the path before /usr/local/bin and /usr/local/sbin. So a "plain" sshd or ssh will use the one from /usr/sbin/sshd and /usr/bin/ssh.
 
My sincere appreciation to everyone that advised me on this thread. How else can I describe the problem and its likely cause than to say the FreeBSD 11.1-RELEASE-p11 binaries conflicted with FreeBSD 11.2-RELEASE's.

While I successfully ran freebsd-update upgrade -r 11.2-RELEASE, the command freebsd-version -k gave 11.1-RELEASE-p11. I noticed it and, on several times, tried running freebsd-update upgrade... and freebsd-update fetch install again and again. It was actually after the upgrade & restarting of the machine that I got locked out.

In addition, I tried building FreeBSD from source and updating ports/pkgs on several occasions but no luck. It was during this time I noticed binaries were conflicting one another particularly OpenSSL related ports/pkgs. My last resort was to re-install the OS.

Thanks to the community.
 
Lamia said:
While I successfully ran [BGCOLOR=#dee3e7] freebsd-update upgrade -r 11.2-RELEASE[/BGCOLOR], the command [BGCOLOR=#dee3e7] freebsd-version -k[/BGCOLOR] gave 11.1-RELEASE-p11.
Click to expand...
The freebsd-update upgrade command only downloads the files required for the upgrade. You will need to run freebsd-update install to actually install the upgrade.
 
SirDice said:
The [BGCOLOR=#dee3e7] freebsd-update upgrade[/BGCOLOR] command only downloads the files required for the upgrade. You will need to run [BGCOLOR=#dee3e7] freebsd-update install[/BGCOLOR] to actually install the upgrade.
Click to expand...
What would have happened on a machine that had 11.1-RELEASE-p9 if one ran freebsd-update upgrade -r 11.2 followed by freebsd-update fetch and lastly freebsd-update install? Of course, one would have restarted the machine in between the commands.
 
Lamia said:
What would have happened on a machine that had 11.1-RELEASE-p9 if one ran freebsd-update upgrade -r 11.2 followed by freebsd-update fetch and lastly freebsd-update install?
Click to expand...
It would be updated to 11.1-RELEASE-p11. The freebsd-update fetch pretty much undid everything and simply fetched updates for your currently installed version.
 
SirDice said:
It would be updated to 11.1-RELEASE-p11. The [BGCOLOR=#dee3e7] freebsd-update fetch[/BGCOLOR] pretty much undid everything and simply fetched updates for your currently installed version.
Click to expand...
That explains why freebsd-version -k shows 11.1-RELEASE-p11. But the top comment in /etc/sshd_config shows 11.2. I was presented with a vi editor interface to approve changes in it. And the same goes for /etc/ttys.

All the time I was checking them, they had 11.2. Undoubtedly, the failure to build a bunch of packages/ports, particularly the reports on investigating the failures, the FreeBSD source and the above mentioned changes in version point to conflicts in the binaries.
 
Well, running freebsd-update fetch pretty much threw a spanner in the works, you really shouldn't have done that.

In order to get things correct again I would start with rm -rf /var/db/freebsd-update/*. So you have a clean slate to start from. Then freebsd-update -r 11.2-RELEASE --currently-running 11.1-RELEASE-p9 upgrade. Then freebsd-update install and follow directions. Do NOT run freebsd-update fetch until after you have finished the ENTIRE upgrade process.
 
You must log in or register to reply here.
Back
Top