Solved How to launch sshd earlier on in the boot sequence?

priyadarshan

priyadarshan

At the moment sshd is launched quite late in the booting process. We would need to have sshd running as early as possible on our servers, right after the firewall starts, or, on some local-network only servers, even as soon as possible.

Where can one read about a safe, upgrade-compatible way to do that?
 
Thank you. It seems I need to change /etc/rc.d/sshd, so that,

Bash: 
# REQUIRE: LOGIN FILESYSTEMS

becomes

Bash: 
# REQUIRE: NETWORKING

But if I do that, likely the change will be reverted at next system update.
 
eternal_noob said:
Yes, but why? Why is it a problem that sshd starts 20 seconds later?
Click to expand...
Sometimes other processes launched before sshd cause quite longer delay. Mounting temporarily non-existent mount points comes to mind, but also other daemons which will stall the system waiting for resources that may not be immediately available. At times, rebooting into single-user may be a last-resort option.
 
Will this actually work? The big dependency here is: the users that log in via sshd need home directories, so you need to wait for the correct set of mounts. And if the users that log in want to run any executables that need shared libraries, they need to wait for ldconfig.

Suggestion: Instead of changing the order by pulling sshd earlier, consider looking at the services that are causing a delay before logins are enabled, and see whether those are configured correctly (maybe they can be speeded up), and see whether maybe they can be pushed later if they are really not needed. That might require no surgery to the ordering, or smaller surgery.
 
ralphbsz said:
Will this actually work?
Click to expand...
Yes, it does work with no glitches, at least for now. I have now two servers running with /etc/rc.d/sshd, edited as shown above. I am not sure how well that change will be handled in upcoming upgrades, but I am aware of the situation and I will keep it under close watch.

ralphbsz said:
Suggestion: Instead of changing the order by pulling sshd earlier, consider looking at the services that are causing a delay before logins are enabled, and see whether those are configured correctly (maybe they can be speeded up), and see whether maybe they can be pushed later if they are really not needed. That might require no surgery to the ordering, or smaller surgery.
Click to expand...

That is not possible, as the potential delays do not happen due to some internal configuration issues, but depending on outside factors, as mentioned above.
 
In the past I experienced a severe trouble with sshd comming up too late. A BLog post tells the story. Coincidently, 3 years ago it got almost the same title than the present thread.

Since then, I keep around the following sed command, and I issues this after installing/updating FreeBSD:
Code: 
sed -i "" -e "s/# REQUIRE: LOGIN FILESYSTEMS/# REQUIRE: NETWORKING\\
# BEFORE: mountcritremote/" /etc/rc.d/sshd

This makes sure that sshd is running before any trials to mount perhaps not present NFS shares.
 
ralphbsz said:
I guess if none of the users on the machine have their home directory on NFS, this is safe.
Click to expand...
I guess, even if users on the machine have their home directory on NFS, this is safe. Because by /etc/rc.d/sshd only the sshd mother daemon is launched, and this does not care about the home directories of the users, as long as nobody tries to login. The actual sessions are operated be sshd child daemons which are spawned by the mother on occasions of login.

Login attempts in the course of system start-up before the NFS with the respective home dir is mounted, won't work of course, but this is in no way different from the default behaviour. With the change of the boot sequence this is also not a fatal failure, since the remedy is, to try again in a few seconds.

In case the sshd mother was successfully started but NFS does not come up, somebody else who got his home not on said NFS share may still login and fix the NFS setup.
 
I understand the logic of your argument. And it makes me wonder: Why isn't sshd in general started earlier? Maybe the correct answer is that someone (the OP?) might want to open a PR to change the startup order permanently. I haven't spent enough time analyzing this to be certain.
 
obsigna said:
Since then, I keep around the following sed command, and I issues this after installing/updating FreeBSD:
Click to expand...
Is it really needed after updating? That would mean freebsd-update(8) indeed doesn't perform a 3-way merge of /etc?

Not saying this isn't the case, I'm just surprised, thinking that using official binary builds should give at least the same level of comfort as updating from source. ?

ralphbsz said:
Why isn't sshd in general started earlier?
Click to expand...
I bet the answer is that nobody thought about network mounts only being a "weak" dependency for ssh. Only requiring networking for ssh seems to make a lot of sense, it should probably be changed...
 
Changing /etc/rc.d/sshd is probably a bit invasive, but ssh can easily be put before all ports and databases and webserver start up for an easily earlier process just by dropping an extra file in /usr/local which won't be overwritten ever.

First of all you can examine your current rc.d order with this command:
% rcorder /etc/rc.d/* /usr/local/etc/rc.d/*

In my case is something like:
Code: 
/usr/local/etc/rc.d/avahi-dnsconfd
/usr/local/etc/rc.d/samba_server
/etc/rc.d/LOGIN
/usr/local/etc/rc.d/garb
/usr/local/etc/rc.d/zrepl
/usr/local/etc/rc.d/vboxwebsrv
/usr/local/etc/rc.d/vboxwatchdog
/usr/local/etc/rc.d/vboxheadless
/usr/local/etc/rc.d/svscan
/usr/local/etc/rc.d/socat
/usr/local/etc/rc.d/dovecot
/usr/local/etc/rc.d/rsyncd
/usr/local/etc/rc.d/plexmediaserver
/usr/local/etc/rc.d/php-fpm
/usr/local/etc/rc.d/photoprism
/usr/local/etc/rc.d/node_exporter
/usr/local/etc/rc.d/nginx-prometheus-exporter
/usr/local/etc/rc.d/nginx
/usr/local/etc/rc.d/mysqld_exporter
/usr/local/etc/rc.d/mysql-server
/usr/local/etc/rc.d/gstat_exporter
/usr/local/etc/rc.d/grafana
/usr/local/etc/rc.d/bsdstats
/usr/local/etc/rc.d/blackbox_exporter
/etc/rc.d/syscons
/etc/rc.d/sshd

So I created this:
% sudo sh -c 'printf "# PROVIDE: ssh_earlier\n# REQUIRE: sshd\n# BEFORE: mysql grafana vboxwebsrv plexmediaserver\n" > /usr/local/etc/rc.d/ssh_earlier'

And now I have:
Code: 
% rcorder /etc/rc.d/* /usr/local/etc/rc.d/* | fgrep -B2 -A2 sshd
/usr/local/etc/rc.d/avahi-dnsconfd
/etc/rc.d/LOGIN
/etc/rc.d/sshd
/usr/local/etc/rc.d/gstat_exporter
/etc/rc.d/ftpd

This needs you to identify specific (slowest) rc.d by inspection, but OTOH is a pretty clean way to have an early sshd.
 
priyadarshan said:
Mounting temporarily non-existent mount points comes to mind, but also other daemons which will stall the system waiting for resources that may not be immediately available. At times, rebooting into single-user may be a last-resort option.
Click to expand...
If those are regular occurrences on this setup you might want to reconsider the hosts dependencies on those other systems and maybe handle them differently? E.g. NFS homes (and other non-critical filesystems) could be automounted on first acces (see man=5]auto_master[/man]) instead of at boot time, which not only removes the dependency on those targets but greatly improves overall boot time, especially if you have dozens or hundreds of nfs-homes! NIS might be another service in that regard that can completely block the boot process - here you might want to use another mechanism (like LDAP /w local caching) to sync user accounts across multiple systems and don't put system/daemon users in it.

I also got myself once in such a situation where a single host was dependent on services on 3 other physical systems (and several zones/jails on those...) at boot, up to the point that on severeal scenarios that host wouldn't come up at all - and all clients were dependant on this very system as it held the users NFS homes.
The only sane way out is really to review all dependencies and find solutions that either completely remove those external dependencies (e.g. local storage instead of network mounts for critical filesystems) or at least transform them from bootup- to runtime-dependencies.

By starting sshd earlier you won't solve the actual problem - you just (minimally) ease the manual labor you shouldn't be required to do in the first place.
 
You must log in or register to reply here.
Back
Top