Nothing in /boot; nothing in /boot/kernel; system boots regardless

byrnejb · Oct 17, 2022

I upgraded a spare host from FreeBSD-12.1 to FreeBSD-13.1p2. During this update I encountered a problem caused by /boot being a soft link to /bootpool/boot. By removing /boot and then adding it back via mkdir I was able to complete the upgrade; or so I thought. I rebooted after the install, ran freebsd-update install again and rebooted.

What I have now is a system that boots and says it is 13.1-RELEASE-p2 but which has 0 contents in /boot. I cannot find the kernel nor any of the kernel modules using find. I am not really sure what to do at this point. I am not keen on reinstalling FreeBSD and wiping the entire zpool as this was/is a vm host and there are several vm's that, while not running, I do need to retrieve records from.

My present concern is to discover how this happened. Once that is accomplished then I need to know how to correct this issue.

Alain De Vos · Oct 17, 2022

Whas /bootpool a mounted filesystem.
E.g. part of /etc/fstab ?
Is the root filesystem / mounted ?
What is the output of:

Code:

mount -v /

richardtoohey2 · Oct 17, 2022

Anything in /var/log/messages e.g.

syslogd: kernel boot file is /boot/kernel/kernel

(This is from a UFS system so not sure if any difference with ZFS)

Alain De Vos · Oct 17, 2022

The output of:

Code:

kenv | egrep "vfs|currdev|kernel_path|kernelname"

W.hâ/t · Oct 17, 2022

You're probably boot from /bootpool/boot. You should recreate the link to /boot

byrnejb said:
My present concern is to discover how this happened.

byrnejb said:
By removing /boot and then adding it back via mkdir

I don't use this setup so I can't tell you if there are extra steps to make an upgrade. Othewise it might just be an issue with 'freebsd-update' and symlink to /boot

byrnejb · Oct 18, 2022

Alain De Vos said:
mount -v /

# mount -v /
mount: /: unknown special file or file system

byrnejb · Oct 18, 2022

richardtoohey2 said:
Anything in /var/log/messages e.g.

syslogd: kernel boot file is /boot/kernel/kernel

(This is from a UFS system so not sure if any difference with ZFS)

Code:

grep boot /var/log/messages
Oct 17 15:49:00 vhost03 syslogd: kernel boot file is /boot/kernel/kernel

Code:

# ll /boot/kernel/kernel
ls: /boot/kernel/kernel: No such file or directory

monwarez · Oct 18, 2022

Does the bootpool still exist ? If so then it means that you are booting from it and then the other pool is mounted via vfs.root.mountfrom
And after the kernel is loaded it does not need to access to the kernel file, so having the /boot is not really necessary to have it to boot.

byrnejb · Oct 18, 2022

monwarez said:
Does the bootpool still exist ? If so then it means that you are booting from it and then the other pool is mounted via vfs.root.mountfrom
And after the kernel is loaded it does not need to access to the kernel file, so having the /boot is not really necessary to have it to boot.

Code:

# tree /bootpool
/bootpool

0 directories, 0 files

# tree /boot
/boot
├── entropy
└── kernel

# lsvfs
Filesystem                              Num  Refs  Flags
-------------------------------- ---------- -----  ---------------
devfs                            0x00000071     1  synthetic, jail
cd9660                           0x000000bd     0  read-only
nfs                              0x0000003a     0  network
procfs                           0x00000002     0  synthetic, jail
tmpfs                            0x00000087     0  jail
ufs                              0x00000035     0
msdosfs                          0x00000032     0
zfs                              0x000000de   406  jail, delegated-administration


# cat /etc/fstab
# Device        Mountpoint    FStype    Options        Dump    Pass#
/dev/ada0p3.eli        none    swap    sw        0    0
/dev/ada1p3.eli        none    swap    sw        0    0
/dev/ada2p3.eli        none    swap    sw        0    0
/dev/ada3p3.eli        none    swap    sw        0    0
fdesc            /dev/fd    fdescfs    rw,late,failok    0    0

SirDice · Oct 18, 2022

zfs list

byrnejb · Oct 18, 2022

SirDice said:
zfs list

Code:

# zfs list
NAME                            USED  AVAIL     REFER  MOUNTPOINT
zroot                          1.31T  3.71T      140K  /zroot
zroot/ROOT                     7.36G  3.71T      140K  none
zroot/ROOT/13.1-RELEASE-p2-1   11.6K  3.71T     4.35G  /
zroot/ROOT/_2022-10-17_154256  11.6K  3.71T     4.34G  /
zroot/ROOT/default             7.36G  3.71T     4.34G  /
zroot/tmp                      9.00M  3.71T      232K  /tmp
zroot/usr                       901M  3.71T      140K  /usr
zroot/usr/home                  140K  3.71T      140K  /usr/home
zroot/usr/ports                 900M  3.71T      900M  /usr/ports
zroot/usr/src                   244K  3.71T      140K  /usr/src
zroot/var                      3.46G  3.71T      140K  /var
zroot/var/audit                 244K  3.71T      140K  /var/audit
zroot/var/crash                 256K  3.71T      140K  /var/crash
zroot/var/log                  83.4M  3.71T     27.4M  /var/log
zroot/var/mail                  465K  3.71T      174K  /var/mail
zroot/var/tmp                  3.38G  3.71T     3.38G  /var/tmp
zroot/vm                       1.29T  3.71T     13.3G  /zroot/vm
zroot/vm/inet13                 235G  3.71T      157K  /zroot/vm/inet13
zroot/vm/inet13/disk0           235G  3.71T      235G  -
zroot/vm/inet14                 111G  3.71T      145K  /zroot/vm/inet14
zroot/vm/inet14/disk0           111G  3.71T      111G  -
zroot/vm/inet16                67.8G  3.71T      157K  /zroot/vm/inet16
zroot/vm/inet16/disk0          67.8G  3.71T     67.8G  -
zroot/vm/inet17                 389G  3.71T      163K  /zroot/vm/inet17
zroot/vm/inet17/disk0           389G  3.71T      389G  -
zroot/vm/inet18                 114G  3.71T      145K  /zroot/vm/inet18
zroot/vm/inet18/disk0           114G  3.71T      114G  -
zroot/vm/inet19                 251G  3.71T      157K  /zroot/vm/inet19
zroot/vm/inet19/disk0           251G  3.71T      251G  -
zroot/vm/samba-01               134G  3.71T      157K  /zroot/vm/samba-01
zroot/vm/samba-01/disk0         134G  3.71T      128G  -
zroot/vm/samba-03              2.80G  3.71T      140K  /zroot/vm/samba-03
zroot/vm/samba-03/disk0        2.80G  3.71T     2.80G  -
zroot/vm/samba-04              3.58G  3.71T      140K  /zroot/vm/samba-04
zroot/vm/samba-04/disk0        3.58G  3.71T     3.58G  -

monwarez · Oct 18, 2022

Are you booting from uefi or legacy bios ?
What does gpart show give you ?

SirDice · Oct 18, 2022

As this is an encrypted setup, you should also have a bootpool. This contains the unencrypted /boot filesystem. You need this or else you won't be able to boot the machine. It needs unencrypted access in order to load the kernel, once the kernel is active the *.eli drives can be unlocked, and the system continues to boot from the filesystems on zroot. This is why you had a symlink from /boot to /bootpool/boot. /bootpool/boot is unencrypted and used to boot the system, it resides on the bootpool zpool and is separate from the zroot zpool.

byrnejb · Oct 18, 2022

The issue with the upgrade was that freebsd-update checks to see if kern.bootfile is a directory:

Code:

   737          # Figure out what directory contains the running kernel
   738          BOOTFILE=`sysctl -n kern.bootfile`
   739          KERNELDIR=${BOOTFILE%/kernel}
   740          if ! [ -d ${KERNELDIR} ]; then
   741                  echo "Cannot identify running kernel"
   742                  exit 1
   743          fi

The upgrade to 13.1 would not proceed unless this was true. Thus I made the changes to /boot. I made no changes to /bootpool. It was empty when I first checked it and it remains so.

I can easily rename /boot and create a softlink to /bootpool with that name. However, that will not solve the problem with freebsd-update. How do I proceed?

SirDice · Oct 18, 2022

byrnejb said:
I made no changes to /bootpool. It was empty when I first checked it and it remains so.

It simply wasn't mounted.

W.hâ/t · Oct 18, 2022

Code:

$ ln -s /boot boot
$ [ -d boot ]
$ echo $?
0

byrnejb · Oct 18, 2022

Is it correct to infer that IF I link /boot to /bootpool and then restart that the system will come up correctly with FILE] /bootpool[/FILE] mounted?

SirDice · Oct 18, 2022

byrnejb said:
with /bootpool mounted?

No, it's probably not automatically mounted. Which is why it was empty in the first place. Check what your settings are: zfs list -o name,canmount,mountpoint bootpool

byrnejb · Oct 18, 2022

SirDice said:
No, it's probably not automatically mounted. Which is why it was empty in the first place. Check what your settings are: zfs list -o name,canmount,mountpoint bootpool

Code:

# zfs list -o name,canmount,mountpoint bootpool
cannot open 'bootpool': dataset does not exis

Alain De Vos · Oct 18, 2022

You have a running kernel without a mounted root filesystem.

Normally one of these 3:

Code:

zroot/ROOT/13.1-RELEASE-p2-1   11.6K  3.71T     4.35G  /
zroot/ROOT/_2022-10-17_154256  11.6K  3.71T     4.34G  /
zroot/ROOT/default             7.36G  3.71T     4.34G  /

should be mounted via legacy zfs mount.

Can you share the output of:

Code:

kenv | egrep "vfs|currdev|kernel_path|kernelname"

And of

Code:

zpool get bootfs

byrnejb · Oct 18, 2022

Code:

# kenv | egrep "vfs|currdev|kernel_path|kernelname"
currdev="zfs:bootpool:"
kernel_path="/boot/kernel"
kernelname="/boot/kernel/kernel"
vfs.root.mountfrom="zfs:zroot/ROOT/default"

# zpool get bootfs
NAME   PROPERTY  VALUE               SOURCE
zroot  bootfs    zroot/ROOT/default  local

Alain De Vos · Oct 18, 2022

That looks ok. You boot from the bootpool-zpool and normally this should mount the zroot-zpool.
The output of mount & zfs mount

byrnejb · Oct 18, 2022

Code:

# mount
zroot/ROOT/default on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs)
zroot/var/mail on /var/mail (zfs, local, nfsv4acls)
zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/var/log on /var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/audit on /var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls)
zroot/var/crash on /var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/usr/src on /usr/src (zfs, local, noatime, nfsv4acls)
zroot/usr/home on /usr/home (zfs, local, noatime, nfsv4acls)
zroot on /zroot (zfs, local, noatime, nfsv4acls)
zroot/var/tmp on /var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/vm on /zroot/vm (zfs, local, noatime, nfsv4acls)
zroot/vm/samba-03 on /zroot/vm/samba-03 (zfs, local, noatime, nfsv4acls)
zroot/vm/samba-04 on /zroot/vm/samba-04 (zfs, local, noatime, nfsv4acls)
zroot/vm/inet17 on /zroot/vm/inet17 (zfs, local, noatime, nfsv4acls)
zroot/vm/inet13 on /zroot/vm/inet13 (zfs, local, noatime, nfsv4acls)
zroot/vm/inet18 on /zroot/vm/inet18 (zfs, local, noatime, nfsv4acls)
zroot/vm/inet16 on /zroot/vm/inet16 (zfs, local, noatime, nfsv4acls)
zroot/vm/inet14 on /zroot/vm/inet14 (zfs, local, noatime, nfsv4acls)
zroot/vm/samba-01 on /zroot/vm/samba-01 (zfs, local, noatime, nfsv4acls)
zroot/vm/inet19 on /zroot/vm/inet19 (zfs, local, noatime, nfsv4acls)

Code:

# zfs mount
zroot/ROOT/default              /
zroot/var/mail                  /var/mail
zroot/tmp                       /tmp
zroot/var/log                   /var/log
zroot/var/audit                 /var/audit
zroot/usr/ports                 /usr/ports
zroot/var/crash                 /var/crash
zroot/usr/src                   /usr/src
zroot/usr/home                  /usr/home
zroot                           /zroot
zroot/var/tmp                   /var/tmp
zroot/vm                        /zroot/vm
zroot/vm/samba-03               /zroot/vm/samba-03
zroot/vm/samba-04               /zroot/vm/samba-04
zroot/vm/inet17                 /zroot/vm/inet17
zroot/vm/inet13                 /zroot/vm/inet13
zroot/vm/inet18                 /zroot/vm/inet18
zroot/vm/inet16                 /zroot/vm/inet16
zroot/vm/inet14                 /zroot/vm/inet14
zroot/vm/samba-01               /zroot/vm/samba-01
zroot/vm/inet19                 /zroot/vm/inet19

W.hâ/t · Oct 18, 2022

Code:

zpool import -f bootpool

byrnejb · Oct 19, 2022

Importing/bootpool worked. However I am still encountering a problem with freebsd-update.

Code:

# freebsd-version
13.1-RELEASE-p2

# uname -a
FreeBSD vhost03.hamilton.harte-lyne.ca 13.1-RELEASE-p2 FreeBSD 13.1-RELEASE-p2 GENERIC amd64

# freebsd-update fetch
src component not installed, skipped
Cannot identify running kernel

As I wrote above this issue appears to be caused by this code in /usr/sbin/freebsd-update:

Code:

   737          # Figure out what directory contains the running kernel
   738          BOOTFILE=`sysctl -n kern.bootfile`
   739          KERNELDIR=${BOOTFILE%/kernel}
   740          if ! [ -d ${KERNELDIR} ]; then
   741                  echo "Cannot identify running kernel"
   742                  exit 1
   743          fi

sysctl -n kern.bootfile returns /boot/kernel/kernel which does not exist:

Code:

# ls -l "$(sysctl -n kern.bootfile)"
ls: /boot/kernel/kernel: No such file or directory

Code:

# ls -l /boot
lrwxr-xr-x  1 root  wheel  9 Oct 18 15:44 /boot -> /bootpool

However
# ls -ld /boot/boot
drwxr-xr-x 15 root wheel 73 Sep 23 09:47 /boot/boot
[/code]

My question is what should /boot point to: /bootpool or /bootpool/boot

Nothing in /boot; nothing in /boot/kernel; system boots regardless

byrnejb

Alain De Vos

richardtoohey2

Alain De Vos

W.hâ/t

byrnejb

byrnejb

monwarez

byrnejb

SirDice

Administrator

byrnejb

monwarez

SirDice

Administrator

byrnejb

SirDice

Administrator

W.hâ/t

byrnejb

SirDice

Administrator

byrnejb

Alain De Vos

byrnejb

Alain De Vos

byrnejb

W.hâ/t

byrnejb