Cannot boot from new disk added to zfs boot pool

T

tight_ship_76

I installed FreeBSD 14.2 onto one brand new SSD, on a UEFI machine. Next I used a script I wrote to add a second brand new SSD to the boot pool, as a mirror. Then I removed the first SSD, and attempted to boot from the second SSD, but I get this repeated error message, followed by the mountroot prompt. I've confirmed that the second SSD is indeed at /dev/ada0:


ZFS WARNING: Unable to open gpt/z-ad12345 for writing (error=1)
ZFS WARNING: Unable to open ada0p4 for writing (error=1)
. . .
Mounting from zfs:zroot/ROOT/default failed with error 1.
. . .
mountroot>


I've referenced sections of the bsdinstall script to try and perform the right steps. My script performs the following steps. When the script runs, the first disk (to which FreeBSD was installed) is at /dev/ada0, and the new disk is at /dev/ada1:

sh: 
# wipe the new disk
swapoff [active_swap_device] # repeated to disable all active swap devices
gpart destroy -F /dev/ada1
zpool labelclear -f /dev/ada1
gpart create -s gpt /dev/ada1
gpart destroy -F /dev/ada1

# clone partition table from old disk to new disk
gpart backup /dev/ada0 | gpart restore -F /dev/ada1
zpool labelclear -f /dev/ada1p2

# label freebsd-boot partition with short disk serial number, and set BIOS bootcode
gpart modify -i 2 -l b-ad12345 /dev/ada1
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 /dev/ada1

# label EFI partition, and copy EFI partition from existing disk to new disk
umount /boot/efi
zpool labelclear -f /dev/ada1p1
dd if=/dev/ada0p1 /dev/ada1p1 bs=1m
gpart modify -i 1 -l e-ad12345 /dev/ada1
mount -t msdosfs /dev/ada1p1 /boot/efi
test -f /boot/efi/efi/freebsd/loader.efi # Note: confirm good copy
efibootmgr --delete -b 0002 # Note: delete existing "ad12345" entry for this disk
efibootmgr --create --activate --loader "/boot/efi/efi/freebsd/loader.efi" --label "FreeBSD OS Disk ad12345"

# Ensure /etc/fstab has only one entry to mount EFI part, and that it's the first mounted disk
sed -i '' '/\dev/\/gpt\/efiboot/d' /etc/fstab
sed -i '' '/\dev\/ada0p1/d' /etc/fstab
echo 'dev/ada0p1 /boot/efi msdosfs rw 2 2' >> /etc/fstab

# label swap partition, ensure only one disk swap entry in /etc/fstab
zpool labelclear -f /dev/ada1p3
gpart modify -i 3 -l s-ad12345 /dev/ada1
sed -i '' '/\/dev\/ad.*swap/d' /etc/fstab
sed -i '' '/\/dev\/nd.*swap/d' /etc/fstab
sed -i '' '/\/dev\/gpt\/s-ad12345/d' /etc/fstab
echo '/dev/gpt/s-ad12345 none swap sw 0 0' >> /etc/fstab
swapon -a

# label zfs partition, attach it to existing disk to make boot pool mirror
zpool labelclear -f /dev/ada1p4
gpart modify -i 4 -l z-ad12345 /dev/ada1
zpool attach -f -w zroot /dev/ada0p4 /dev/gpt/z-ad12345

I can boot normally with both disks present though. And when I boot normally, the new disk appears online with zpool status. Did I miss a step for configuring the zfs partition on the new disk?
 
T-Daemon said:
Try zpool-detach(8) of the partition on the old disk from pool before removing the SSD permanently.

If this is a temporary disk removal try zpool-offline(8).
Click to expand...

Sorry, I should have been more clear with the objective. I'm trying to be prepared for "what if the install disk fails, and I need to boot from the new disk that I added to the mirror?" I intend to run this system, long term, with both disks in a boot mirror.

For my "practicing" right now, I could boot with both disks, and offline or detach the install disk, and then try and boot from the new disk. But some day, if the install disk goes bad, I'd be stuck with the same problem I'm facing right now. On my other FreeBSD systems that did a mirror install, if one disk fails, I can boot from the other disk. That's what I'm trying to achieve right now. Was the suggestion that I should offline/detach the install disk as part of the troubleshooting process, just to get more information?
 
Your main install disk should have an efi partition and if that disk fails you may not be able to boot unless you have the loader which is efi partition in your other mirrored disk too. I'd suggest partition the second disk to be same with main disk then install loader to it's efi partition too.
 
I am doing
sh: 
gpart backup /dev/ada0 | gpart restore -F /dev/ada1
I thought this was the correct way to partition the second disk, because it ensures that the second disk is partitioned the same as the first disk?

For the EFI loader, currently I am just DDing the EFI partition from the old disk to the new disk. I figured that was safer as well, than copying things manually? I also tried this method, but I got the same resulting failure when trying to boot from the second disk:

sh: 
zpool labelclear -f /dev/ada1p1
gpart modify -i 1 -l e-ad12345 /dev/ada1
newfs_mdsos -F 32 -c 1 -L EFISYS /dev/ada1p1
mkdir -p /tmp/efi_tempmount
mount -t msdosfs /dev/ada1p1 /tmp/efi_tempmount
mkdir -p /tmp/efi_tempmount/EFI/freebsd
cp /boot/loader.efi /tmp/efi_tempmount/EFI/freebsd/loader.efi
. . .
# add efibootmgr entry as before

After some trouble, I thought I had succeeded in doing the right things to at least boot from the new disk's EFI partition. The efibootmgr entry I added is visible on my BIOS screen, and the OS gets pretty far into the boot, before I see the error message about failure to mount /dev/ada0p4. Is this command correct, to add the efibootmgr entry?

sh: 
efibootmgr --create --activate --loader "/boot/efi/efi/freebsd/loader.efi" --label "FreeBSD OS Disk ad12345"
 
I have Windows on an M.2 ssd and FreeBSD on the secondary sata ssd. 1 thing that helped me was having FreeBSD on 1 usb stick as opposed to multi-boot with multiple OS via eg ventoy etc.

The initial installer menu on a fresh usb stick with only freebsd helped me a lot. I would install on 1 drive before moving to 2 drives. This simplifies initial installing where you can then get into complex workflow setups eg 2 drives, mirror OS etc. But it is totally up to you. Hope this helps the community.
 

Attachments

  • IMG_5153.png
    IMG_5153.png
    6.8 KB · Views: 11
tight_ship_76 said:
Was the suggestion that I should offline/detach the install disk as part of the troubleshooting process, just to get more information?
Click to expand...
No, I misinterpreted your problem description as a disk swap, not a disk failure of a mirrored pool.

But, no matter, the solution is the same, use zpool-detach(8) to remove the first disk from the pool before removing the disk physically. I suspect if the partition is not removed from the pool, ZFS expects a mirror to boot from.

You can simulate the test objective by failing the first disk artificially, after the second disk is attached and has created a mirror.

!!! Be adviced, the zinject(8) program is described as dangerous in the manual, test on a system without important data, or better in a VM (as I do) before applying on metal.

In this example, ada0p4 is the first disk, gpt/zfs1 the later attached.
Code: 
 # zinject -d ada0p4 -A fault zroot

 # zpool status
  pool: zroot
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
    repaired.
  scan: resilvered 502M in 00:00:03 with 0 errors on Sun Mar  9 21:02:19 2025
config:

    NAME          STATE     READ WRITE CKSUM
    zroot         DEGRADED     0     0     0
      mirror-0    DEGRADED     0     0     0
        ada0p4    FAULTED      0     0     0  too many errors
        gpt/zfs1  ONLINE       0     0     0

errors: No known data errors

zpool-detach(8) ada0p4, remove disk physically (metal or VM), reboot system. No disk controller port swap needed, ZFS doesn't care about changed disk names.

Note: The DEGRADED pool status can be cleared with zpool-clear(8) if the first disk is reattached to the machine.
 
T-Daemon said:
use zpool-detach(8) to remove the first disk from the pool before removing the disk physically. I suspect if the partition is not removed from the pool, ZFS expects a mirror to boot from.
Click to expand...

It was a good suggestion. It worked. I could boot from the second/new disk after I tried the above step.

But then, due to an experimental mistake, I had to start all over, and do a fresh FreeBSD install from a USB stick onto the first disk. So then I tried again: physically add the second disk, run my add-disk script on the second disk, physically remove the first disk (this time, without zpool-detaching), and try to boot from the second disk. This time, it worked! I'm not sure what exactly I did give myself the "could not mount ada0p4 for writing" error. I haven't been able to replicate it in a few more tries.

It's good to know I can format and attach a brand new disk, yank out the other disk, and operate as normal with the new disk. I do wonder what gave me the problems. Is efibootmgr sensitive to PC physical disk slots? I'd been running efibootmgr while swapping disks in slots, so maybe that had something to do with it.
 
You must log in or register to reply here.
Back
Top