What is the correct procedure to restore a mirrored boot disk?

C

cb000

Today, I am practicing the replacement of a boot disk in an encrypted mirrored setup.
I know how to restore the partition table and ZFS drive using these commands:
Code: 
# Backup the partition table
gpart backup da0 > da0.gpart
# Restore the partition table to the new disk
gpart restore da1 < da0.gpart
# Create encrypted partition
geli init -b -e "AES-XTS" -l 256 -s 4096 da1p4
# Attach the encrypted partition
geli attach da1p4

# Check the zpool status
zpool status zroot
# Offline broken vdev
zpool offline zroot da1p4.eli
# Check if vdev is offline
zpool status zroot
# Replace the broken with the new one
zpool replace zroot da1p4.eli da1p4.eli
# Check the zpool status, should be resilvering
zpool status zroot


For the mirrored swap, it has nothing to do with /etc/fstab, just use gmirror.
Code: 
# Check gmirror status
gmirror status
# Forget the broken swap partition
gmirror forget swap
# Insert the new swap partition into the mirrored swap
gmirror insert swap da1p3
# Check gmirror status, should be SYNCHRONIZING
gmirror status

However, I am unsure about restoring the EFI and freebsd-boot type partitions.

For the EFI partition, can I simply mount the good EFI partition and the new EFI partition, and then rsync everything to the new one? Similarly, can I do the same for the freebsd-boot type partition?

Any guidance or tips would be greatly appreciated!

Thanks in advance!
 
Emrion said:
Take your inspiration there, for instance: https://forums.freebsd.org/threads/update-of-the-bootcodes-for-a-gpt-scheme-x64-architecture.80163/
Click to expand...
It doesn't work.

After I copied the loader.efi to the new EFI partition, I powered off the VM, removed one of the disks, powered it back on, and got the error: ERROR: cannot open /boot/lua/loader.lua: no such file or directory.

It seems that it still points to the old disk, or cannot find the new freebsd-zfs partition. Maybe the problem is due to the GELI encrypted zroot?

1734991463885.png
 
Possible, I never used geli encrypted zpool. Take a look at efibootmgr(8).
In particular, see the output of efibootmgr -v.
Also, what is the result of gpart show for both disks?
 
Emrion said:
Possible, I never used geli encrypted zpool. Take a look at efibootmgr(8).
In particular, see the output of efibootmgr -v.
Also, what is the result of gpart show for both disks?
Click to expand...
I tried to follow this message, but it still does not work for my case.

Code: 
Boot to FW : false
BootCurrent: 0006
BootOrder  : 0008, 0002, 0003, 0004, 0007, 0000, 0005, 0001, 0006
 Boot0008  FreeBSD-Disk1 HD(1,GPT,069f5025-a163-11ef-b304-000c29a04052,0x28,0x82000)/File(\efi\boot\bootx64.efi)
                            gpt/efiboot0:/efi/boot/bootx64.efi /boot/efi//efi/boot/bootx64.efi
 Boot0002  EFI VMware Virtual SCSI Hard Drive (2.0) PciRoot(0x0)/Pci(0x10,0x0)/Scsi(0x2,0x0)
 Boot0003  EFI VMware Virtual SCSI Hard Drive (3.0) PciRoot(0x0)/Pci(0x10,0x0)/Scsi(0x3,0x0)
 Boot0004  EFI VMware Virtual SCSI Hard Drive (4.0) PciRoot(0x0)/Pci(0x10,0x0)/Scsi(0x4,0x0)
 Boot0007  EFI Internal Shell (Unsupported option) MemoryMapped(0xb,0xefe6018,0xf3f5017)/FvFile(c57ad6b7-0515-40a8-9d21-551652854e37)
 Boot0000  EFI VMware Virtual SCSI Hard Drive (5.0) PciRoot(0x0)/Pci(0x10,0x0)/Scsi(0x5,0x0)
 Boot0005  EFI Network PciRoot(0x0)/Pci(0x11,0x0)/Pci(0x0,0x0)/MAC(000c29a04052,0x0)
 Boot0001  EFI VMware Virtual SCSI Hard Drive (0.0) PciRoot(0x0)/Pci(0x10,0x0)/Scsi(0x0,0x0)
+Boot0006* EFI VMware Virtual SCSI Hard Drive (1.0) PciRoot(0x0)/Pci(0x10,0x0)/Scsi(0x1,0x0)

Here is my gpart output:
Code: 
root@freebsd_test:~ # gpart show
=>      34  62914493  da0  GPT  (30G)
        34         6       - free -  (3.0K)
        40    532480    1  efi  (260M)
    532520      1024    2  freebsd-boot  (512K)
    533544       984       - free -  (492K)
    534528  10485760    3  freebsd-swap  (5.0G)
  11020288  51892224    4  freebsd-zfs  (25G)
  62912512      2015       - free -  (1.0M)

=>      40  62914480  da1  GPT  (30G)
        40    532480    1  efi  (260M)
    532520      1024    2  freebsd-boot  (512K)
    533544       984       - free -  (492K)
    534528  10485760    3  freebsd-swap  (5.0G)
  11020288  51892224    4  freebsd-zfs  (25G)
  62912512      2008       - free -  (1.0M)

=>      40  20971440  da2  GPT  (10G)
        40      2008       - free -  (1.0M)
      2048  20967424    1  freebsd-zfs  (10G)
  20969472      2008       - free -  (1.0M)

=>      40  20971440  da3  GPT  (10G)
        40      2008       - free -  (1.0M)
      2048  20967424    1  freebsd-zfs  (10G)
  20969472      2008       - free -  (1.0M)

=>      40  20971440  da4  GPT  (10G)
        40      2008       - free -  (1.0M)
      2048  20967424    1  freebsd-zfs  (10G)
  20969472      2008       - free -  (1.0M)

=>      40  20971440  da5  GPT  (10G)
        40      2008       - free -  (1.0M)
      2048  20967424    1  freebsd-zfs  (10G)
  20969472      2008       - free -  (1.0M)

Code: 
root@freebsd_test:~ # gpart list da1
Geom name: da1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 62914519
first: 40
entries: 128
scheme: GPT
Providers:
1. Name: da1p1
   Mediasize: 272629760 (260M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r1w1e2
   efimedia: HD(1,GPT,069f5025-a163-11ef-b304-000c29a04052,0x28,0x82000)
   rawuuid: 069f5025-a163-11ef-b304-000c29a04052
   rawtype: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
   label: efiboot0
   length: 272629760
   offset: 20480
   type: efi
   index: 1
   end: 532519
   start: 40
2. Name: da1p2
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 272650240
   Mode: r0w0e0
   efimedia: HD(2,GPT,06a5e483-a163-11ef-b304-000c29a04052,0x82028,0x400)
   rawuuid: 06a5e483-a163-11ef-b304-000c29a04052
   rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
   label: gptboot0
   length: 524288
   offset: 272650240
   type: freebsd-boot
   index: 2
   end: 533543
   start: 532520
3. Name: da1p3
   Mediasize: 5368709120 (5.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 273678336
   Mode: r1w1e1
   efimedia: HD(3,GPT,06ad285f-a163-11ef-b304-000c29a04052,0x82800,0xa00000)
   rawuuid: 06ad285f-a163-11ef-b304-000c29a04052
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: swap0
   length: 5368709120
   offset: 273678336
   type: freebsd-swap
   index: 3
   end: 11020287
   start: 534528
4. Name: da1p4
   Mediasize: 26568818688 (25G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 5642387456
   Mode: r1w1e1
   efimedia: HD(4,GPT,06b26943-a163-11ef-b304-000c29a04052,0xa82800,0x317d000)
   rawuuid: 06b26943-a163-11ef-b304-000c29a04052
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: zfs0
   length: 26568818688
   offset: 5642387456
   type: freebsd-zfs
   index: 4
   end: 62912511
   start: 11020288
Consumers:
1. Name: da1
   Mediasize: 32212254720 (30G)
   Sectorsize: 512
   Mode: r3w3e7

Code: 
root@freebsd_test:~ # gpart list da0
Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 62914526
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da0p1
   Mediasize: 272629760 (260M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 20480
   Mode: r0w0e0
   efimedia: HD(1,GPT,333e7b74-c1bb-11ef-9f6a-000c29a04052,0x28,0x82000)
   rawuuid: 333e7b74-c1bb-11ef-9f6a-000c29a04052
   rawtype: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
   label: efiboot1
   length: 272629760
   offset: 20480
   type: efi
   index: 1
   end: 532519
   start: 40
2. Name: da0p2
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 272650240
   Mode: r0w0e0
   efimedia: HD(2,GPT,333eef9a-c1bb-11ef-9f6a-000c29a04052,0x82028,0x400)
   rawuuid: 333eef9a-c1bb-11ef-9f6a-000c29a04052
   rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
   label: (null)
   length: 524288
   offset: 272650240
   type: freebsd-boot
   index: 2
   end: 533543
   start: 532520
3. Name: da0p3
   Mediasize: 5368709120 (5.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 273678336
   Mode: r1w1e1
   efimedia: HD(3,GPT,333f28c8-c1bb-11ef-9f6a-000c29a04052,0x82800,0xa00000)
   rawuuid: 333f28c8-c1bb-11ef-9f6a-000c29a04052
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 5368709120
   offset: 273678336
   type: freebsd-swap
   index: 3
   end: 11020287
   start: 534528
4. Name: da0p4
   Mediasize: 26568818688 (25G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 5642387456
   Mode: r1w1e1
   efimedia: HD(4,GPT,333f7a79-c1bb-11ef-9f6a-000c29a04052,0xa82800,0x317d000)
   rawuuid: 333f7a79-c1bb-11ef-9f6a-000c29a04052
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 26568818688
   offset: 5642387456
   type: freebsd-zfs
   index: 4
   end: 62912511
   start: 11020288
Consumers:
1. Name: da0
   Mediasize: 32212254720 (30G)
   Sectorsize: 512
   Mode: r2w2e4
 
One thing is sure: the loader you copied get executed. The messages on the screen come from it.
The problem seems to be it doesn't find the pool, thus cannot access to files.

On this disk, on the ESP partition, do you have a file /efi/freebsd/loader.env? If yes, what is its content?
 
Emrion said:
One thing is sure: the loader you copied get executed. The messages on the screen come from it.
The problem seems to be it doesn't find the pool, thus cannot access to files.

On this disk, on the ESP partition, do you have a file /efi/freebsd/loader.env? If yes, what is its content?
Click to expand...
Should be no.
Code: 
root@freebsd_test:~ # find / -name loader.env
root@freebsd_test:~ #

root@freebsd_test:~ # tree /boot | grep loader*
│   └── loader.conf
│           └── loader.efi
├── loader
├── loader.4th
├── loader.conf
├── loader.conf.d
├── loader.efi
├── loader.help.bios
├── loader.help.efi
├── loader.help.userboot
├── loader.rc
├── loader_4th
├── loader_4th.efi
├── loader_lua
├── loader_lua.efi
├── loader_simp
├── loader_simp.efi
│   ├── loader.lua
└── zfsloader
 
Emrion said:
I meant on the da1p1 efi partition, you have to mount it.
Click to expand...
No. I cannot find loader.env anywhere...

Code: 
root@freebsd_test:~ # tree /boot/efi
/boot/efi
└── efi
    ├── boot
    │   └── bootx64.efi
    └── freebsd
        └── loader.efi

4 directories, 2 files
root@freebsd_test:~ # tree /boot/efi1
/boot/efi1
└── efi
    ├── boot
    │   └── bootx64.efi
    └── freebsd
        └── loader.efi

4 directories, 2 files
root@freebsd_test:~ # cat /etc/fstab
# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/gpt/efiboot0               /boot/efi       msdosfs rw              2       2
/dev/gpt/efiboot1               /boot/efi1       msdosfs rw              2       2
/dev/mirror/swap                none    swap    sw              0       0
 
Ok. The copy of your disk seems ok. Have you installed pmbr on the da1 mbr and filled da1p2 with gptzfsboot?
If yes, you can try a boot in CSM/BIOS mode to see what you get.

Beside the main problem you have with the detection of your pool at startup, you might want to change your fstab. Here, if you boot with only one disk, you will be dropped in single user mode after kernel has finished its initialisation because efiboot0 or efiboot1 won't exist. You may use the noauto option to prevent this but they will remain unmounted.
 
cb000 said:
geli init -b -e "AES-XTS" -l 256 -s 4096 da1p4
Click to expand...
This won't work with the -b option. You need the -g option.

The -b option doesn't decrypt the geli provider to boot from. It decrypts the provider mid boot, after a kernel is booted.

The -g option enables booting from the encrypted root filesystem.

geli(8)
Code: 
    init       Initialize providers which need to be encrypted.
               ...
               -b                 Try to decrypt this partition during boot,
                                  before the root partition is mounted.  This
                                  makes it possible to use an encrypted root
                                  partition.  One will still need bootable
                                  unencrypted storage with a /boot/ directory,
                                  which can be a CD-ROM disc or USB pen-drive,
                                  that can be removed after boot.
                ...
                -g                Enable booting from this encrypted root
                                  filesystem.  The boot loader prompts for the
                                  passphrase and loads loader(8) from the
                                  encrypted partition.




Try this: Boot installation media, enter "Live System", log in as root, execute
Code: 
 # geli load
 # geli attach  da0p4 da1p4

 Check flags (all system disks must list "GELIBOOT"):
 # geli list | grep -e name -e Flags

Assuming da1p4 is missing the flag
# geli configure -g da1p4


Next, GPT label the ESP partition. gpart restore doesn't restore GPT labels. Otherwise the ESP partition, labeled "efiboot0", can't be found when /etc/fstab is read, and the system halts in single-user mode.
Code: 
 # gpart modify -i 1 -l efiboot0 da1

Create new efi boot menu, assuming da1 is the replacement, check ESP
Code: 
# fstyp /dev/da1p1
msdosfs

delete old, create new
Code: 
 # efibootmgr -B -b 8
 # efibootmgr -c -a -L FreeBSD-Disk1 -l da1p1:/efi/boot/bootx64.efi




Ideal would be following approach:

Code: 
 # gpart backup da0 | gpart restore da1

 # gpart modify -i 1 -l efiboot0 da1

(copy ESP on all system disks)
 # dd if=/dev/da0p1  of=/dev/da1p1 bs=1m
 # dd if=/dev/da0p1  of=/dev/da2p1 bs=1m (this is the replacement disk)

 # geli init -g -l 256 -s 4096 da2p4   (AES-XTS is default, no need to set explicitly)
 # geli attach da2p4

 # zpool replace zroot da0p4.eli da2p4.eli

(delete FreeBSD efi boot menu, create new)
 # mount_msdosfs /dev/da1p1 /mnt
 # efibootmgr -B -b 8
 # efibootmgr -c -a -L FreeBSD-Disk1 -l /mnt/efi/freebsd/loader.efi

Alternatively change the boot order, so the system disks are in front

Check also swapinfo(8) after booting the system if all swap devices are enabled (see /etc/fstab).
 
You must log in or register to reply here.
Back
Top