Other GELI for second ZFS pool, prompted twice instead of once at boot

RussellASC · Feb 14, 2023

I have a second GELI encrypted ZFS pool separate from my OS disks. At boot the loader prompts for my GELI password for OS, but it does not unlock the non-OS disks with the same key. Instead while booting I am prompted a second time to unlock the disks in zdata.

I've tried using FDE raw w/ GELI (no partitions) and with a GELI protected freebsd-zfs type partition, neither is unlocked at boot loader time.

Yes, all the disks share the same passphrase.

Is there something I can add to /boot/loader.conf which could unlock all of the disk devices at the first password prompt?

T-Daemon · Feb 15, 2023

Please post geli list and uname -a.

RussellASC · Feb 15, 2023

This is a new install, no data yet. I've recreated my RAIDZ pool twice while testing. After the boot loader the geli init prompts for password for all of my geli objects not already loaded. That includes three drives without partitions and geli on the raw disk, and a fourth drive I added a partition to for testing. My goal is just to have all 6 of my geli disks (2x OS, 4xRAIDZ) load from the first password prompt in the boot loader as they use the same password.

Code:

root@odin4:/var/log # uname -a
FreeBSD odin4 13.1-RELEASE-p6 FreeBSD 13.1-RELEASE-p6 GENERIC amd64

Via serial console at boot, BIOS sees 6 disks. Geli prompts for the password for disk0p1, and then also unlocks disk1p1 and continues without unlocking others.

dmesg, the geli in rc clearly sees and unlocks all devices. I have to enter another password at a second prompt.

Code:

messages:Feb 15 13:47:26 odin4 kernel: GEOM_ELI: Device da0p3.eli created.
messages:Feb 15 13:47:26 odin4 kernel: GEOM_ELI: Encryption: AES-XTS 256
messages:Feb 15 13:47:26 odin4 kernel: GEOM_ELI:     Crypto: accelerated software
messages:Feb 15 13:47:26 odin4 kernel: GEOM_ELI: Wrong key for da2. Tries left: 2.
messages:Feb 15 13:47:26 odin4 kernel: Enter passphrase for da2: GEOM_ELI: Device da2.eli created.
messages:Feb 15 13:47:26 odin4 kernel: GEOM_ELI: Encryption: AES-XTS 256
messages:Feb 15 13:47:26 odin4 kernel: GEOM_ELI:     Crypto: accelerated software
messages:Feb 15 13:47:26 odin4 kernel: GEOM_ELI: Device da3.eli created.
messages:Feb 15 13:47:26 odin4 kernel: GEOM_ELI: Encryption: AES-XTS 256
messages:Feb 15 13:47:26 odin4 kernel: GEOM_ELI:     Crypto: accelerated software
messages:Feb 15 13:47:26 odin4 kernel: GEOM_ELI: Device da4.eli created.
messages:Feb 15 13:47:26 odin4 kernel: GEOM_ELI: Encryption: AES-XTS 256
messages:Feb 15 13:47:26 odin4 kernel: GEOM_ELI:     Crypto: accelerated software
messages:Feb 15 13:47:26 odin4 kernel: GEOM_ELI: Device da1p3.eli created.
messages:Feb 15 13:47:26 odin4 kernel: GEOM_ELI: Encryption: AES-XTS 256
messages:Feb 15 13:47:26 odin4 kernel: GEOM_ELI:     Crypto: accelerated software
messages:Feb 15 13:47:26 odin4 kernel: GEOM_ELI: Device da5p1.eli created.
messages:Feb 15 13:47:26 odin4 kernel: GEOM_ELI: Encryption: AES-XTS 256
messages:Feb 15 13:47:26 odin4 kernel: GEOM_ELI:     Crypto: accelerated software
messages:Feb 15 13:47:26 odin4 kernel: GEOM_ELI: Device mirror/swap.eli created.
messages:Feb 15 13:47:26 odin4 kernel: GEOM_ELI: Encryption: AES-XTS 128
messages:Feb 15 13:47:26 odin4 kernel: GEOM_ELI:     Crypto: accelerated software

The first two SAMSUNG SSD's are the OS zroot, setup by the installer. The remaining four WDC disks will be my RAIDZ.

Code:

root@odin4:/var/log # camcontrol devlist
<ATA SAMSUNG MZ7KM480 CSLB>        at scbus1 target 0 lun 0 (pass0,da0)
<ATA SAMSUNG MZ7KM480 CSLB>        at scbus1 target 1 lun 0 (pass1,da1)
<ATA WDC WD40EFRX-68N 0A82>        at scbus1 target 4 lun 0 (pass2,da2)
<ATA WDC WD40EFRX-68N 0A82>        at scbus1 target 5 lun 0 (pass3,da3)
<ATA WDC WD40EFRX-68N 0A82>        at scbus1 target 6 lun 0 (pass4,da4)
<ATA WDC WD40EFRX-68N 0A82>        at scbus1 target 7 lun 0 (pass5,da5)
<AHCI SGPIO Enclosure 2.00 0001>   at scbus8 target 0 lun 0 (ses0,pass6)

Partitions are only setup on some devices.

Code:

root@odin4:/var/log # gpart show
=>       40  937703008  da0  GPT  (447G)
         40       1024    1  freebsd-boot  (512K)
       1064        984       - free -  (492K)
       2048   16777216    2  freebsd-swap  (8.0G)
   16779264  920922112    3  freebsd-zfs  (439G)
  937701376       1672       - free -  (836K)

=>       40  937703008  da1  GPT  (447G)
         40       1024    1  freebsd-boot  (512K)
       1064        984       - free -  (492K)
       2048   16777216    2  freebsd-swap  (8.0G)
   16779264  920922112    3  freebsd-zfs  (439G)
  937701376       1672       - free -  (836K)

=>        40  7814037088  da5  GPT  (3.6T)
          40        2008       - free -  (1.0M)
        2048  7814033408    1  freebsd-zfs  (3.6T)
  7814035456        1672       - free -  (836K)

da0 and da1 are OS zroot created by the installer. da5 has a single partition because I was trying to see if geli at loader time needed a partition. It was still ignored. da2-da4 are full disk geli w/o partitions.

Code:

root@odin4:/var/log # geli status
           Name  Status  Components
      da0p3.eli  ACTIVE  da0p3
        da2.eli  ACTIVE  da2
        da3.eli  ACTIVE  da3
        da4.eli  ACTIVE  da4
      da1p3.eli  ACTIVE  da1p3
      da5p1.eli  ACTIVE  da5p1
mirror/swap.eli  ACTIVE  mirror/swap

Only da5 is used in the data RAIDZ currently for testing. da2-da4 are unused.

Code:

root@odin4:/var/log # zpool list -v
NAME            SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zraid          3.62T   624K  3.62T        -         -     0%     0%  1.00x    ONLINE  -
  da5p1.eli    3.62T   624K  3.62T        -         -     0%  0.00%      -    ONLINE
zroot           436G  1.69G   434G        -         -     0%     0%  1.00x    ONLINE  -
  mirror-0      436G  1.69G   434G        -         -     0%  0.38%      -    ONLINE
    da0p3.eli      -      -      -        -         -      -      -      -    ONLINE
    da1p3.eli      -      -      -        -         -      -      -      -    ONLINE

I confirmed that the geli boot flag was enabled:

Code:

root@odin4:/var/log # geli list da0p3.eli   (first boot disk made by installer)
Geom name: da0p3.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 256
Crypto: accelerated software
Version: 7
UsedKey: 0
Flags: BOOT, GELIBOOT, AUTORESIZE      <<<<<<<<<<<<<<<<<<
KeysAllocated: 110
KeysTotal: 110
Providers:
1. Name: da0p3.eli
   Mediasize: 471512117248 (439G)
   Sectorsize: 4096
   Mode: r1w1e1
Consumers:
1. Name: da0p3
   Mediasize: 471512121344 (439G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1

root@odin4:/var/log # geli list da2.eli (unused FDE w/o partitions, only geli)
Geom name: da2.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 256
Crypto: accelerated software
Version: 7
UsedKey: 0
Flags: BOOT, GELIBOOT, AUTORESIZE   <<<<<<<<<<<<<<
KeysAllocated: 932
KeysTotal: 932
Providers:
1. Name: da2.eli
   Mediasize: 4000787025920 (3.6T)
   Sectorsize: 4096
   Mode: r0w0e0
Consumers:
1. Name: da2
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1

root@odin4:/var/log # geli list da5p1.eli (disk with one partition, geli in raidz)
Geom name: da5p1.eli
State: ACTIVE
EncryptionAlgorithm: AES-XTS
KeyLength: 256
Crypto: accelerated software
Version: 7
UsedKey: 0
Flags: BOOT, GELIBOOT, AUTORESIZE   <<<<<<<<<<<<<<<<<<
KeysAllocated: 932
KeysTotal: 932
Providers:
1. Name: da5p1.eli
   Mediasize: 4000785100800 (3.6T)
   Sectorsize: 4096
   Mode: r1w1e1
Consumers:
1. Name: da5p1
   Mediasize: 4000785104896 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1

So the question remains, why aren't all of the geli devices configured during the boot loader, resulting in a second password prompt?

I am using a serial console over iDRAC with mixed success, and the second password prompt isn't using the serial console. I don't get all of my startup output on the console. I'm troubleshooting that still. That's part of why this is a problem.

T-Daemon · Feb 15, 2023

RussellASC said:
BIOS sees 6 disks

RussellASC said:
So the question remains, why aren't all of the geli devices configured during the boot loader, resulting in a second password prompt?

This is a bug on BIOS systems. It results that the entered GELI passphrase at the very beginning is not exported into the loader environment. A bug report was filed, but it didn't attract any attention.

Here the forums Thread geli-zfs.83316, my posting, and the PR 260566.

You can test it as follows: At the boot menu "3. Escape to loader prompt", enter "show". There should be a kern.geom.eli.passphrase=<passphrase in clear text> variable but there isn't.

Now enter set kern.geom.eli.passphrase=<your passphrase>, then boot.

RussellASC said:
Is there something I can add to /boot/loader.conf which could unlock all of the disk devices at the first password prompt?

As a workaround set the variable in /boot/loader.conf.

RussellASC · Feb 15, 2023

T-Daemon said:
As a workaround set the variable in /boot/loader.conf.

Meaning have the password in the loader.conf? I do want to input the password on boot, but just once.

T-Daemon · Feb 15, 2023

RussellASC said:
Meaning have the password in the loader.conf?

Yes, for the WDC disks.

RussellASC said:
I do want to input the password on boot, but just once.

You can enter the password once, which will unlock the OS disks only. The password to unlock the WDC disks won't be the one you entered at the GELI prompt but the one set in loader.conf as loader variable (it can be another password than the OS disks password).

The GELI unlock process in this setup is as follows: After entering the password at the GELI prompt the OS disks are unlocked. After passing the boot menu, the system bootstraps further configuration information for the loader from loader.conf, including the GELI password for the WDC disks

For the time being there is no other way I'm aware of to unlock those WDC disks. Except from /etc/rc.conf, by automatically attaching the disks with keyfiles. Search in /etc/defaults/rc.conf for "geli".

On a UEFI system there is no such issue. I can't tell whether this problem will be solved in the foreseeable future, or at all. The issue was reported 2021-12-20, since then no one has taken care of it.

RussellASC · Feb 15, 2023

Unfortunately I had to revert to GPT/BIOS because the EFI installer doesn't properly mirror the OS.

258987 – 13.0-RELEASE installer broken redundancy with UEFI and ZFS

bugs.freebsd.org

I haven't fully debugged the serial console issue, and the second prompt I only get on the full remote iDrac java application.

Do you think it's worth reinstalling and manually repairing the EFI installation to get the password unlock function? For instance if the EFI version will be the majority of developer attention in the future, etc.

T-Daemon · Feb 15, 2023

I wouldn't consider the missing EFI loader partition on the redundant disk as such a big problem that you shouldn't use UEFI. It takes only a few seconds to create it manually:

Code:

 # dd if=/dev/da0p1 of=/dev/da1p1 bs=1m.

If one of the OS disks has to be replaced, the creation of the partition scheme and table on the new disk takes even less time. Less than one second:

Code:

 # gpart backup <disk> | gpart restore <new disk>

RussellASC · Feb 16, 2023

Can the EFI loader partition be mirrored with gmirror?

RussellASC · Feb 16, 2023

I tried created a gmirror and writing an MSDOS filesystem, then copying over the EFI files. Afterward I mirrored back over the original efi partition. Won't boot.

Guess I'll just dd over the partition from the first disk to the second.

RussellASC · Feb 17, 2023

After reinstalling with EFI, I dd'ed one efi partition over the other. I tested disk failure and it booted ok.

I setup my 4 WD drives with FDE GELI w/o partitions, and created a new zpool. Only a single password prompt at boot. Much better.