ZFS Migrate system from GELI encrypted zfs from a single disk to a RAID-1 mirror with two SSDs

[madwebness]

My current installation of FreeBSD 13.2 is on a single SSD disk. Unfortunately, at the time I was installing it, I had Linux running on the other two SSDs, which I now intend to use for FreeBSD. Those 2 SSD were, in fact, configured as RAID-1 mirror zfs under Linux and that worked great. Only, I could tell it was clearly the case, that those two zfs Linux disks were not GELI-encrypted. Rather, only zsh pools were encrypteed, while the UEFI and boot partitions remained unencrypted.

I now want to migrate my current installation (on the same PC of, course) from this single GELI encrypted disk to the two (now free) SSDs. The file system shall remain the same - zfs, but I want it to be RAID-1 now. And I would like to keep the encryption. I don't mind re-encrypting it, but it would be much preferred to have both RAID SSDs to be GELI encrypted, rather then the option where part of the system is booted already before I get to enter the password. I don't know if it's possible, however.

Please advise on the easiest and safest way to perform this operation. Thank you.

 

[T-Daemon]

I could tell it was clearly the case, that those two zfs Linux disks were not GELI-encrypted. Rather, only zsh pools were encrypteed, while the UEFI and boot partitions remained unencrypted.

That will be also the case after migrating on the Linux disks. The UEFI partition must remain unencrypted, otherwise the FreeBSD system can't boot.

have both RAID SSDs to be GELI encrypted, rather then the option where part of the system is booted already before I get to enter the password. I don't know if it's possible

It is possible.

Eventually simulate the migration on virtual machine first, created for example in VirtualBox.
Are all of the disks the same size?

If there important data on the FreeBSD system, backup.

Eventually dd(1) random data on the two Linux disks. If not, gpart destroy -F <Linux disks>

Copy partition scheme and table to Linux disks

gpart backup <FreeBSD disk> | gpart restore -F disk1 disk2

Best is to GPT label all partitions. The FreeBSD disk, when installed guided, not manually, has already those labels (viewable with gpart show -l). Label the Linux disks ( gpart modify -i <index> -l <name> <disk>).
You can choose any label you wish. For example

partition type old label  Linux disks label
efi             efi0      FBSD-efi0  FBSD-efi1
freebsd-boot    boot0     FBSD-boot0 ...
freebsd-swap    swap0     FBSD-swap0 ...
freebsd-zfs     zfs0      FBSD-zfs0  ...

dd(1) efi, freebsd-boot partitions on Linux disk partitions.

encrypt swap

geli onetime gpt/FBSD-swap0 gpt/FBSD-swap1

GELI initialize provider with the same password the FreeBSD system has, attach provider, check status. Use the GPT labels:

geli init -g -l 256 -s 4096 gpt/FBSD-zfs0 gpt/FBSD-zfs1
geli attach gpt/FBSD-zfs0 gpt/FBSD-zfs1
geli status

Attach disks to FreeBSD pool disk ( mentioned here as disk0p4.eli)

zpool attach zroot disk0p4.eli gpt/FBSD-zfs0.eli

Wait untill finished resilver (check zpool-status(8)).
Attach second disk.

zpool attach zroot disk0p4.eli gpt/FBSD-zfs1.eli

Edit /etc/fstab, add swap devices of Linux disks by theire GPT label, remove FreeBSD disk swap.

/dev/gpt/FBSD-swap0.eli  none  swap  sw  0  0

Same with FBSD-swap1.

Wait until finished resilver. Then zpool detach zroot disk0p4.eli

Reboot system.

When the Linux disks are larger than the FreeBSD disk,

gpart backup ... | gpart restore -F ...
gpart resize -i 4 ...

If you need more detailed instructions, please ask.

 

[madwebness]

I think there was some difference in how Linux disks were encrypted and how this one is encrypted. In Linux, it started the booting process and I could see the GPU drivers were loaded, because fonts became much smaller already and it started printing loads of info. Only then it prompted me for the encryption password. Perhaps the case was that the /root zfs was unencrypted and only /home was?

On FreeBSD right now, when I start the PC, I see nothing, but that GELI password prompt. Only after I enter the password, FreeBSD logo appears and it starts booting.

Another thing to mention: my current FreeBSD disk is larger than the two Linux disks, but the used storage on it does not exceed the capacity of the Linux disks.

 

[madwebness]

Sorry, I misread your reply. Basically, what you are saying, it's impossible to clone everything to another disk and make it such that it's encrypted BEFORE the OS starts booting? Or, to be more precise, impossible if I want to have zfs RAID-1, correct?

 

[T-Daemon]

my current FreeBSD disk is larger than the two Linux disks,

Then the above method won't work

It's not possible to attach smaller disks to a zpool to form a mirror.

my current FreeBSD disk is larger than the two Linux disks, but the used storage on it does not exceed the capacity of the Linux disks.

The only method in your setup is to install fresh on the two Linux disks (Root-on-ZFS encrypted mirror, choose another pool name, other than "zroot" (i.e. zroot2, zroot-fbsd). Then

• create on the old FreeBSD sytem a recursive snapshot of all descendent datasets ( zfs snap -r zroot@snapname)
• boot the new system
• import the old system on a alternative mount point: zpool import -R /mnt zroot
• zfs-send(8) | zfs-receive(8) the snapshot of the old system to the new. That will transfer all applications installed, theire databases, user accounts, theire passwords, system configurations, etc.
• dd(1) the efi and "freebsd-boot" partitions to the second Linux disk of the mirror, those are empty. In case the first disk fails, the other will be able to boot. Make sure "if=" <disk/partition> is the one with the FreeBSD loader.efi, not the empty on the secondary disk.

what you are saying, it's impossible to clone everything to another disk and make it such that it's encrypted BEFORE the OS starts booting? Or, to be more precise, impossible if I want to have zfs RAID-1, correct?

No. You said

it was clearly the case, that those two zfs Linux disks were not GELI-encrypted. Rather, only zsh pools were encrypteed, while the UEFI and boot partitions remained unencrypted.

The UEFI (efi) and "freebsd-boot" partitions can't be encryped (as they aren't on the old FreeBSD disk), otherwise the system can't be booted. That's because the FreeBSD efi loader is on the "efi" partition (on a MSDOS file system). The "freebsd-zfs" (and eventually "freebsd-swap") partition is geli(8) enabled to boot from a encryped root filesystem, it asks for a passphrase, before the kernel is loaded.

I suggest you practice in a virtual machine, same setup, before carry out on the real system. VirtualBox is good, it`s available on FreeBSD. If you decide to use it, Choose "efi" in the motherboard settings.(8) manual page" href="https://man.freebsd.org/cgi/man.cgi?query=geli[\man] encrypted.

When the system is powered up the UEFI of the motherboard finds the FreeBSD efi loader on the efi partition, let have it to take over to boot the OS. The FreeBSD loader probes all disks and partitions for a kernel to boot, if that kernel resides in a geli encrypted partition, and that partition is geli(8) enabled to boot from a encryped root filesystem, it asks for a passphrase, before the kernel is loaded.

I suggest you practice in a virtual machine, same setup, before carry out on the real system. VirtualBox is good, it`s available on FreeBSD. If you decide to use it, Choose "efi" in the motherboard settings.&sektion=8&manpath=freebsd-release-ports">geli[\man] encrypted.

When the system is powered up the UEFI of the motherboard finds the FreeBSD efi loader on the efi partition, let have it to take over to boot the OS. The FreeBSD loader probes all disks and partitions for a kernel to boot, if that kernel resides in a geli encrypted partition, and that partition is geli(8) enabled to boot from a encryped root filesystem, it asks for a passphrase, before the kernel is loaded.

I suggest you practice in a virtual machine, same setup, before carry out on the real system. VirtualBox is good, it`s available on FreeBSD. If you decide to use it, Choose "efi" in the motherboard settings.(8)

 

[T-Daemon]

Not sure what's going on here, but that's not what I wrote (on Android Firefox Focus) and formated:

I can't edit it to correct. In editor mode (the image above) it looks as it should be.

 

[T-Daemon]

Removed [/man], which point to section 8 of manuals, but it doesn't change.

I suggest you practice in a virtual machine, same setup, before carry out on the real system. VirtualBox is good, it`s available on FreeBSD. If you decide to use it, Choose "efi" in the motherboard settings.[/man]

 

[madwebness]

It's not possible to attach smaller disks to a zpool to form a mirror.

What if I made my current FreeBSD zfs pool smaller?

As for Virtual Machines... Bloody VirtualBox wouldn't even start a VM on my FreeBSD machine. I wasted a day to set it up. Bhyve is on my list of "learn soon", but no time right now. I just need a working system prone to disk failures. I do have backups set up, but recovering from backups, even if you use the nicest tools, is not a pleasant activity at all.

I'm think now, what would be faster... Reinstall FreeBSD on those two SSDs and then just copy every config I have and install every app I installed or what you suggested is simpler and more robust.