ZFS GELI vs ZFS encrypted dataset

J

jbo@

Developer
I've been using ZFS encrypted datasets pretty much since the first day of availability. On this forum and also the mailing lists I see a lot of people running ZFS "on top of" GELI.
I'd like to ask: Is that usually more of a legacy thing or are there situations where one would prefer ZFS on GELI rather than just using native ZFS encrypted datasets?
 
Encrypted ZFS has only been available since the import of OpenZFS with 13.0. Before that the only way to get an encrypted volume was by leveraging geli(8) and putting ZFS on top of that.

jbodenmann said:
Is that usually more of a legacy thing or are there situations where one would prefer ZFS on GELI rather than just using native ZFS encrypted datasets?
Click to expand...
One thing I've noticed with GELI+ZFS is when you get disk errors, because ZFS isn't talking directly to the disk but to the geli(8) layer, bad sectors cause some real issues. geli(8) freaks out, and ZFS on top of it just stalls without errors. Until it got enough timeouts and just drops the pool.
 
Jup - I get that. Hence my question: Is using ZFS on top of GELI mostly a legacy thing or are there situations where one would prefer to use that over encrypted ZFS (assuming that it is available)?
 
So far I'm preferring the ZFS encryption. But it has a few drawbacks. Even if the dataset is locked you can still see that it exists. In that respect GELI+ZFS was 'better' as it also blocked access to the entire ZFS structure.
 
Can you boot from a ZFS-encrypted root partition? I think that's only possible with GELI.

Also, I remember reading that GELI would be faster, but I can't speak from personal experience (never used ZFS encryption).
 
bsduck said:
Can you boot from a ZFS-encrypted root partition? …
Click to expand...

I suspect not. I mean, I should not expect the loader to present a prompt to enter a passphrase; <https://cgit.freebsd.org/src/log/?qt=grep&q=loader&h=main> no mention of zfs on the first page.


What's below is essentially fixed, however this is for the zfskeys service script – long after boot:
– from /var/log/console.log yesterday, with 50e244964e9:

Code: 
Apr  7 01:17:35 mowa219-gjp4-8570p-freebsd kernel: Key file /usr/home/grahamperrin/Documents/personal/VirtualBox.key not found, empty or unreadable. Skipping Transcend/VirtualBox..
Apr  7 01:17:35 mowa219-gjp4-8570p-freebsd kernel: Loading key for Transcend/VirtualBox from file:///usr/home/grahamperrin/Documents/personal/VirtualBox.key..

– essentially fixed, however the first of those two lines is debatably spurious (the file that's reportedly not found is found a split-second later).
 
SirDice said:
Encrypted ZFS has only been available since the import of OpenZFS with 13.0. Before that the only way to get an encrypted volume was by leveraging geli(8) and putting ZFS on top of that.


One thing I've noticed with GELI+ZFS is when you get disk errors, because ZFS isn't talking directly to the disk but to the geli(8) layer, bad sectors cause some real issues. geli(8) freaks out, and ZFS on top of it just stalls without errors. Until it got enough timeouts and just drops the pool.
Click to expand...
Hi SirDice,
This is surprising to read, in one of the last journals of freebsdfoundation they were still saying geli and zfs is OK. The issue you mention about the bad sectors, I though this was exactly what ZFS was doing, keeping track of bads ones and never use them again. But if I understand correctly GELI is the issue here. And just to be clear I believe you, but gives me the feeling I think my data is safe but it's maybe not at all.

Personally I learned a lot from ZFS by having situations where failures or crashes occurred. The more crashes the better, then I learn more. With FreeBSD on virtualbox on another OS, the chance of crashes are higher. But I wonder, how exactly do you know that the sector is bad? Is there a command in zfs or with zdb that can give you all bad sectors?

And coming back to the geli-zfs story, does it mean in the future ideally we better move to native encryption?

Thanks in advance!

Best Regards,
R
 
Let me comment a bit on the two encryption approaches. Both are still very viable today, but serve a different kind of audience.

GELI + ZFS is a full disk encryption scheme, 0 information is leaked to the attacker that would insert said hard drive in its own computer. GELI + ZFS is perfectly fine to keep ZFS working as it should, fixing bad sectors, check summing, etc. I think GELI has the option to also enable that feature but that would collide with ZFS, so only use GELI for the encryption part. You can boot from it (support in the bootloader is present).

ZFS native encryption is a dataset encryption scheme (not full DISK/GEOM), you can google a bit about this but it does leak certain metadata, like dataset names, sizes of files?? (don't quote me on the latter) etc. Depending on your use case, one will be better suited than the other. For our home server applications, typically, ZFS native encryption is fine. The bootloader has not been modified yet to boot from ZFS native encryption. It has been modified to support booting from a ZSTD compression enabled root (both are OpenZFS features).
 
Last edited:
driesm Thank you for summarizing/comparing those two methods.

driesm said:
The bootloader has not been modified yet to boot from ZFS native encryption.
Click to expand...
The "yet" part seems particularly interesting to me. Is there any evidence that would result in reasons to believe that in the future we'll be able to boot FreeBSD from an encrypted ZFS dataset (where "encrypted ZFS dataset) is referring to a ZFS dataset that was encrypted through ZFS's native encryption capabilities)?
 
  • Thanks
Reactions: veg
SirDice said:
One thing I've noticed with GELI+ZFS is when you get disk errors, because ZFS isn't talking directly to the disk but to the geli(8) layer, bad sectors cause some real issues. geli(8) freaks out, and ZFS on top of it just stalls without errors. Until it got enough timeouts and just drops the pool.
Click to expand...

This should not be a problem.
Weren't we running GEOM NOP devices for sector size translation back 10 years ago when a bunch of WD Green drives were "misreporting" sector size?

P.S. there are a lot of GELI-pros but one con would be that ZFS is portable and GELI isn't (in practice). If your storage setup depends on a GELI feature then it's bound to FreeBSD versions that implement that feature. For 99.9% of our cases this is irrelevant but it's still worth to mention.
 
jbodenmann said:
… reasons to believe that in the future we'll be able to boot FreeBSD from an encrypted ZFS dataset …
Click to expand...

Whilst the bootloader has not yet gained support, it should (I guess) be possible to begin with an encrypted pool/ROOT, then install FreeBSD the long way, then whenever you need to boot a boot environment e.g. pool/ROOT/default: first, boot from something that allows you to unlock the ROOT. That thing might be a minimalist installation of FreeBSD on a USB flash drive.

That's clumsily worded, but hopefully enough for people to get the idea. Not to say that it should be done (obviously, it's smoother to use GELI at the moment), but it probably can be done, in theory.

Postscript

grahamperrin said:
FreeBSD bug 263171 – loader(8) and boot loader menu support for boot with OpenZFS-encrypted ROOT
Click to expand...

Holistically, support for OpenZFS-encrypted ROOT should be added to bsdinstall(8), however I don't imagine this happening. Gut feeling: it'll be a feature of a modernised installer (not bsdinstall).

More realistic (than bsdinstall): 263234 – Add support for OpenZFS encryption to adduser.
 
grahamperrin@ said:
Holistically, support for OpenZFS-encrypted ROOT should be added to bsdinstall(8), however I don't imagine this happening. Gut feeling: it'll be a feature of a modernised installer (not bsdinstall).
Click to expand...
Has this effort started, and if so, would anyone know where to best track its progress, by any chance?

In the meantime, I’d love to figure out what’s considered best practice for manually (until an installer implements it) setting up boot on a natively encrypted root dataset.

Cheers!
 

Attachments

  • yes-YES.png
    yes-YES.png
    827.3 KB · Views: 228
jbo@ said:
[…] Is that usually more of a legacy thing or are there situations where one would prefer ZFS on GELI rather than just using native ZFS encrypted datasets?
Click to expand...
Human brains are horribly unreliable storage. 🧠 Considering that I usually deploy full‑disk encryption, geli(8) allows you to have two user keys (≈ passphrases). One’s for daily use, the other is a contingent passphrase. 👩‍🚀 ZFS encryption does not give you this option.

Apart from that there’s the booting into ZFS‑encrypted data sets issue that’s already been mentioned. 🚸 Since I don’t like maintaining both kinds of encryption in parallel, I consistently use geli(8) even if ZFS encryption wouldn’t pose a problem (← separate pools).

See also Thread 76973.​
 
Well, ZFS' native encryption works on individual datasets. That's probably great for lots of scenarios like e.g. encrypted home directories.

For "full disk encryption", doing it on the GEOM layer below is just the overall simpler setup (supporting ZFS encryption in the bootloader would need a decision how to handle multiple datasets possibly needed for booting, no such ambiguity with GELI) and makes sure the full pool is encrypted and therefore "hidden" without the key (which ZFS can't do natively).
 
zirias@ said:
Well, ZFS' native encryption works on individual datasets. That's probably great for lots of scenarios like e.g. encrypted home directories.

For "full disk encryption", doing it on the GEOM layer below is just the overall simpler setup (supporting ZFS encryption in the bootloader would need a decision how to handle multiple datasets possibly needed for booting, no such ambiguity with GELI) and makes sure the full pool is encrypted and therefore "hidden" without the key (which ZFS can't do natively).
Click to expand...
Caveat: I don't do encryption on my systems because they are desktops at my house and I'm the only one using them and not everyone is in the same situation as I am.

Biggest Caveat: All this is just my opinion. Agree, disagree, all good with me.

I've often wondered on this, "what is best in encryption". Whole device certainly is easier because "it's all mucked with". But at what price? From what I've read it's easy to wind up with an unbootable system (pay attention to the details).
What does it buy having the boot/root media encrypted?

If the OS/boot/root is not encrypted, what is exposed? DNS info, local IP addresses, packages installed?
I've always been of the mind "It's my data that is important, not the program or OS". Like documents I create, my photos my emails, my financial data (browser cache typically falls into this). All kind of leading towards "put your data in it's own dataset and encrypt it if it's important"; take into account browser cache and "everyone's home directory is a distinct dataset and encrypt them individually".


"Just because one can, doesn't mean one should" is my overall attitude towards device encryption.
 
mer said:
If the OS/boot/root is not encrypted, what is exposed? DNS info, local IP addresses, packages installed?
Click to expand...
Logs, configurations, runtime data (in the worst case even containing secrets), temporary fragments of your personal data (tmp files, swap ...). All in all giving away at least your usage patterns, maybe personal preferences (by software installed), in the worst case ways to access more of your personal data.

mer said:
From what I've read it's easy to wind up with an unbootable system (pay attention to the details).
Click to expand...
No idea where this fear comes from. loader(8)'s GELI support is proven and solid for ages, it just works.
 
  • Like
Reactions: mer
zirias@ said:
No idea where this fear comes from. loader(8)'s GELI support is proven and solid for ages, it just works.
Click to expand...
It's not the software parts, it's more "user ID10T errors" causing the problem. Yes one can render an unencrypted system unbootable, but encryption adds a few more ways to make it "easier" :)
 
You must log in or register to reply here.
Back
Top