ZFS How to get OpenZFS 2.0 working on FreeBSD 12.2?

[Morris]

As OpenZFS 2.0 has now been released some people will be interested in getting it up and running to play around with in FreeBSD 12.2 instead of waiting for it being included in FreeBSD 13 by default.

What would be required to get this working? The old ZFS system needs to be disabled and replaced by the OpenZFS kernel mod and the OpenZFS userland tools. Both these ports are slightly outdated at the moment (going by the date it would probably correspond to rc4) but that is presumably only a matter of time.

I understand that the kernel can be recompiled without the legacy ZFS implementation by putting WITHOUT_ZFS in /etc/src.conf. Is that correct?

In /boot/loader.conf we need to remove zfs_load=“YES” and replace it with openzfs_load="YES". Is that correct?

In /etc/rc.conf we need to remove zfs_enable=“YES” and replace it with openzfs_enable="YES”. We also need to add kld_list="openzfs". Is that correct?

The helpful OpenZFS guide on installation on FreeBSD suggests to make sure you correct the setting of PATH as when using the version from ports the ZFS utilities will be in /usr/local/sbin/ and not /sbin/.

Upgrading the pool is obviously an irreversible process so afterwards you can’t take the pool back to stock FreeBSD 12.2 or earlier. Is there some special stuff required to import and/or upgrade the old pool? Can you import but not upgrade the pool and expect it to work (without the new features obviously)?

What is the deal with booting from an OpenZFS volume? The OpenZFS FAQ suggests that booting not available until FreeBSD 13. Is that correct or does the FAQ just say that because it doesn’t want to complicate things by talking about the ports?

What have I missed? Any other things to bear in mind?

 

[SirDice]

I understand that the kernel can be recompiled without the legacy ZFS implementation by putting WITHOUT_ZFS in /etc/src.conf. Is that correct?

src.conf(5):

     WITHOUT_ZFS
             Set to not build ZFS file system kernel module, libraries, and
             user commands.

 

[Morris]

Great news, both the OpenZFS ports are now updated to the final release. They appear to actually be synchronised with the OpenZFS master repository so I assume they will keep getting regular updates and not stay frozen until OpenZFS 2.1 early next year.

Alain De Vos You tried OpenZFS on 12.1 with an older version of the ports, have you tried it using 12.2 now that that’s out and more recent ports?

PMc How did you deal with importing the pool in the end? Have you tried running it without upgrading?

 

[PMc]

Hi Morris,
I think using the rc.d script from current for importing would not be too much of a hassle - I did practically the same things manually in single-user and then started up, and that worked.

I dropped the effort for now, because the machine for which I was actually interested, is still with Rel. 11.4 , which has to be changed first, anyway. (i did just a test on another machine)
Also I found a different approach for my problem: the l2arc filling too slowly. I'm not so much interested for performance matters, but when unused disks should stay in standby (rainforests, you know ), some of my metadata must be completely in l2arc, and that tends to take a few days with default settings. Thoroughly upping l2arc_headroom solves that for now.

 

[PMc]

As OpenZFS 2.0 has now been released some people will be interested in getting it up and running to play around with in FreeBSD 12.2 instead of waiting for it being included in FreeBSD 13 by default.

What would be required to get this working? The old ZFS system needs to be disabled and replaced by the OpenZFS kernel mod and the OpenZFS userland tools. Both these ports are slightly outdated at the moment (going by the date it would probably correspond to rc4) but that is presumably only a matter of time.

I understand that the kernel can be recompiled without the legacy ZFS implementation by putting WITHOUT_ZFS in /etc/src.conf. Is that correct?

I don't think this is precise. The kernel will by default compile without ZFS (and load the zfs module from base at boot time if required from loader.conf); only if you have a custom kernel with ZFS included, that must be removed.

The userland tools should not get installed when compiling world WITHOUT_ZFS. But that means: /usr/local cannot be on ZFS, because you need /usr/local/sbin/zpool first in order to import at all.
I did import the openzfs pool with the base-ZFS zpool command - but that is probably not "the right thing". (In my case, all buildworld is on UFS, everything else on ZFS.)

In /boot/loader.conf we need to remove zfs_load=“YES” and replace it with openzfs_load="YES". Is that correct?

In /etc/rc.conf we need to remove zfs_enable=“YES” and replace it with openzfs_enable="YES”. We also need to add kld_list="openzfs". Is that correct?

Yes, but see above: the rc.d openzfs script needs to invoke the zpool command in /usr/local - which may be mounted later for various reasons.

Upgrading the pool is obviously an irreversible process so afterwards you can’t take the pool back to stock FreeBSD 12.2 or earlier. Is there some special stuff required to import and/or upgrade the old pool? Can you import but not upgrade the pool and expect it to work (without the new features obviously)?

Without the new features, and with annoying messages from zpool status, that should work. Persistent l2arc should also work even without the new features.

What is the deal with booting from an OpenZFS volume? The OpenZFS FAQ suggests that booting not available until FreeBSD 13. Is that correct or does the FAQ just say that because it doesn’t want to complicate things by talking about the ports?

The deal is basically that you need to compile the openzfs code into the bootcode files: zfsboot(8) gptzfsboot(8).
Then the user needs to properly install these new bootcode files into their partition header, MBR, or whatever they use to boot.

 

[Morris]

I don't think this is precise. The kernel will by default compile without ZFS (and load the zfs module from base at boot time if required from loader.conf); only if you have a custom kernel with ZFS included, that must be removed.

The userland tools should not get installed when compiling world WITHOUT_ZFS. But that means: /usr/local cannot be on ZFS, because you need /usr/local/sbin/zpool first in order to import at all.
I did import the openzfs pool with the base-ZFS zpool command - but that is probably not "the right thing". (In my case, all buildworld is on UFS, everything else on ZFS.)

Yes, but see above: the rc.d openzfs script needs to invoke the zpool command in /usr/local - which may be mounted later for various reasons.

OK, so would it actually make more sense to compile the OpenZFS ports into the 12.2 kernel?

The deal is basically that you need to compile the openzfs code into the bootcode files: zfsboot(8) gptzfsboot(8).
Then the user needs to properly install these new bootcode files into their partition header, MBR, or whatever they use to boot.

Of course, I hadn't thought of that. Never mind then, too much hassle for just a couple of months, considering FreeBSD 13 is scheduled for March 2021. I might run some experiments with the CURRENT version of 13 early February.

 

[PMc]

OK, so would it actually make more sense to compile the OpenZFS ports into the 12.2 kernel?

No, this is about the userland tools and lib. For the kernel it is just as good if loading the module from loader.conf, or compiling it into the kernel - but then to import and mount, we need the commands.
And then all depends on the individual filesystem layout. A port does put them into /usr/local, and this may or may not be an individual filesystem, and may or may not be within ZFS. If it is, there is a hen+egg problem.

Ideally, all commands that are necessary to mount the basic filesystems, should reside in /bin. It wouldn't be difficult to copy the new zpool into /bin, but it probably needs some shared libs also, and that's when I considered it not worth to figure that out, as I don't want to make my system an unintellegible haywire - even less as there is soon the option to go for R.13.

For somebody who has only the application disks in ZFS, and the OS (including /usr/local) on a system disk (in UFS), there should be no problem here. In my case, I made the cut between UFS and ZFS right above the base installation (because I don't need to encrypt the base OS, but need to encrypt /usr/local/etc), so I'm out of luck here.

Maybe someone has a good idea how to approach this for various use-cases, I currently don't.

 

[Morris]

I just tried opensolaris_load="YES" in loader.conf, i.e. opensolaris.ko with 12.2-RELEASE-p2.
The bhyve guest on zfs is working. I think there are bugfixes, I'm switching to openzfs.

That's great to hear. Let us know how you get on.

I'm hoping to find some time soon to free up a machine and start testing OpenZFS too.

 

[Argentum]

That's great to hear. Let us know how you get on.

I'm hoping to find some time soon to free up a machine and start testing OpenZFS too.

I have some Bhyve VM running with OpenZFS. See my post.

Today I have also one physical machine with FreeBSD 12.2 and OpenZFS, all good, boots from OpenZFS pool and is running as expected. With persistent L2ARC ?

 

[Argentum]

I have some Bhyve VM running with OpenZFS. See my post.

Today I have also one physical machine with FreeBSD 12.2 and OpenZFS, all good, boots from OpenZFS pool and is running as expected. With persistent L2ARC ?

Now I must quote myself and add what I have found out during few days of playing around with OpenZFS:

1. It works fine until you perform zpool upgrade mypool;
2. It works fine even after pool upgrade when you do not add or remove devices;
3. If you play around with the pool configuration (add SLOG for example), the pool becomes unbootable;
4. All the data is preserved and pool itself is OK, but the FreeBSD boot loader does not recognize it any more. This was also mentioned elsewhere and is true. Be careful when upgrading your production system!
5. Upgraded pool can be imported to original ZFS read only. zpool import -o readonly=on mypool

 

[Argentum]

Upgrading the pool is obviously an irreversible process so afterwards you can’t take the pool back to stock FreeBSD 12.2 or earlier. Is there some special stuff required to import and/or upgrade the old pool? Can you import but not upgrade the pool and expect it to work (without the new features obviously)?

Now, when I have experimented few more days with OpenZFS I can share the following:

1. After upgrading the pool can be imported as read-only to the base ZFS. But it is perfectly OK. The new features affect only writing;
2. This is how I brought back my upgraded and unbootable pool -- ran zfs snapshot -r source_pool@replica in OpenZFS system, connected the drive to base ZFS system, imported the pool R/O and zfs send -R source_pool@replica. All good . So, the base ZFS can read OpenZFS without problems;
3. Upgraded pools become unbootable after some time. Not instantly, but seems that after some pool configuration changes (removed a device). Investigated the boot problem a little, but it is still unclear for me. When the pool is readable, it should also be bootable. To do that, I built and installed a boot1 loader with debug printouts. The boot protsess ends in
vdev_init function (/usr/src/stand/libsa/zfs/zfsimpl.c). There is a comparison in that file:

if (strcmp(type, VDEV_TYPE_MIRROR)
            && strcmp(type, VDEV_TYPE_DISK)
#ifdef ZFS_TEST
            && strcmp(type, VDEV_TYPE_FILE)
#endif
            && strcmp(type, VDEV_TYPE_RAIDZ)
            && strcmp(type, VDEV_TYPE_INDIRECT)
            && strcmp(type, VDEV_TYPE_REPLACING)) {
                printf("ZFS: DEBUG: can only boot from disk, mirror, raidz1, raidz2 and raidz3 vdevs\n");
                return (EIO);
        }

Printed out the type (which is string here) and it appears to be "hole". printf("ZFS_DEBUG: VDEV type: %s\n ", type);

VDEV_TYPE_HOLE also exists in include files, but is not too much documented. Then added an additional comparison to this command && strcmp(type, VDEV_TYPE_HOLE) to let the boot go on and it reached the next boot stage. This is where I am today.