Solved Unable to delete BE

jbo@ · Jan 2, 2023

So, I just wanted to fix up my "playground" host. Back then, I used boot environments "incorrectly". i.e. what I did whenever I performed an upgrade or some test I wanted to be able to roll-back from is this:

Create new BE
Activate new BE
Reboot
Do critical stuff
Then either roll back to previous BE or just leave as is

However, since then I came to learn that this is a much more sane approach:

Create new BE
Do critical stuff
Then either roll back to previous BE or just leave as is

With the second approach, one just always uses the default BE (unless something went wrong).
Given that I didn't do it this way back then on this particular host my current situations look like this:

Code:

jbo@knox:~ % bectl list
BE                             Active Mountpoint Space Created
13.1-RELEASE_2023-01-02_150042 -      -          3.11M 2023-01-02 15:00
20220525                       -      -          11.3M 2022-05-25 13:19
20230102                       NR     /          10.3G 2023-01-02 14:55
default                        -      -          29.4G 2020-07-21 16:09

Now I'd like to get back to the approach where the host usually remains on the default BE. My approach would be to delete the currently existing default BE, then creating a new default BE from the currently active BE ( 20230102).
However, deleting the default didn't work:

Code:

jbo@knox:~ % sudo bectl destroy -o default
cannot destroy 'zroot/ROOT/default/git': dataset is busy
unknown error

I use sysutils/cbsd to manage jails (and VMs). git is a jail hosting a git server. This is the point where I started to get concerned. To me, it looks like my system is running on the 20230102 BE but my jails are all mounting form zroot/ROOT/default:

Code:

jbo@knox:~ % zfs list
NAME                                        USED  AVAIL     REFER  MOUNTPOINT
storage                                    10.6T  6.80T      128K  /storage
storage/emby                               10.4T  6.80T     10.4T  /storage/emby
storage/nextcloud                           132G  6.80T      132G  /storage/nextcloud
zroot                                      39.6G   183G       88K  /zroot
zroot/ROOT                                 39.5G   183G       88K  none
zroot/ROOT/13.1-RELEASE_2023-01-02_150042     8K   183G     7.21G  /
zroot/ROOT/20220525                         884K   183G     7.10G  /
zroot/ROOT/20230102                        10.3G   183G     7.19G  /
zroot/ROOT/default                         29.2G   183G     5.52G  /
zroot/ROOT/default/blog                     473M   183G      219M  /usr/jails/jails-data/blog-data
zroot/ROOT/default/cppproperties            199M   183G      114M  /usr/jails/jails-data/cppproperties-data
zroot/ROOT/default/emby                    16.0G   183G     5.20G  /usr/jails/jails-data/emby-data
zroot/ROOT/default/git                     2.34G   183G     2.34G  /usr/jails/jails-data/git-data
zroot/ROOT/default/irc                      965M   183G      574M  /usr/jails/jails-data/irc-data
zroot/ROOT/default/nextcloud               4.76G   183G     1.93G  /usr/jails/jails-data/nextcloud-data
zroot/tmp                                   884K   183G      488K  /tmp
zroot/usr                                   840K   183G       88K  /usr
zroot/usr/home                              464K   183G      220K  /usr/home
zroot/usr/ports                             144K   183G       88K  /usr/ports
zroot/usr/src                               144K   183G       88K  /usr/src
zroot/var                                  7.09M   183G       88K  /var
zroot/var/audit                             144K   183G       88K  /var/audit
zroot/var/crash                             144K   183G       88K  /var/crash
zroot/var/log                              4.08M   183G     1.36M  /var/log
zroot/var/mail                             2.44M   183G     2.24M  /var/mail
zroot/var/tmp                               208K   183G       88K  /var/tmp

As of my current knowledge, there are basically two possible scenarios:

bectl creates an alias of the currently active BE to zroot/ROOT/default no matter the name of the currently active BE.
I hosed my system: The base system is running on zroot/ROOT/20230102 but my jails are all mounting zroot/ROOT/default

What exactly is going on here? Am I missing something obvious? Any ideas?

Alain De Vos · Jan 2, 2023

You have 4 different boot environments. You can also destroy them with "zfs destroy",

Code:

zroot/ROOT/13.1-RELEASE_2023-01-02_150042     8K   183G     7.21G  /
zroot/ROOT/20220525                         884K   183G     7.10G  /
zroot/ROOT/20230102                        10.3G   183G     7.19G  /
zroot/ROOT/default                         29.2G   183G     5.52G  /

As for the jails.
You must decide/choose if you want them to be part of the boot environment yes-or-no.
Off-course you must first stop the jails before manipulating them.

mer · Jan 2, 2023

jbo said:
bectl creates an alias of the currently active BE to zroot/ROOT/default no matter the name of the currently active BE.

I hosed my system: The base system is running on zroot/ROOT/20230102 but my jails are all mounting zroot/ROOT/default

The first one does not happen. bectl create blah creates a new BE named blah that is a clone of the currently active BE.

I'm not a jail user, but your second: I would not consider the system "hosed" but agree that the jails are using zroot/ROOT/default. Your post says "dataset git" is busy, so something in the jails may be using that dataset. I would stop all the jails and then look at the configuration for all the jails looking specificially for references to git. It may be "hidden" look for processes in the jails that are using git repos in that zroot/ROOT/default/git dataset.

These entries "bother" me. They are datasets created off the zroot/ROOT/default. I think they were created in the wrong place; they likely prevent the zroot/ROOT/default BE from ever being deleted. I would have thought typical processes would have created an initial zroot/jails dataset (like zroot/usr) and then these underneath it.

zroot/ROOT/default/blog 473M 183G 219M /usr/jails/jails-data/blog-data
zroot/ROOT/default/cppproperties 199M 183G 114M /usr/jails/jails-data/cppproperties-data
zroot/ROOT/default/emby 16.0G 183G 5.20G /usr/jails/jails-data/emby-data
zroot/ROOT/default/git 2.34G 183G 2.34G /usr/jails/jails-data/git-data
zroot/ROOT/default/irc 965M 183G 574M /usr/jails/jails-data/irc-data
zroot/ROOT/default/nextcloud

Alain De Vos · Jan 2, 2023

I don't know cbsd, If,if you want the jails to be part of the boot-environment you should destroy the datasets,

Code:

zroot/ROOT/default/blog                     473M   183G      219M  /usr/jails/jails-data/blog-data
zroot/ROOT/default/cppproperties            199M   183G      114M  /usr/jails/jails-data/cppproperties-data
zroot/ROOT/default/emby                    16.0G   183G     5.20G  /usr/jails/jails-data/emby-data
zroot/ROOT/default/git                     2.34G   183G     2.34G  /usr/jails/jails-data/git-data
zroot/ROOT/default/irc                      965M   183G      574M  /usr/jails/jails-data/irc-data
zroot/ROOT/default/nextcloud               4.76G   183G     1.93G  /usr/jails/jails-data/nextcloud-data

Erichans · Jan 2, 2023

Alain De Vos said:
You have 4 different boot environments. You can also destroy them with "zfs destroy",

Code:

zroot/ROOT/13.1-RELEASE_2023-01-02_150042 8K 183G 7.21G / zroot/ROOT/20220525 884K 183G 7.10G / zroot/ROOT/20230102 10.3G 183G 7.19G / zroot/ROOT/default 29.2G 183G 5.52G /

I'd be carefull in trying to "zfs destroy" the underlying dataset of a BE, especially very carefull with zroot/ROOT/default as it is at the moment.

Generally, a dataset zroot/ROOT/default/xxx such as zroot/ROOT/default/blog cannot exist without its parent dataset: zroot/ROOT/default*. So, in reverse, if you were succesful in destroying the BE default or its underlying ZFS clone by means of zsf destroy or zsf destroy -R (or even -Forcing the issue) you might destroy more then expected.

jbo said:
[...] However, since then I came to learn that this is a much more sane approach:

Create new BE

Do critical stuff

Then either roll back to previous BE or just leave as is

With the second approach, one just always uses the default BE (unless something went wrong).
Given that I didn't do it this way back then on this particular host my current situations look like this:

Code:

jbo@knox:~ % bectl list BE Active Mountpoint Space Created 13.1-RELEASE_2023-01-02_150042 - - 3.11M 2023-01-02 15:00 20220525 - - 11.3M 2022-05-25 13:19 20230102 NR / 10.3G 2023-01-02 14:55 default - - 29.4G 2020-07-21 16:09

As BE "20230102" was created today very recently, do you know how that BE got to have the "NR" attributes; instead of (I think you intended) remain at the "default" BE)? Anything in the ZFS history perhaps?

jbo said:
Now I'd like to get back to the approach where the host usually remains on the default BE. My approach would be to delete the currently existing default BE, then creating a new default BE from the currently active BE ( 20230102).
However, deleting the default didn't work:

Code:

jbo@knox:~ % sudo bectl destroy -o default cannot destroy 'zroot/ROOT/default/git': dataset is busy unknown error

My guess is that the error is generated because the datasets zroot/ROOT/default/blog etc. are mounted (=busy) and are "tied" to the dataset zroot/ROOT/default.

As mer mentioned, I think the dataset structure with all the zroot/ROOT/default/xxx and its current mounts is ill-conceived.

As I haven't done any such thing before, my suggestion is something like the following; I'd consider a dry run somewhere else first. (I'm assuming you'd want all the jails-stuff outside of any BEs but that may not be the case). I don't think that just a creation of another new dataset, such as ~~zroot/ROOT/jail-stuff~~ zroot/jail-stuff, and then renaming all datasets such as zroot/ROOT/default/xxx would suffice. My suggestion is that you try creating a new dataset like ~~zroot/ROOT/jail-stuff~~ zroot/jail-stuff, then creating its subordinate datasets ~~zroot/ROOT/jail-stuff/blog~~ zroot/jail-stuff/blog etc. and then moving the contents of the orginal datasets to the new ones; this might cost you temporarily double the storage space of all those datasets you move. Then you should probably unmount the appropriate datasets (like zroot/ROOT/default/xxx etc. ); lastly see what bectl destroy does.

___
* (edit): The same principle applies to ROOT:

Rich (BB code):

jbo@knox:~ % zfs list
NAME                                        USED  AVAIL     REFER  MOUNTPOINT
<snap>
zroot                                      39.6G   183G       88K  /zroot
[bgcolor=yellow]zroot/ROOT[/bgcolor]                                 39.5G   183G       88K  none
<snap>
[bgcolor=yellow]zroot/ROOT/default[/bgcolor]                         29.2G   183G     5.52G  /

zroot/ROOT enables the existance of zroot/ROOT/default; the former is necessary but does not need to have a mountpoint.

Alain De Vos · Jan 2, 2023

What i would do is create a dataset:

Code:

zroot/myjails
zroot/myjails/blog                     473M   183G      219M  /usr/jails/jails-data/blog-data
zroot/myjails/cppproperties            199M   183G      114M  /usr/jails/jails-data/cppproperties-data
zroot/myjails/emby                    16.0G   183G     5.20G  /usr/jails/jails-data/emby-data
zroot/myjails/git                     2.34G   183G     2.34G  /usr/jails/jails-data/git-data
zroot/myjails/irc                      965M   183G      574M  /usr/jails/jails-data/irc-data
zroot/myjails/nextcloud               4.76G   183G     1.93G  /usr/jails/jails-data/nextcloud-data

Then this dataset is not part of boot-environments. So you can create&destroy boot-environments as you like.
Of-course you can take recursive snapshot of zroot/myjails.

jbo@ · Jan 12, 2023

Alright, everything is back in order.

As mentioned by mer, selecting/booting a particular BE does not 'remap' to `zroot/ROOT/default`. Instead, it's a simple matter of mounting the corresponding underlying ZFS clone.

As for the hosing the system: Also as correctly guesstimated by mer this was easily recoverable. Jails were now all moved outside of the "BE datasets".

Erichans said:
As mer mentioned, I think the dataset structure with all the zroot/ROOT/default/xxx and its current mounts is ill-conceived.

I agree.

Thanks guys!

mer · Jan 12, 2023

In the words of the immortal bards Monty Python "And there was much rejoicing!"

jbo thanks for updating. I like when "we" (everyone trying to help) get an answer.