bectl confusion

J

jbo@

Developer
Prior to "properly" understanding BEs (Boot Environments) I used the following workflow:
Code: 
bectl create 20220312
bectl activate 20220312
shutdown -r now
# perform actual update (eg. freebsd-update or update from source)
shutdown -r now
This basically means that I create a new BE, boot into it, perform modifications and then either stay in that newly created BE if things worked out or roll back to the previous BE.
However, I came to the understanding that a more reasonable workflow is to create a BE, NOT activating & booting into it but rather staying in the same BE (eg. default) and only rolling back to the newly created BE if necessary.

This means that I now have a machine where bectl list shows:
Code: 
jbo@knox:~ % bectl list
BE       Active Mountpoint Space Created
20220506 NR     /          7.68G 2022-05-06 14:08
default  -      -          29.6G 2020-07-21 16:09

I wanted to "migrate" to the "new" approach of using BEs as described above. For this, my planned workflow is:
  1. Remove BE default
  2. Rename the currently active BE to default
  3. Activate & boot into the renamed default BE
  4. Optionally remove other BEs
This way I'd be back to just simply sticking to the default BE and only booting into newly created BEs if something went wrong / a rollback is desired.

However, bectl destroy -o deafult shows:
Code: 
jbo@knox:~ % doas bectl destroy -o default
cannot destroy 'zroot/ROOT/default/jail4': dataset is busy
unknown error

zfs list shows:
Code: 
jbo@knox:~ % zfs list
NAME                               USED  AVAIL     REFER  MOUNTPOINT
nvme                              1.23M   225G       96K  /nvme
nvme/cbsd                           96K   225G       96K  /nvme/cbsd
storage                           10.7T  6.34T      128K  /storage
storage/emby                      10.6T  6.34T     10.6T  /storage/emby
storage/nextcloud                  114G  6.34T      114G  /storage/nextcloud
zroot                             37.2G   186G       88K  /zroot
zroot/ROOT                        37.1G   186G       88K  none
zroot/ROOT/20220506               7.68G   186G     5.66G  /
zroot/ROOT/default                29.5G   186G     5.52G  /
zroot/ROOT/default/jail1           462M   186G      209M  /usr/jails/jails-data/jail1-data
zroot/ROOT/default/jail2           199M   186G      114M  /usr/jails/jails-data/jail2-data
zroot/ROOT/default/jail3          16.3G   186G     5.45G  /usr/jails/jails-data/jail3-data
zroot/ROOT/default/jail4          2.30G   186G     2.30G  /usr/jails/jails-data/jail4-data
zroot/ROOT/default/jail5           965M   186G      574M  /usr/jails/jails-data/jail5-data
zroot/ROOT/default/jail6          1.01G   186G     1.01G  /usr/jails/jails-data/jail6-data
zroot/ROOT/default/jail7          3.45G   186G     2.61G  /usr/jails/jails-data/jail7-data
zroot/ROOT/default/jail8          4.84G   186G     2.00G  /usr/jails/jails-data/jail8-data
zroot/tmp                          852K   186G      456K  /tmp
zroot/usr                          872K   186G       88K  /usr
zroot/usr/home                     496K   186G      252K  /usr/home
zroot/usr/ports                    144K   186G       88K  /usr/ports
zroot/usr/src                      144K   186G       88K  /usr/src
zroot/var                         6.66M   186G       88K  /var
zroot/var/audit                    144K   186G       88K  /var/audit
zroot/var/crash                    144K   186G       88K  /var/crash
zroot/var/log                     4.13M   186G     1.42M  /var/log
zroot/var/mail                    1.96M   186G     1.76M  /var/mail
zroot/var/tmp                      208K   186G       88K  /var/tmp
Note: My jails are not named jail1 to jail8. I just replaced the actual jail names for the sake of this post.

So I'm a bit confused here.
Why would bectl list show that only BE 20220506 is active and mounted (BE default shows no mountpoint) yet I'm not able to delete it due to the dataset being busy.

How does one go about figuring out the actual problem (why the dataset is busy) and eventually resolving the problem?
 
Very interesting question, indeed. Because I wonder something similar every time I use BE as well. The way how I use it is to keep `default` as current environment and simply create new BE as point of recovery. That should create a new ZFS snapshot. On the other hand, sometimes I really do use this snapshot, simply reboot into it and delete the old "default" and rename the snapshot to default. Now that works well, but makes me always wonder if I'm not creating a mess. I think my main confusion comes from 2 things:
  • not fully understanding how snapshot works in ZFS
  • perhaps a bit of more documentation in the FreeBSD manual wouldn't hurt to clarify for newbies like me
To answer your question: I think you want to reboot to one BE and delete the other one, but since this error never happened to me, I can't say.

Another very interesting feature would to be use jails with "bect jail" command. The only problem I have is the lack of networking. I don't know how to run this command, so I get access to the network from the jail and I can actually perform the update in it. Once I will figure it out, I will be able to perform an update, test it, even install new packages in it and if all goes well, then I'll know I can simply do the update on the current BE and just create new BE as snapshot for cases I missed during jail testing.

Edit: I just noticed another strange thing - is it just me or you have your jails tied to the ROOT? Because my setup is different:

Code: 
NAME                               USED  AVAIL     REFER  MOUNTPOINT
zroot                              129G   316G      104K  /zroot
zroot/ROOT                        28.4G   316G       96K  none
zroot/ROOT/default                28.4G   316G     28.4G  /
zroot/pot                         1.63G   316G       96K  /zroot/pot
zroot/pot/bases                     96K   316G       96K  /zroot/pot/bases
zroot/pot/cache                    180M   316G      180M  /var/cache/pot
zroot/pot/fscomp                    96K   316G       96K  /zroot/pot/fscomp
zroot/pot/jails                   1.45G   316G       96K  /zroot/pot/jails
zroot/pot/jails/dns                592M   316G      108K  /zroot/pot/jails/dns
zroot/pot/jails/dns/m              592M   316G      592M  /zroot/pot/jails/dns/m
zroot/pot/jails/searx              896M   316G      116K  /zroot/pot/jails/searx
zroot/pot/jails/searx/m            895M   316G      895M  /zroot/pot/jails/searx/m
zroot/reserved                       5G   321G       96K  /zroot/reserved
zroot/tmp                          160K   316G      160K  /tmp
zroot/usr                         74.6G   316G       96K  /usr
zroot/usr/home                    71.9G   316G     71.9G  /usr/home
zroot/usr/ports                   1.92G   316G     1.92G  /usr/ports
zroot/usr/src                      764M   316G      764M  /usr/src
zroot/var                         2.91M   316G       96K  /var
zroot/var/audit                     96K   316G       96K  /var/audit
zroot/var/crash                     96K   316G       96K  /var/crash
zroot/var/log                     2.35M   316G     2.35M  /var/log
zroot/var/mail                     136K   316G      136K  /var/mail
zroot/var/tmp                      148K   316G      148K  /var/tmp
zroot/vm                          18.3G   316G     8.59G  /zroot/vm
zroot/vm/alpine                    163M   316G      163M  /zroot/vm/alpine
zroot/vm/openbsd                  1.06G   316G     1.06G  /zroot/vm/openbsd
zroot/vm/ubuntu                    136K   316G      136K  /zroot/vm/ubuntu
 
BEs are flexible and any approach could be the "most reasonable" for someone. But first, I don't really understand what you tried:
jbodenmann said:
However, I came to the understanding that a more reasonable workflow is to create a BE, NOT activating & booting into it but rather staying in the same BE (eg. default) and only rolling back to the newly created BE if necessary.
Click to expand...
With that, how could a different BE, not named default, be active, if you didn't use it for "rollback", which I now assume?

BTW, to me personally, the most reasonable approach is
  • Create a new BE
  • Install the upgrade there, just mounting it (and using appropriate flags/variables to freebsd-update or make install[world|kernel].
  • Temporarily activate the new BE and reboot for a smoke test.
  • If everything is fine, remove the old "default", rename the new one to "default" and activate it.
Edit: here's how I performed my upgrade to 13.1 from source:
https://twitter.com/x/status/1523253983900356609
View: https://twitter.com/8bitsound/status/1523253983900356609
 
Both variants work. I always used the workflow of creating a new BE; boot it in a jail and run freebsd-update in it; if all goes well activate & reboot to that BE and finish the update with one last freebsd-update (that last step usually doesn't finish completely in a jail because of some symlinks that are missing) and then upgrade packages and jails.

I've already used that workflow with sysutils/beadm and it always worked well and saved me several times.
 
algi said:
Edit: I just noticed another strange thing - is it just me or you have your jails tied to the ROOT? Because my setup is different:
Click to expand...
I'm using sysutils/cbsd for jails management so the layout is just different.

Erichans said:
What is the output of zfs get origin zroot/ROOT/default and zfs get origin zroot/ROOT/default/jail4 ?
Click to expand...
Here we go:
Code: 
jbo@knox:~ % zfs get origin zroot/ROOT/default
NAME                PROPERTY  VALUE                                      SOURCE
zroot/ROOT/default  origin    zroot/ROOT/20220506@2022-05-06-14:08:44-0  -


jbo@knox:~ % zfs get origin zroot/ROOT/default/jail4
NAME                    PROPERTY  VALUE   SOURCE
zroot/ROOT/default/jail4  origin    -       -
 
I don't know (not a big jail user).; my reason for asking was the original error message implied to me at least that jail4 is using default "right now". Similar to trying to unmount a filesystem when it's being used.
 
Your problem seems to resemble Dan Langille's one: snapshot ; ls ; destroy; dataset is busy – WTF?. Something seems to have "a hold" on zroot/ROOT/default/jail4; that might be an explicit hold as in zfs-hold(8)

This is probably a long shot but, do you happen to have snapshots of zroot/ROOT/default or of any of its descendants such as more specifically zroot/ROOT/default/jail4? zfs list -t snapshot -r zroot/ROOT/default
Pick one of the snapshots highest in the dependancy hierarchy. Then:
zfs holds -r zroot/ROOT/default/<optional extra path spec>/<specific snapshot>

If there is a hold on any snapshot of zroot/ROOT/default/jail4 then you can release that hold with zfs-release(8)
 
  • Like
Reactions: mer
jbodenmann said:
I wanted to "migrate" to the "new" approach of using BEs as described above. For this, my planned workflow is:
  1. Remove BE default
  2. Rename the currently active BE to default
  3. Activate & boot into the renamed default BE
  4. Optionally remove other BEs
This way I'd be back to just simply sticking to the default BE and only booting into newly created BEs if something went wrong / a rollback is desired.
Click to expand...
Is it actually necessary to have a BE named 'default' or has it only been adopted as a convention? Neither bectl nor beadm appear to rely on it being used though it does appear in a comment in beadm:
Code: 
curlew:/home/mike% strings /sbin/bectl | grep default
curlew:/home/mike% strings /usr/local/sbin/beadm | grep default
# use other prefix then the 'pool/ROOT/bename' default
curlew:/home/mike%
I've been using BEs since the introduction of sysutils/beadm and have recently started to use bectl but always used the approach of creating a new BE with an appropriate name for an upgrade and booting into that. So the name 'default' was only ever used for the original installation and was eventually deleted, probably about nine years ago.
 
rawthey To the best of my knowledge "default" is a convention for the initial BE when you do a fresh install. I've renamed it, deleted it, so I don't think it's special. Basically "your last sentence" matches my experience.
 
jbodenmann said:
yes
But the jail isn't supposed to use a snapshot/clone/dataset from the default BE anyway, right?
Click to expand...
Well this is the thing, I feel like it's actualy part of your snapshot. That's why I asked about the layout. I think you can verify it by creating new jail and new BE and simply delete the jail. Then reboot back to the old BE and you'll see if I'm right or wrong.
 
This is what I have been using. Works great. Just change the appropriate info and off you go. It works with bectl just the same.

https://vermaden.wordpress.com/2021/02/23/upgrade-freebsd-with-zfs-boot-environments/

Code: 
(host) # beadm create 13                        # create new '13' ZFS Boot Environment
       Created successfully
(host) # beadm mount 13 /var/tmp/BE-13          # mount new '13' BE somewhere
       Mounted successfully on '/var/tmp/BE-13'
(host) # chroot /var/tmp/BE-13                  # make chroot(8) into that place
  (BE) # mount -t devfs devfs /dev              # mount the devfs(8) in that BE
  (BE) # rm -rf /var/db/freebsd-update          # remove any old patches
  (BE) # mkdir /var/db/freebsd-update           # create fresh dir for patches
  (BE) # freebsd-update upgrade -r 13.0-BETA3   # fetch the patches needed for upgrade
  (BE) # freebsd-update install                 # install kernel and kernel modules
  (BE) # freebsd-update install                 # install userspace/binaries/libraries
  (BE) # pkg upgrade                            # upgrade all packages with pkg(8)
  (BE) # freebsd-update install                 # remove old libraries and files
  (BE) # exit                                   # leave chroot(8) environment
(host) # umount /var/tmp/BE-13/dev              # umount the devfs(8) in that BE
(host) # beadm activate 13                      # activate new '13' BE
       Activated successfully

vermaden
 
jbodenmann said:
… How does one go about figuring out the actual problem (why the dataset is busy) and eventually resolving the problem?
Click to expand...

bectl list -c creation

bectl list -s -c creation

If you can make a separate post for the output from each command, it'll be easier to compare. Thanks.
 
This just happened to me as well, now I understand that my idea of "help" was totally misplaced. Here's what I did:

doas bectl jail -o host=inherit -o ip4=inherit pokus

Then inside I just run pkg upgrade. After I hit Ctrl-D, I got the error message about busy mountpoint too. It didn't happen on 13.0, so I wonder if it's not a bug?

Also:

Code: 
marian@beastie ~ % bectl list -c creation
BE      Active Mountpoint         Space Created
default NR     /                  28.0G 2021-12-15 13:48
pokus   -      /tmp/be_mount.yBoq 32.9M 2022-05-29 09:12
marian@beastie ~ % bectl list -s -c creation
BE/Dataset/Snapshot                          Active Mountpoint         Space Created

default
  zroot/ROOT/default                         NR     /                  28.0G 2021-12-15 13:48
  default@2022-05-29-09:12:32-0              -      -                  156K  2022-05-29 09:12

pokus
  zroot/ROOT/pokus                           -      /tmp/be_mount.yBoq 32.7M 2022-05-29 09:12
    zroot/ROOT/default@2022-05-29-09:12:32-0 -      -                  156K  2022-05-29 09:12
marian@beastie ~ % doas bectl destroy pokus
cannot destroy mounted boot env unless forced

Edit: After reboot, I can destroy BE as usual.
 
IIRC bectl also mounts devfs, but bectl umount only tries to unmount the actual BE. check with 'mount' what is mounted at the jail path.

I usually only mount the BE with bectl/beadm, then start the jail manually with jail so I can set several options for the jail.
 
Here you get delivered what is in the title.

A single foolproove method of operation still is a desirable design of usage.
 
sko said:
IIRC bectl also mounts devfs, …
Click to expand...

Within the mount point?

As far as I can tell, it does not:

Code: 
root@mowa219-gjp4-8570p-freebsd:~ # mount | grep devfs
devfs on /dev (devfs)
devfs on /compat/linux/dev (devfs)
devfs on /compat/ubuntu/dev (devfs)
root@mowa219-gjp4-8570p-freebsd:~ # bectl list -c creation
BE                        Active Mountpoint Space Created
n250511-5f73b3338ee-d     -      -          64.6G 2021-11-13 15:43
n252381-75d20a5e386-b     -      -          23.5G 2022-01-12 23:23
n252450-5efa7281a79-a     -      -          6.87G 2022-01-14 19:27
n252483-c8f8299a230-b     -      -          7.85G 2022-01-17 14:24
n252505-cc68614da82-a     -      -          5.24G 2022-01-18 14:26
n252531-0ce7909cd0b-h     -      -          10.1G 2022-02-06 12:24
n252997-b6724f7004c-c     -      -          8.10G 2022-02-11 23:07
n253116-39a36707bd3-e     -      -          7.58G 2022-02-20 07:03
n253343-9835900cb95-c     -      -          9.32G 2022-02-27 14:58
n253776-d5ad1713cc3-b     -      -          14.1G 2022-03-18 09:31
n253861-92e6b4712b5-e     -      -          14.5G 2022-04-02 16:02
n254268-50e244964e9-d     -      -          11.0G 2022-04-09 18:50
n254693-d7696096209-f     -      -          14.2G 2022-04-27 17:41
n255078-e140d551b78-h     -      -          7.09M 2022-05-11 21:18
n255588-01235012e5b-d     -      -          15.7G 2022-05-15 00:29
n255588-01235012e5b-b-c1d -      -          1.16G 2022-05-23 18:11
n255769-f16e38162c7-c     -      -          167M  2022-05-29 05:15
n255769-f16e38162c7-d     NR     /          18.1G 2022-05-30 03:46
root@mowa219-gjp4-8570p-freebsd:~ # bectl mount n255769-f16e38162c7-c /tmp/up
Successfully mounted n255769-f16e38162c7-c at /tmp/up
root@mowa219-gjp4-8570p-freebsd:~ # mount | grep devfs
devfs on /dev (devfs)
devfs on /compat/linux/dev (devfs)
devfs on /compat/ubuntu/dev (devfs)
root@mowa219-gjp4-8570p-freebsd:~ # bectl umount n255769-f16e38162c7-c
root@mowa219-gjp4-8570p-freebsd:~ #
 
You must log in or register to reply here.
Back
Top