bectl & upgrade with full rollback ability

Selin · Mar 26, 2021

Hi

I'd like to figure out how to do an upgrade with the ability to rollback fully if anything goes wrong.
Taking into account the info from the following topics:

Initial system state

Code:

> uname -a
FreeBSD test 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64

> freebsd-version -kru
12.1-RELEASE
12.1-RELEASE
12.1-RELEASE

> pkg info
dialog4ports-0.1.6             Console Interface to configure ports
diffutils-3.7                  GNU differential compare utilities
gettext-runtime-0.20.1         GNU gettext runtime libraries and programs
gettext-tools-0.20.1_1         GNU gettext development and translation tools
glib-2.56.3_6,1                Some useful routines of C programming (current stable version)
gmake-4.2.1_3                  GNU version of 'make' utility
indexinfo-0.3.1                Utility to regenerate the GNU info page index
libffi-3.2.1_3                 Foreign Function Interface
libiconv-1.14_11               Character set conversion library
libsigsegv-2.12                Handling page faults in user mode
libtextstyle-0.20.1            Text styling library
mc-4.8.22_1                    Midnight Commander, a free Norton Commander Clone
pcre-8.43_2                    Perl Compatible Regular Expressions library
perl5-5.30.0                   Practical Extraction and Report Language
pkg-1.12.0                     Package manager
pkgconf-1.6.3,1                Utility to help to configure compiler and linker flags
python27-2.7.16_1              Interpreted object-oriented programming language
readline-8.0.0                 Library for editing command lines as they are typed

I do the following:

zfs snap -r zroot@original
bectl create -r new
bectl mount new
chroot {path to mounted "new" snapshot}
mount -t devfs devfs /dev
rm -rf /var/db/freebsd-update
mkdir /var/db/freebsd-update
freebsd-update upgrade -r {NEW_VERSION}
freebsd-update install
bectl activate new
reboot

What I have at this point

Code:

> uname -a
FreeBSD test 13.0-RC3 FreeBSD 13.0-RC3 #0 releng/13.0-n244696-8f731a397ad: Fri Mar 19 04:00:20 UTC 2021     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64

> freebsd-version -kru
13.0-RC3
13.0-RC3
12.1-RELEASE

> pkg info
dialog4ports-0.1.6             Console Interface to configure ports
diffutils-3.7                  GNU differential compare utilities
gettext-runtime-0.20.1         GNU gettext runtime libraries and programs
gettext-tools-0.20.1_1         GNU gettext development and translation tools
glib-2.56.3_6,1                Some useful routines of C programming (current stable version)
gmake-4.2.1_3                  GNU version of 'make' utility
indexinfo-0.3.1                Utility to regenerate the GNU info page index
libffi-3.2.1_3                 Foreign Function Interface
libiconv-1.14_11               Character set conversion library
libsigsegv-2.12                Handling page faults in user mode
libtextstyle-0.20.1            Text styling library
mc-4.8.22_1                    Midnight Commander, a free Norton Commander Clone
pcre-8.43_2                    Perl Compatible Regular Expressions library
perl5-5.30.0                   Practical Extraction and Report Language
pkg-1.12.0                     Package manager
pkgconf-1.6.3,1                Utility to help to configure compiler and linker flags
python27-2.7.16_1              Interpreted object-oriented programming language
readline-8.0.0                 Library for editing command lines as they are typed

freebsd-update install
rebuild/update all ports/packages installed
freebsd-update install
gpart bootcode -p /boot/boot1.efifat -i 1 nvd0
reboot

Final System state

Code:

> uname -a
FreeBSD test 13.0-RC3 FreeBSD 13.0-RC3 #0 releng/13.0-n244696-8f731a397ad: Fri Mar 19 04:00:20 UTC 2021     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64

> freebsd-version -kru
13.0-RC3
13.0-RC3
13.0-RC3

> pkg info
dialog4ports-0.1.6             Console Interface to configure ports
diffutils-3.7                  GNU differential compare utilities
gettext-runtime-0.21           GNU gettext runtime libraries and programs
gettext-tools-0.21             GNU gettext development and translation tools
glib-2.66.7_1,1                Some useful routines of C programming (current stable version)
gmake-4.3_2                    GNU version of 'make' utility
indexinfo-0.3.1                Utility to regenerate the GNU info page index
libffi-3.3_1                   Foreign Function Interface
libiconv-1.16                  Character set conversion library
libsigsegv-2.12                Handling page faults in user mode
libtextstyle-0.21              Text styling library
mc-nox11-4.8.26                Midnight Commander, a free Norton Commander Clone
meson-0.57.1                   High performance build system
ninja-1.10.2,2                 Small build system closest in spirit to Make
pcre-8.44                      Perl Compatible Regular Expressions library
perl5-5.32.1_1                 Practical Extraction and Report Language
pkg-1.16.3                     Package manager
pkgconf-1.7.4,1                Utility to help to configure compiler and linker flags
py37-setuptools-44.0.0         Python packages installer
python37-3.7.10                Interpreted object-oriented programming language
readline-8.1.0                 Library for editing command lines as they are typed

Here I have my system upgraded and what to rollback to the initial state (including kernel and versions of all installed ports).
So I'm going to do the following:

zfs snap -r zroot@new
bectl activate default
reboot # here I'm booting from LiveCD
import and mount my zpool
cd {mount dir}
mkdir backup
mv bin boot etc lib libexec rescue sbin usr var backup/
cp -pR {mount dir}/.zfs/snapshot/original/. {mount dir}
gpart bootcode -p {mount dir}/boot/boot1.efifat -i 1 nvd0

Rollback result

Code:

> uname -a
FreeBSD test 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64

> freebsd-version -kru
12.1-RELEASE
12.1-RELEASE
12.1-RELEASE

> pkg info
dialog4ports-0.1.6             Console Interface to configure ports
diffutils-3.7                  GNU differential compare utilities
gettext-runtime-0.20.1         GNU gettext runtime libraries and programs
gettext-tools-0.20.1_1         GNU gettext development and translation tools
glib-2.56.3_6,1                Some useful routines of C programming (current stable version)
gmake-4.2.1_3                  GNU version of 'make' utility
indexinfo-0.3.1                Utility to regenerate the GNU info page index
libffi-3.2.1_3                 Foreign Function Interface
libiconv-1.14_11               Character set conversion library
libsigsegv-2.12                Handling page faults in user mode
libtextstyle-0.20.1            Text styling library
mc-4.8.22_1                    Midnight Commander, a free Norton Commander Clone
pcre-8.43_2                    Perl Compatible Regular Expressions library
perl5-5.30.0                   Practical Extraction and Report Language
pkg-1.12.0                     Package manager
pkgconf-1.6.3,1                Utility to help to configure compiler and linker flags
python27-2.7.16_1              Interpreted object-oriented programming language
readline-8.0.0                 Library for editing command lines as they are typed

Now it seems the system is the same as I had before the upgrade.
Is that correct?
Or I should adjust them somehow?

If some user files were updated in a new system - they can be copied from the /.zfs/snapshot/new/usr/home/{user}

And what about this part of the upgrade/rollback?
gpart bootcode -p /boot/boot1.efifat -i 1 nvd0
Is that correct to update/rollback UEFI loader that way?

Thanks

zirias@ · Mar 26, 2021

I only had a quick look now, but doing a global snapshot of zroot looks a bit over the top to me. Boot environments already handle the base system. If you also upgrade ports/packages and /usr/local is on a different dataset (which isn't the case in the structure created by the installer), just add a snapshot there as well.

As for things like /var and /usr/home or /home, you should probably think twice whether you want to roll them back at all. Normally, a boot environment will be enough. As long as you don't upgrade your pool (you shouldn't unless you're SURE you don't want to go back), bootcode shouldn't be an issue either, but if you already updated it, just rewrite it from your other BE to go back.

SirDice · Mar 26, 2021

Note that with 13.0 that boot1.efifat file doesn't exist anymore. You need to mount the efi partition with msdosfs(5) and copy /boot/loader.efi to EFI/BOOT/bootx64.efi (FAT is not case-sensitive, so actual case doesn't matter) of that filesystem.

There shouldn't be a need to rollback the EFI loader, it's backwards compatible. You can boot 12.x with the 13.0 loader.efi but not the other way around.

Selin · Mar 26, 2021

I need to rollback the whole system state: kernel + set of ports installed + all their configs, etc.
Sometimes it might be needed to rollback some application config and data. E.g. DB files might be converted to a new format after DB engine upgrade.
So I decided to do a full backup.

Here is the ZFS structure I have with default installation:

Code:

> zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
zroot               6.24G  6.84G    88K  /zroot
zroot/ROOT          4.00G  6.84G    88K  none
zroot/ROOT/default  2.47G  6.84G  1.89G  /
zroot/tmp            336K  6.84G   120K  /tmp
zroot/usr           2.22G  6.84G    88K  /usr
zroot/usr/home       268K  6.84G   128K  /usr/home
zroot/usr/ports     1.53G  6.84G   880M  /usr/ports
zroot/usr/src        704M  6.84G   704M  /usr/src
zroot/var            932K  6.84G    88K  /var
zroot/var/audit       88K  6.84G    88K  /var/audit
zroot/var/crash       88K  6.84G    88K  /var/crash
zroot/var/log        420K  6.84G   156K  /var/log
zroot/var/mail        88K  6.84G    88K  /var/mail
zroot/var/tmp        160K  6.84G    88K  /var/tmp

bectl -r ... creates snapshot only for the zroot/ROOT/default volume.

What about zroot/usr - isn't it affected during upgrading the system and ports?

zirias@ · Mar 26, 2021

I wonder how your /usr ended up containing anything, it should normally be set to canmount=no, so everything in /usr (except for the specific datasets you create below) will be stored on your root dataset.

edit: forget the nonsense, looked at the wrong column. Your zroot/usr seems to be empty, as expected.

SirDice · Mar 26, 2021

Selin said:
What about zroot/usr - isn't it affected during upgrading the system and ports?

It's not mounted, it's only there to provide easy access to for /usr/home, /usr/ports and /usr/src. The actual data in /usr/ is stored within zroot/ROOT/default.

Code:

root@molly:~ # zfs get canmount zroot/usr
NAME       PROPERTY  VALUE     SOURCE
zroot/usr  canmount  off       local

/usr/ports doesn't need to be rolled back (or forward), all versions use the same ports tree. /usr/src/ doesn't really matter either, since it's version controlled you can more easily deal with that using svnlite(1) or git(1) (just switch branches).

Selin · Mar 29, 2021

Thanks everybody for advices.

Updated my notes to the following:

Upgrade

bectl create -r {new}
bectl mount {new}
chroot {path to mounted "new" snapshot}
mount -t devfs devfs /dev
rm -rf /var/db/freebsd-update
mkdir /var/db/freebsd-update
freebsd-update upgrade -r {NEW_VERSION}
freebsd-update install
bectl activate {new}
reboot
freebsd-update install
rebuild/update all ports/packages installed
freebsd-update install
gpart bootcode -p /boot/boot1.efifat -i 1 nvd0
reboot
check if everything works good because after the next step there will be no ability to rollback
zpool upgrade -a

Rollback

bectl activate {old}
chroot {path to mounted "old" snapshot}
mount -t devfs devfs /dev
gpart bootcode -p /boot/boot1.efifat -i 1 nvd0
reboot

NOTE
In 13.0 the boot1.efifat file doesn't exist anymore.
So instead of gpart bootcode -p /boot/boot1.efifat -i 1 nvd0 need to mount the efi partition with msdosfs(5) and copy /boot/loader.efi to EFI/BOOT/bootx64.efi

mtu · Mar 29, 2021

Selin said:
Upgrade

bectl create -r {new}

bectl mount {new}

chroot {path to mounted "new" snapshot}

mount -t devfs devfs /dev

rm -rf /var/db/freebsd-update

mkdir /var/db/freebsd-update

freebsd-update upgrade -r {NEW_VERSION}

freebsd-update install

bectl activate {new}

reboot

freebsd-update install

rebuild/update all ports/packages installed

freebsd-update install

gpart bootcode -p /boot/boot1.efifat -i 1 nvd0

reboot

check if everything works good because after the next step there will be no ability to rollback

zpool upgrade -a

Note that you can take as much time as you want for the last step. You might miss out on some cool new features, but otherwise there's no penalty for leaving your zpool at an older version.

Mjölnir · Mar 29, 2021

bectl create previous
Perform freebsd-update(8) (on default boot environment)... Of course, the chroot(8) method's advantage is reduced interrupt of service
EDIT or use freebsd-update -b <mountpoint> /EDIT
reboot(8), run into issues, decide to rollback

"Rollback":

bectl activate previous
reboot
bectl destroy default
bectl rename previous default

Conclusio: freebsd-update rollback is only for UFS, and then only if you don't have 2 root filesystem partitions to switch between (e.g. IIRC nanobsd(8) does that by default).

zirias@ · Mar 29, 2021

FWIW, if you install from source, you can fully install a new release without the need for chroot:

Code:

bectl create 13rc4
bectl mount 13rc4
cd /usr/src
make DESTDIR=/tmp/be-XXXX installkernel
make DESTDIR=/tmp/be-XXXX installworld
make DESTDIR=/tmp/be-XXXX BATCH_DELETE_OLD_FILES=yes delete-old
bectl umount 13rc4
bectl activate 13rc4

But then, having to use chroot isn't that bad either. When upgrading to a new release, I'd always prepare it in the new BE, not in the currently running one, because otherwise it isn't safe to install the userland without booting with the new kernel first.

Edit: Some support for a destination dir in freebsd-update(8) would be nice!

Mjölnir · Mar 29, 2021

From RTFM freebsd-update(8)

Code:

OPTIONS
     The following options are supported:

     -b basedir     Operate on a system mounted at basedir.  (default: /, or
                    as given in the configuration file.)

Unfortunately, that option is not mentioned in freebsd-update.conf(5)

zirias@ · Mar 29, 2021

Ha! I didn't even look, cause everyone seems to describe the chroot method. Looks like it isn't necessary after all.

Cath O'Deray · Apr 20, 2021

The chroot method is appealing for its ability to streamline things, however it's too easy for things to go wrong if the end user follows on-screen instructions.

freezr · Nov 8, 2021

Mjölnir said:
bectl create previous
Perform freebsd-update(8) (on default boot environment)... Of course, the chroot(8) method's advantage is reduced interrupt of service
EDIT or use freebsd-update -b <mountpoint> /EDIT
reboot(8), run into issues, decide to rollback

"Rollback":

bectl activate previous
reboot
bectl destroy default
bectl rename previous default

Conclusio: freebsd-update rollback is only for UFS, and then only if you don't have 2 root filesystem partitions to switch between (e.g. IIRC nanobsd(8) does that by default).

Thank you very much, this was revealing!

Just a quick question on another "tutorials" I read to use -r as in:

bectl create -r previous

The -r option is the recursive option, it is needed to make sure we get all the relevant datasets.
source: https://unixsheikh.com/tutorials/migrating-zfs-from-linux-to-freebsd.html

Is it important this detail in your opinion?

Cath O'Deray · Nov 9, 2021

I don't fully understand the pros and cons of recursion, in this context, but I do recall a suggestion that it might become a default.

I'm checking in a chat room.

Selin · Nov 9, 2021

tgl said:
Is it important this detail in your opinion?

I think this might be useful when different ZFS volumes are used for the system files - to have snapshots of all these volumes.

bectl & upgrade with full rollback ability

Selin

zirias@

SirDice

Administrator

Selin

zirias@

SirDice

Administrator

Selin

mtu

Mjölnir

zirias@

Mjölnir

zirias@

Cath O'Deray

freezr

Cath O'Deray

Selin