Upgrade notes, using /usr/src from 12.2 -> 13.1

Rudy · May 2, 2022

I just completed upgrading a few boxes from 12.x to 13.1-RC4 and kept some running notes in a TXT file.
May it help you.

------------------- Get the souce --------------------
FIRST TIME:

   # git clone -b releng/13.1 --depth 1 https://git.freebsd.org/src.git /usr/src

UPDATE SRC:
# cd /usr/src && git pull

And build the sources... make the j be the number of cores you have.

   # cd /usr/src

  # make -j40 buildworld

  # make -j40 buildkernel KERNCONF=MONKEY13

I like to run the make in a 'screen' and then jump out and watch with 'htop'.
If you have 40 cores, '-j40' is a good number to pass to make on an idle machine.

-------------------- Create new environment --------------------
Create new boot environment (be)

    # bectl create 13.1-RC4

   # bectl mount 13.1-RC4

   # bectl list

Type 'df' and find the place it was mounted...
... and mount your usr/src below that new "be"

   # mount_nullfs /usr/src /tmp/be_mount.xYpS/usr/src

  # mount_nullfs usr/obj /tmp/be_mount.xYpS/usr/obj

I like to do it this way so I can run the 'make' from inside the new environment.
(Other admins set DESTDIR and install without chroot'ing.)

Now, 'chroot' into that "be" and add a devfs.

   # chroot /tmp/be_mount.xYpS

  chroot# mount -t devfs devfs /dev

Now you are ready to install your sources into the new "be".

-------------------- Install new bits --------------------
If you have IMPI to your box, now is a good time to login.

   chroot# cd /usr/src

  chroot# make installworld

  chroot# make installkernel KERNCONF=MONKEY13

  chroot# mergemaster  (see reply below about switching to etcupdate)

Historically, I did a "installkernel", reboot, and then a "installworld".
I did the world/kernel backwards here to test it out. I'm in a "be" so,
I can always wipe it and start over.

Check to see your kernel, 'exit' your chroot, and check again:

   chroot# ll /boot/kernel*/kernel

  -r-xr-xr-x  1 root  wheel  27510752 May 27  2021 /boot/kernel.old/kernel

  -r-xr-xr-x  1 root  wheel  21971408 Apr 27 19:52 /boot/kernel/kernel?

  # exit

  # ll /boot/kernel*/kernel

  -r-xr-xr-x  1 root  wheel  14206560 Feb 25  2019 /boot/kernel.old/kernel

  -r-xr-xr-x  1 root  wheel  27510752 May 27  2021 /boot/kernel/kernel

Neat, there they are. See the kernel moved from boot/kernel to /boot/kernel.old?

-------------------- Bootloader update --------------------
Updating the bootloader is important or you may see the "mountroot" prompt
when you boot into 13.x. Run 'gpart' to see what your disk layout looks like.

Code:

  # gpart show
  =>      40  1875384928  ada0  GPT  (894G)
          40        1024     1  freebsd-boot  (512K)
        1064         984        - free -  (492K)
        2048     4194304     2  freebsd-swap  (2.0G)
     4196352  1871187968     3  freebsd-zfs  (892G)

This box has a freebsd-boot not big enough for an efi image. You have two
options, resize the partition to fit the efi image to support UEFI booting,
or update your freebsd-boot partition and continue with Legacy BIOS boot.

My freebsd-boot is in ada0 slice 1, so I would run this code from the new "be".

   chroot# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

  partcode written to ada0p1

If you have a RAID with a second disk, check look for another freebsd-boot
partition to update (eg ada1 index 1).

   chroot# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

  partcode written to ada1p1

If your first partition is efi, 'gpart' will look like this:

Code:

  # gpart show
  =>      40  2000409184  da0  GPT  (954G)
          40      532480    1  efi  (260M)
      532520        2008       - free -  (1.0M)
      534528     4194304    2  freebsd-swap  (2.0G)
     4728832  1995679744    3  freebsd-zfs  (952G)

Make sure you are chrooting into your new "be" with the new "/boot/loader.efi"
This box is amd64 and not arm or something else, so I name the file BOOTx64.efi
Now is a good time to read 'man uefi'

   chroot# mount -t msdosfs /dev/ada0p1 /mnt

  chroot# mkdir -p /mnt/efi/boot/

  chroot# cp /boot/loader.efi /mnt/efi/boot/BOOTx64.efi

If you want to get really bold, you and convert from freebsd-boot (legacy) to
an efi (UEFI) boot, erase your parition 1 and partition 2 (must be your
swap),

    untested# swapinfo

   untested# swapoff -a

   untested# gpart destory -i 2

   untested# gpart destory -i 1

   untested# gpart add -t efi -s 6M? ad0 -i 1 -a 40? (need to calculate that offset)

   untested# gpart show (get offset/alignment for adding swap)

   untested# gpart add -b 4096 -t freebsd-swap -s 10G? da1 -a?

Then add the loader.efi to the new efi partition.

Here is a truncated output of the 'df -g' command:

Code:

-----------------------------------------------------
Filesystem          Size   Mounted on
-----------------------------------------------------
zroot/ROOT/13.1-RC4 762G   /tmp/be_mount.xYpS
zroot/usr/obj        97G   /usr/obj
zroot/usr/src        92G   /usr/src
/usr/src             92G   /tmp/be_mount.xYpS/usr/src
/usr/obj             97G   /tmp/be_mount.xYpS/usr/obj
-----------------------------------------------------

Everything intesting in my box is in a jail, so I run
'zfs set quota=100g zroot/usr' to keep it in check.

-------------------- Boot into the new kernel --------------------
If it went smoothly, set the next boot "be".

   # bectl activate 13.1-RC4

  # reboot

When it reboots, upgrade your ZFS pools:
# zpool upgrade -a

If a secondary bool does not show up after reboot, you may need to reimport it.

   # zpool import

  # zpool import POOL_NAME

Note, there is a new '/etc/rc.d/zpool' installed, so that should take care of
mounting all your pools. If you forgot mergemaster, you could get an issue with
secondary pools not automatically mounting upon reboot.

-------------------- Update your cluster --------------------
If you need to repeat on another machine, you can "send" your src and obj to
ANOTHER_HOST. After the build is complete, snapshot the src and obj, and then send.

   # zfs snapshot zroot/usr/src@13.1RC4

  # zfs snapshot zroot/usr/obj@13.1RC4

  # zfs send zroot/usr/src@13.1RC4 | pv | ssh ANOTHER_HOST "zfs receive zroot/usr/src"

  # zfs send zroot/usr/obj@13.1RC4 | pv | ssh ANOTHER_HOST "zfs receive zroot/usr/obj"

'pv' is a progress bar tool to pipe data through. Not neccessary, but fun!

SirDice · May 3, 2022

Rudy said:
Code:

chroot# mergemaster

Use etcupdate(8) instead. mergemaster(8) depends on VCS tags that don't exist anymore since we migrated to git. It works on 12.x because 12 is still being maintained with subversion (which adds the VCS tags to the files). From 13.0 onward everything is in git and the VCS tags don't exist anymore.

Cath O'Deray · May 4, 2022

Rudy said:
chroot# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

Please, is this consistent with the advice to stop using gpart(8) for updates to old EFI system partitions?

The same question, unanswered, here: <https://forums.freebsd.org/posts/565335>

… Type 'df' and find the place it was mounted …

Instead, you can specify the mount point at mount time.

No need to create the mount point beforehand. From <https://forums.freebsd.org/posts/565344>:

Code:

root@mowa219-gjp4-8570p-freebsd:~ # file /tmp/up
/tmp/up: cannot open `/tmp/up' (No such file or directory)
root@mowa219-gjp4-8570p-freebsd:~ # du -hs /var/cache/pkg
1.5M    /var/cache/pkg
root@mowa219-gjp4-8570p-freebsd:~ # bectl create n255078-e140d551b78-b
root@mowa219-gjp4-8570p-freebsd:~ # bectl mount n255078-e140d551b78-b /tmp/up
Successfully mounted n255078-e140d551b78-b at /tmp/up
root@mowa219-gjp4-8570p-freebsd:~ # …

Upgrade notes, using /usr/src …

Maybe out of scope, but don't forget to also upgrade packages … and so on.

pkg-upgrade(8)

– resist the temptation to use chroot(8) for this. As far as I can tell, doing so causes there to be no log of what was upgraded.

Last but not least, read both sets of release notes – before upgrading – and be aware that notes for 13.1-RC5 are incomplete:

{link removed}

{link removed}

Rudy · May 4, 2022

grahamperrin said:
Please, is this consistent with the advice to stop using gpart(8) for updates to old EFI system partitions?

I have some production boxes that do not have EFI partitions. I image other people only have the freebsd-boot partition as well. Not sure of any other way to upgrade other than "gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0". I'm hesitant to resizing my first partition to fit BOOTx64.efi. That step, I will put off until disks are swapped / box is replaced a few years down the line.

grahamperrin said:
Maybe out of scope, but don't forget to also upgrade packages … and so on.

Yep, good idea. I hold off on the pkg updates until I have successfully booted into the new kernel.

The next step is to plow through all the jails and upgrade the userland and packages there as well. That's a different set of notes.

Rudy · May 4, 2022

SirDice said:
Use etcupdate(8) instead.

Never used it, I just gave it a shot!

Code:

  D /etc/amd.map
  D /etc/newsyslog.conf.d/amd.conf
  M /etc/group
  M /etc/master.passwd
  M /etc/ntp.conf
  C /etc/rc.firewall
  M /etc/shells
  U /etc/ssh/sshd_config

etcupdate diff shows me the difference on ntp.conf, is there a way to use etcupdate to update that file, or do I need to manually edit ntp.conf to match what is in the repo?

Rudy · May 5, 2022

Ouch, etcupdate looks like it is not too smart. It wants to roll back my files to 2018 (12.0-RELEASE). Box is running 12.2-STABLE and in the new "be" it is 13.1-RC4.
Example:

 Resolving conflict in '/etc/network.subr':

Select: (p) postpone, (df) diff-full, (e) edit,

        (h) help for more options: df

--- /etc/network.subr    2021-01-31 02:39:24.154249000 -0800

+++ /var/db/etcupdate/conflicts/etc/network.subr    2022-05-04 10:31:30.408952000 -0700

@@ -22,7 +22,13 @@

 # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF

 # SUCH DAMAGE.

 #

+<<<<<<< yours

 # $FreeBSD: stable/12/libexec/rc/network.subr 362933 2020-07-04 15:16:48Z eugen $

+||||||| original

+# $FreeBSD: releng/12.0/libexec/rc/network.subr 339604 2018-10-22 17:27:36Z eugen $

+=======

+# $FreeBSD$

+>>>>>>> new

Upgrade notes, using /usr/src from 12.2 -> 13.1

Rudy

SirDice

Administrator

Cath O'Deray

Rudy

Rudy

Rudy