Solved UEFI boot anomaly after update to 12.2

Argentum · Nov 5, 2020

Hello!

Just recently I updated desktop machine from 12.1 to 12.2. Compiled wold and custom kernel, installed and everything looked good. Then updated the zpool version, which then advised me to update the boot partition. Agree, that updating the boot loader with command gpart bootcode -p ./boot1.efifat -i 1 ada0was my mistake. Unfortunately I did it on both mirror drives.

After that, you can guess, the system refused to boot. It showed the normal BSD boot menu, but after starting kernel, it hanged up. Looking the computer settings it did not show UEFI drives as before. Even stranger, when I tried to boot from USB stick, it actually booted up my HDD system. Everything was OK and stable, no errors.

I had saved the 12.1 source as a separate zfs dataset, so I tried to recompile the /usr/src/stand/ but the compile ended with errors. After that I recompiled the 12.2 /usr/src/stand/ which was OK. Then updated the boot partitions with /usr/obj/usr/src/amd64.amd64/stand/efi/boot1/boot1.efifat and the boot got better but not completely OK as it was before. Still it does not show my UEFI disks as bootable and a strange Windows boot appeared. Selecting that boots up my FreeBSD (writing this message on the very machine), but it is not stable and I am still worried.

What is wrong? Please help!

SirDice · Nov 5, 2020

Post the output from gpart(8) so we have a better idea of your partitions.

Argentum · Nov 5, 2020

SirDice said:
Post the output from gpart(8) so we have a better idea of your partitions.

Code:

=>        40  3907029088  ada0  GPT  (1.8T)
          40      409600     1  efi  (200M)
      409640     8388608     2  freebsd-swap  (4.0G)
     8798248  3889842272     3  freebsd-zfs  (1.8T)
  3898640520     8388608        - free -  (4.0G)

=>        40  3907029088  ada1  GPT  (1.8T)
          40      409600     1  efi  (200M)
      409640    16777216     2  freebsd-swap  (8.0G)
    17186856  3889842272     3  freebsd-zfs  (1.8T)

=>       40  250069600  ada2  GPT  (119G)
         40  134217728     3  freebsd-zfs  (64G)
  134217768   33554432     4  freebsd-zfs  (16G)
  167772200   33554432     2  freebsd-swap  (16G)
  201326632   48743008        - free -  (23G)

=>       40  250069600  ada3  GPT  (119G)
         40   33554432     2  freebsd-swap  (16G)
   33554472   33554432     4  freebsd-zfs  (16G)
   67108904  134217728     3  freebsd-zfs  (64G)
  201326632   48743008        - free -  (23G)

SirDice said:
Post the output from gpart(8) so we have a better idea of your partitions.

Also, I can mount the EFI partition

Code:

/dev/msdosfs/EFISYS on /media/EFISYS (msdosfs, local, nosuid)


# ls -l
total 385
-rwx------  1 ott  ott  393216 Apr 16  2018 BOOTx64.efi*
-rwx------  1 ott  ott      12 Apr 16  2018 startup.nsh*

Code:

# cat startup.nsh
BOOTx64.efi

# file BOOTx64.efi
BOOTx64.efi: PE32+ executable (EFI application) x86-64, for MS Windows

The main pool is also perfect:

Code:

# zpool status; zpool iostat -v
  pool: kelder
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:29:52 with 0 errors on Thu Nov  5 18:19:46 2020
config:

        NAME        STATE     READ WRITE CKSUM
        kelder      ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0
            ada1p3  ONLINE       0     0     0
        logs
          mirror-1  ONLINE       0     0     0
            ada2p4  ONLINE       0     0     0
            ada3p4  ONLINE       0     0     0
        cache
          ada2p3    ONLINE       0     0     0

errors: No known data errors
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
kelder       221G  1.59T  1.25K     21  51.1M   309K
  mirror     221G  1.59T  1.25K     19  51.1M   198K
    ada0p3      -      -     93      6  51.3M   199K
    ada1p3      -      -    190      6  51.3M   199K
  mirror     528K  15.5G      0      1      0   110K
    ada2p4      -      -      0      1    217   111K
    ada3p4      -      -      0      1    217   111K
cache           -      -      -      -      -      -
  ada2p3     582M  63.4G      0      1    123   147K
----------  -----  -----  -----  -----  -----  -----

SirDice · Nov 5, 2020

Does sysctl machdep.bootmethod show you booted with EFI? Then you can use efibootmgr(8) to fix the entries in your BIOS/UEFI.

The efi partition can be restored with dd(1): dd if=/boot/boot1.efifat of=/dev/ada0p1.

Argentum · Nov 5, 2020

SirDice said:
Does sysctl machdep.bootmethod show you booted with EFI? Then you can use efibootmgr(8) to fix the entries in your BIOS/UEFI.

Code:

# sysctl machdep.bootmethod
machdep.bootmethod: UEFI

Argentum · Nov 5, 2020

SirDice said:
Does sysctl machdep.bootmethod show you booted with EFI? Then you can use efibootmgr(8) to fix the entries in your BIOS/UEFI.

The efi partition can be restored with dd(1): dd if=/boot/boot1.efifat of=/dev/ada0p1.

Sure, but that is the problem that I have new code in /boot/

I can not build the 12.1 EFI boot code any more with 12.2

Code:

cc: error: unknown argument: '-fformat-extensions'
*** Error code 1

Stop.
make[1]: stopped in /usr/src.12.1/stand/efi/libefi
*** Error code 1

12.2 EFI compiles OK, but I have the issue I described.

Code:

objcopy --only-keep-debug loader_simp.sym.full loader_simp.sym.debug
objcopy --strip-debug --add-gnu-debuglink=loader_simp.sym.debug  loader_simp.sym.full loader_simp.sym
if nm loader_simp.sym | grep ' U '; then  echo "Undefined symbols in loader_simp.sym";  exit 1;  fi
SOURCE_DATE_EPOCH=1451606400  objcopy -j .peheader -j .text -j .sdata -j .data  -j .dynamic -j .dynsym -j .rel.dyn  -j .rela.dyn -j .re
loc -j .eh_frame -j set_Xcommand_set  -j set_Xficl_compile_set  --output-target=efi-app-x86_64 loader_simp.sym loader_simp.efi

SirDice · Nov 5, 2020

Argentum said:
I can not build the 12.1 EFI boot code any more with 12.2

Simplest solution, boot from the 12.2 install media and use that to fix your system. There's a perfectly fine boot1.efifat on the install media you can use. There's no need to (re)build it from source.

While you are booted with the install media, double check machdep.bootmethod to make sure it's EFI booted. Then you can use efibootmgr(8) to fix your EFI boot options.

Argentum · Nov 5, 2020

SirDice said:
Simplest solution, boot from the 12.2 install media and use that to fix your system. There's a perfectly fine boot1.efifat on the install media you can use. There's no need to (re)build it from source.

While you are booted with the install media, double check machdep.bootmethod to make sure it's EFI booted. Then you can use efibootmgr(8) to fix your EFI boot options.

I will try it. Takes some time to load USB image.

Meanwhile, I had old 12.1 USB stick and I installed 12.1 EFI boot and built yet another kernel. It got worse. Also does not want to boot old 12.1 kernel. Here is how it looks like

BOOM! And ctrl-c does not even work. Needs power off. The same s*** with all my 4 kernels...

Now it is up after several tries and magic I do not understand. I will try boot 12.2 loader from from USB stick when it completes.

dd if=FreeBSD-12.2-RELEASE-amd64-memstick.img of=/dev/da1 bs=1m conv=sync

Argentum · Nov 5, 2020

SirDice said:
Simplest solution, boot from the 12.2 install media and use that to fix your system. There's a perfectly fine boot1.efifat on the install media you can use. There's no need to (re)build it from source.

While you are booted with the install media, double check machdep.bootmethod to make sure it's EFI booted. Then you can use efibootmgr(8) to fix your EFI boot options.

Now downloaded 12.2 install stick and mounted. The boot code is identical to my verision diff /boot/boot1.efifat /mnt/boot/boot1.efifat

Code:

# ls -l /boot/boot1.efifat /mnt/boot/boot1.efifat
-r--r--r--  1 root  wheel  819200 Nov  4 18:08 /boot/boot1.efifat
-r--r--r--  1 root  wheel  819200 Oct 23 08:53 /mnt/boot/boot1.efifat

Code:

# gpart bootcode -p ./boot1.efifat -i 1 ada0
partcode written to ada0p1

SirDice · Nov 5, 2020

Argentum said:
Code:

# gpart bootcode -p ./boot1.efifat -i 1 ada0 partcode written to ada0p1

Once that's done. Run efibootmgr. Remove that Windows boot option, remove everything that doesn't belong. Add a boot option for FreeBSD:
efibootmgr -c -l /mnt/EFI/BOOT/BOOTX64.EFI -L FreeBSD

Code:

     The UEFI standard defines how hosts may control what is used to bootstrap
     the system.  Each method is encapsulated within a persistent UEFI
     variable, stored by the UEFI BIOS of the form BootXXXX.  These variables
     are numbered, describe where to load the bootstrap program from, and
     whether or not the method is active.  The boot order of these methods is
     controlled by another variable BootOrder.  The currently booting method
     is communicated using BootCurrent.  A global timeout can also be set.

The path to BOOTX64.EFI doesn't really matter, efibootmgr(8) just has to have access to it when you enter this command.

Argentum · Nov 5, 2020

SirDice said:
That is the wrong procedure.

dd if=./boot1.efifat of=/dev/ad0p1

I can try dd, but what is the difference? I have always used gpart and the boot code is there.

Also, with 12.1 I used gpart. One of the drives in mirror is brand new and booted OK just yesterday before the story I described in the beginning.

SirDice · Nov 5, 2020

Argentum said:
I can try dd, but what is the difference? I have always used gpart and the boot code is there.

I double checked and gpart(8) should work just as well. I didn't realize gpart(8) does the right thing.

Argentum · Nov 5, 2020

SirDice said:
I double checked and gpart(8) should work just as well. I didn't realize gpart(8) does the right

SirDice said:
Once that's done. Run efibootmgr. Remove that Windows boot option, remove everything that doesn't belong. Add a boot option for FreeBSD:
efibootmgr -c -l /mnt/EFI/BOOT/BOOTX64.EFI -L FreeBSD

Will try this tomorrow and let you know. Meanwhile recompiling and installing the world under new system just to be sure.

Argentum · Nov 6, 2020

SirDice said:
Once that's done. Run efibootmgr. Remove that Windows boot option, remove everything that doesn't belong. Add a boot option for FreeBSD:
efibootmgr -c -l /mnt/EFI/BOOT/BOOTX64.EFI -L FreeBSD

Hello!

Almost bricked my computer, but it is up again now. Recompiled the world, installed EFI bootcodes and this was OK.
Then, first, did efibootmgr -o B,C. After that it was able to boot, but still showed Windows in boot menu.

After that I did efibootmgr -B 0 - removed that Windows BootVar. After that it was a brick. Did not let me to the motherboard settings or boot order menu, nothing. Fortunately I could clear the cmos settings by replacing jumper on motherboard and right now it is up again.

Not sure that it is completely OK, but it is up and running after some struggle.

Argentum · Nov 6, 2020

Now it booted first time like before.

What is the retrospective of that saga?
Maybe - "If it is not broken, do not fix it!" -- 12.1 worked fine and I rushed to install 12.2. And all I got were boot issues... Nothing else. I am happy that it works like two days ago.

... but I got some experience and learned from my mistakes

... will not mark this thread 'solved' before I have tested both mirror drives boot and convinced it is all working OK ?

Code:

# efibootmgr -v
BootCurrent: 0003
Timeout    : 0 seconds
BootOrder  : 0001, 0002, 0003, 0004, 0005, 0006
Boot0001* USB Floppy/CD VenMedia(b6fef66f-1495-4584-a836-3492d1984a8d,0500000001)
Boot0002* USB Hard Drive VenMedia(b6fef66f-1495-4584-a836-3492d1984a8d,0200000001)
+Boot0003* TOSHIBA DT01ACA200 PciRoot(0x0)/Pci(0x11,0x0)/Sata(0x0,0xffff,0x0)/HD(1,GPT,c9c3ffb0-8a29-11ea-90ac-38607777c922,0x28,0x6400
0)
Boot0004* TOSHIBA HDWD120 PciRoot(0x0)/Pci(0x11,0x0)/Sata(0x1,0xffff,0x0)/HD(1,GPT,a1d9dbe2-0328-11eb-a732-38607777c922,0x28,0x64000)
Boot0005* USB Floppy/CD VenMedia(b6fef66f-1495-4584-a836-3492d1984a8d,0500000000)
Boot0006* Hard Drive BBS(HD,,0x0)

If I touch that again, does it blow up? ?

... some time later ...

Code:

# efibootmgr -v
BootCurrent: 0004
Timeout    : 0 seconds
BootOrder  : 0001, 0002, 0003, 0004, 0005, 0006
 Boot0001* USB Floppy/CD VenMedia(b6fef66f-1495-4584-a836-3492d1984a8d,0500000001)
 Boot0002* USB Hard Drive VenMedia(b6fef66f-1495-4584-a836-3492d1984a8d,0200000001)
 Boot0003* TOSHIBA DT01ACA200 PciRoot(0x0)/Pci(0x11,0x0)/Sata(0x0,0xffff,0x0)/HD(1,GPT,c9c3ffb0-8a29-11ea-90ac-38607777c922,0x28,0x64000)
+Boot0004* TOSHIBA HDWD120 PciRoot(0x0)/Pci(0x11,0x0)/Sata(0x1,0xffff,0x0)/HD(1,GPT,a1d9dbe2-0328-11eb-a732-38607777c922,0x28,0x64000)
 Boot0005* USB Floppy/CD VenMedia(b6fef66f-1495-4584-a836-3492d1984a8d,0500000000)
 Boot0006* Hard Drive BBS(HD,,0x0)

good so far ? ?

VladiBG · Nov 6, 2020

Stop using efifat image.

⚙ D20562 The efifat files are no longer used: remove the code to build them.

reviews.freebsd.org

UEFI - FreeBSD Wiki

Argentum · Nov 6, 2020

VladiBG said:
Stop using efifat image.

⚙ D20562 The efifat files are no longer used: remove the code to build them.

reviews.freebsd.org

UEFI - FreeBSD Wiki

Which image do you recommend?

VladiBG · Nov 6, 2020

It's not an image anymore read the wiki page how to copy the efi into fat32 partition

quote from the wiki

mount -t msdosfs /dev/da0p1 /mnt
mkdir -p /mnt/EFI/BOOT
cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.efi
umount /mnt

Argentum · Nov 6, 2020

VladiBG said:
It's not an image anymore read the wiki page how to copy the efi into fat32 partition

quote from the wiki

I have this BOOTX64.efi file in EFI partition. I can boot it up, but with an anomaly.

When my boot order is

Code:

# efibootmgr -v
BootCurrent: 0003
Timeout    : 0 seconds
BootOrder  : 0004, 0003, 0005, 0006, 0007, 0001
 Boot0004* TOSHIBA HDWD120 PciRoot(0x0)/Pci(0x11,0x0)/Sata(0x1,0xffff,0x0)/HD(1,GPT,a1d9dbe2-0328-11eb-a732-38607777c922,0x28,0x64000)
+Boot0003* TOSHIBA DT01ACA200 PciRoot(0x0)/Pci(0x11,0x0)/Sata(0x0,0xffff,0x0)/HD(1,GPT,c9c3ffb0-8a29-11ea-90ac-38607777c922,0x28,0x64000)
 Boot0005* USB Floppy/CD VenMedia(b6fef66f-1495-4584-a836-3492d1984a8d,0500000001)
 Boot0006* USB Hard Drive VenMedia(b6fef66f-1495-4584-a836-3492d1984a8d,0200000001)
 Boot0007* Hard Drive  BBS(HD,,0x0)
 Boot0001* USB Floppy/CD VenMedia(b6fef66f-1495-4584-a836-3492d1984a8d,0500000001)

i.e. 4,3 I should seclect the next boot device manually from BIOS boot menu. When I change the boot order by

Code:

# efibootmgr -o 3,4
BootCurrent: 0003
Timeout    : 0 seconds
BootOrder  : 0003, 0004
+Boot0003* TOSHIBA DT01ACA200
 Boot0004* TOSHIBA HDWD120

then I can not boot automatically, but should select the next boot device. Strange magic

VladiBG · Nov 6, 2020

Check if the TOSHIBA HDWD120 efi partition is the same as TOSHIBA DT01ACA200

Entire boot process is described here:

uefi

www.freebsd.org

Argentum · Nov 7, 2020

VladiBG said:
Check if the TOSHIBA HDWD120 efi partition is the same as TOSHIBA DT01ACA200

Entire boot process is described here:

uefi

www.freebsd.org

I know that. I have deleted both partitions, recreated and installed exactly the same code.

I have one laptop and 2 desktop FreeBSD machines with EFI boot and ZFS.

First, I upgraded the laptop which contains no critical data. Everything went well. After that I upgraded the other desktop, which also does not contain any critical data except the system configuration. That upgrade went well also.

Agree, that upgrading the critical machine was my mistake. I did not check the hardware with 12.2. Now I have found out that even the original FreeBSD 12.2 USB stick does not boot.

Looks like my HP motherboard ans FreeBSD 12.2 do not love each other. The board loved or at least tolerated 12.1

The philosophy here is that all the source code of 12.2. is inside my machine and it compiles and works well. The boot problem should also reside inside that code and is maybe just one line or some parameter. It could be fixed if I knew what to fix. Unfortunately I do not. Maybe somebody here has s deeper knowledge. The HP motherboard can not be changed. No BIOS updates are available. The only alternative is to replace the motherboard. But it is not broken. When it boots up, it is perfectly stable and can run days and weeks.

The tricky part is that I can not even isolate the problem with 100% confidence. With some sorcery it boots up. I can not exactly define that sorcery. It involves playng around in boot order settings and restarting. Another clue is that, however the original USB stick does not boot, its presence helps to break the ice.

Argentum · Nov 19, 2020

Argentum said:
Looks like my HP motherboard ans FreeBSD 12.2 do not love each other. The board loved or at least tolerated 12.1

Has been some time since I wrote my last message here, but it is still not good.

As I already mentioned, even the original FreeBSD 12.2 USB USB image does not boot on this machine, but the presence of USB installation stick somehow makes my main mirror to boot.

As an experiment, I have changed the EFI loader on USB stick by cp /boot/loader.efi /mnt/efi/boot/BOOTx64.efi. On another machine there is no change - the stick still boots. The same applies to this machine - no change. The same stick does not boot, but its presence helps the main mirror to boot.

Just to remember, with 12.1 everything was OK. The machine is stable - no problem to keep it up a week even under heavy load.

Do you have any further ideas? Want my machine to become bootable again without changing the motherboard.

:beer:

guidok · Nov 20, 2020

VladiBG said:
Stop using efifat image.

So you're saying man 8 uefi is out of date? That would be a real shame. As it is I'm trying to devise what to do with regards to the boot code after a zfs pool upgrade -a, but if I can't trust the man pages, things will get hairy rather quickly.

VladiBG · Nov 20, 2020

The man page of uefi(8) is not out of date. The image of the fat file system is used by bsdinstall and overwrite the entire EFI partition which can contain different EFI files with different configurations like PC Hardware UEFI diagnostic tools, Raid configuration firmware, IPMI configuration , etc.
So instead of overwriting the entire efi partition with image file it's better to mount this FAT32 partition then copy the .efi files and modify the startup.nsh then create the boot record inside UEFI (bios) on your motherboard using efibootmgr

UEFI - FreeBSD Wiki

⚙ D20562 The efifat files are no longer used: remove the code to build them.

reviews.freebsd.org

https://www.intel.com/content/dam/support/us/en/documents/motherboards/server/sb/efi_instructions.pdf

guidok · Nov 20, 2020

VladiBG What I meant with out of date is that uefi(8) mentions boot1.efifat whereas you say we should stop using it. I think you are right though. Because when checking the loader in my ESP it seems to be a version of loader.efi instead of boot1.efi judging by its size:

Bash:

% sudo mount -t msdos /dev/nvd0p1 /mnt

% ll -h /mnt/efi/boot
total 480
drwxr-xr-x  1 root  wheel   8.0K Nov 29  2019 .
drwxr-xr-x  1 root  wheel   8.0K Nov 29  2019 ..
-rwxr-xr-x  1 root  wheel   450K Nov 29  2019 BOOTx64.efi
-rwxr-xr-x  1 root  wheel    12B Nov 29  2019 startup.nsh

Had boot1.efifat been used to create this ESP I would have expected boot1.efi (92K), to have been installed as BOOTx64.efi. Which clearly is not the case. Instead it is a 450K file, which matches more with current versions of loader.efi (487K)

Anyway now I know how to proceed to properly update my UEFI loader code:

Bash:

sudo cp /boot/loader.efi /mnt/efi/boot/BOOTx64.efi

I'm somewhat amazed this is required for a minor bump of FreeBSD (12.1 -> 12.2)

Anyway, thank you VladiBG

Solved UEFI boot anomaly after update to 12.2

Argentum

SirDice

Administrator

Argentum

SirDice

Administrator

Argentum

Argentum

SirDice

Administrator

Argentum

Argentum

SirDice

Administrator

Argentum

SirDice

Administrator

Argentum

Argentum

Argentum

VladiBG

⚙ D20562 The efifat files are no longer used: remove the code to build them.

Argentum

⚙ D20562 The efifat files are no longer used: remove the code to build them.

VladiBG

Argentum

VladiBG

uefi

Argentum

uefi

Argentum

guidok

VladiBG

⚙ D20562 The efifat files are no longer used: remove the code to build them.

guidok