FreeBSD boot sequence under the hood

B

birbignano

Hi guys. I must admit I am starting to lose it :D

I have been investigating FreeBSD internals the last week and it really looks like magic to me. First thing I want to state my objectives. I want to build a sort of live/ro/non tamperable installation of FreeBSD. I have investigated NomadBSD to this end and "the magic" has confused me even more.

By reading the handbook, absolute freebsd and the design and implementation of the freebsd os, I have started to understand a bit how it works. So it all starts with Bootx64.efi in the efi partition ( I have uefi ). Then it loads loader.efi. Loader.efi starts the kernel.

But, for it to work ( except in the standard case ) you should set vfs.root.mountfrom variable. But it isn't set in the livecd, nor in the nomadbsd images. So, I suppose the first part is "standard". It just loads /boot/kernel/kernel which, in turn, mounts the "current" freebsd-ufs partition as root filesystem. Then nomadbsd does some things with an image called /uzip/usr.local.uzip and unionfs. The problem is I cannot find where the mounting of the uzip image and of the unionfs happens.

The same goes for the freebsd livecd. It mounts the iso readonly. Ok, that is simple. But to boot, it needs a read-write overlay. Right? Where and when it mounts this overlay? I have watched in /boot. Nothing. I have watched in /etc/rc.d. Nothing. But there must be some script running at some point to mount the rw layer.

Also, what do you think is a good way to reach my objective of having a stateless system image + rw overlay in ram? Maybe just mount the root partition ro? Or maybe is it better to use an uzip image with md?

p.s. why the iso only has 2 partitions, one uefi and the other freebsd-boot? Shouldn't be a freebsd-ufs partition too, with the root fs on it? Instead the "root fileystem" is the entire image. I can access it by mounting the image as cd9660.
 
Seems like you found FreeBSD boot process in the handbook. AFAIK the boot loader have to know about root filesystem in order to locate and load the kernel. For MBR boot loader will use partition marked as active. For GPT gptboot(8) will boot from partition with bootme flag set (or bootonce; see gpart(8)). The kernel will run init(8), which will run /etc/rc (rc(8)).

Take a look at NanoBSD for instructions on how to build a FreeBSD system, running from a read-only filesystem, with /etc and /var in RAM-drive.
 
Have a look at diskless(8) too. While it's primarily about PXE booting, one of the features is that it boots from a read-only filesystem.
 
Interesting hints. But I have one question about UEFI booting. In uefi(8) it states that boot1.efi searches for the boot partition. So there is another boot loader stage in the middle!?! I thought it was bootx64.efi -> loader.efi. Also, where does bootx64.efi come from? Taking md5 hashes of it and loader.efi, boot1.efi, boot2, boot1.efifat, they don't match. So bootx64.efi is yet another boot loader? What is the point of having some many stages when you can host something complex like loader.efi on the UEFI partition?
 
The boot1.efifat file is a FAT32 (disk) image that's written directly to the efi partition.
 
No, it's written to the efi partition. The /boot/gptboot or /boot/gptzfsboot contents are written to the freebsd-boot partition.

CSM; MBR partitioning: Reads from sector 0 (MBR), and boots as described in the handbook: 12.2. FreeBSD Boot Process

CSM; GPT + UFS: Boots from code in the freebsd-boot partition. It contains gptboot(8).
CSM; GPT + ZFS: Boots from code in the freebsd-boot partition. It contains gptzfsboot(8).

EFI; GPT + UFS/ZFS: Reads efi partition, loads /EFI/BOOT/BOOTX64.EFI from it. See efi(8)
 
  • Thanks
Reactions: JAW
SirDice said:
EFI; GPT + UFS/ZFS: Reads efi partition, loads /EFI/BOOT/BOOTX64.EFI from it. See efi(8)
Click to expand...

I see. An important note in efi(8) is


"The path to the loader may be set by an EFI environment variable. If not set, an architecture-specific default is used."

The default is bootx64.efi. The specific one, that is run directly when a boot entry is added to the efi, is boot1.efi. So the boot chain is simplified and there is only one detour ( boot1.efi ). I will play with it a bit, setting loader.efi as the first stage boot loader. I think it works, because I have no boot.config variables to pass around.

The only thing I can't understand is where bootx64.efi comes from. MD5 hash doesn't match anyone of the bootloaders in /boot. So it must be something different. Some kind of secure boot signed bootloader? I cannot find its source code on github.
 
I feel like I have reached my AHA moment. Reading NanoBSD and TinyBSD scripts ( boot are in /usr/src/tools/tools...it looks like TinyBSD is a sort of modernized NanoBSD ), I have concluded that I was massively misinterpreting the role and features of GEOM_MD ( memory disks ). Coming from Linux, where I have built a sort of stateless system, using a squashfs image as root fs over which a tmpfs is overlayed ( through overlayfs ), I thought it was the same with FreeBSD. NOPE! MD already builds an "union" fs. The image file I set with -t vnode -f imagename, is just the "disk part" of the memory disk device. But MD overlays it with a chunk of ram, to use as the rw medium.

Am I right or am I seeing things? :D
 
aragats said:
This is incorrect. With current implementation it won't, we discussed this in another thread, see this post:
«boot1.efi ... will _always_ boot from the first freebsd-[zfs|ufs] partition it finds»․
Click to expand...
I believe that is a limitation for UEFI boot loader only: I've tried and failed to make dual root filesystem NanoBSD work with UEFI; gptboot(8) is for BIOS booting and does support bootme. On the other hand I've overlooked that Op's question refers to EFI boot.
 
Hi people, read here https://lists.freebsd.org/pipermail/freebsd-hackers/2009-April/028257.html

Looks like this is the solution.

There is only one catch, that the FreeBSD man states that these

Code: 
rootfs_load="YES"
rootfs_type="mfs_root"
rootfs_name="/boot/world.uzip"

are somewhat deprecated in favor of reboot -r. But the concept is the same. Just make an image, that kernel will start as a md device at boot, with an init script which will get executed, will mount the various components of the root fs, unionfs, etc... and then call reboot -r to chroot into the new root.

p.s. playing with the LiveCD, I see that it is dead simple. You have a "normal" ufs root fs ( mounted as ro ) and two scripts in /etc/rc.d ( var and tmp ) that mount /var and /tmp as md devices ( and read-write!! ). Is it that simple? I mean, the only places where FreeBSD writes something during operation, are /tmp and /var? I come from Linux and it writes things around the filesystem, in places you cannot even think of. And this forced me to overlay a tmpfs over the entire root fs /
 
You must log in or register to reply here.
Back
Top