Solved How To Dual Boot Windows 11 and FreeBSD 14 (GELI Encrypted ZFS Root + UFS Boot Drive)

Introduction:
This is a guide on how to dual boot windows 11 and FreeBSD 14.
I recently received a fantastic laptop that happens to be surprisingly compatible with FreeBSD,
so I highly recommend it, and no, I am not sponsored by MSI;
I have merely fought enough laptops to recognize a keeper when I see one, and I figured I would spread the news ;D

The laptop I am talking about, is the MSI KATANA GF76 12UG (MS-17L3)
It will serve as our installation target.

Assumptions:
The situation we are in, can be compared to the chicken and the egg problem;
You need a PC to create a live USB stick.
But you also need a live USB stick to install a system on your PC!

I am going to assume you possess a working FreeBSD 14 OS on your PC.
I've never met a human being that knew what FreeBSD was, who did not possess at least three USB sticks,
but in case you don't, make sure to acquire three for the purposes of this tutorial.
They should be at least 4GB, 8GB, and 8GB in size, but 4GB, 16GB, and 8GB is probably recommended, as 8GB will be cutting it close.

Don't Be Like Bob!
Bob is on vacation with his family, including his beautiful, but not so smart wife, who posts everything on social media.
Burglars viewing his wife's profile quickly realize their house is empty and decide to break in...
"Oh no! They stole my laptop!" - Bob screams in terror, as he realizes his belongings are gone, upon returning to the house.
"It had my private diary and rare bird collection on it!" - Bob cries quietly in the corner, mourning the loss of his data,
in fear of what the future will hold...

Don't be like Bob, encrypt your data!
Especially if it's something that should not see the light of day!
The method we are going to employ, won't save you from data loss,
but it is likely to prevent anyone else from using it against you, in case of theft, for example.

General Overview:
Here's the rundown; We are going to install Windows 11 first.
We will then install rEFInd, and finally FreeBSD.
We will configure the UEFI in such a manner, that rEFInd will be booted from by default.
In rEFInd we will provide two menu entries, one being the Windows loader,
and the second, the volume containing the FreeBSD loader and kernel.
We will encrypt a ZFS root partition on the laptop's internal drive;
We will configure the FreeBSD loader in such a way that, in combination with a passphrase
and a key, it will decrypt the zfs partition for us at boot.


Step 0: Acquire The Goodies!
You will need the following:

[TABLE=collapse]
[TR]
[TD]The Windows 11 image[/TD]

[TD] [/TD]
[/TR]
[TR]
[TD]The FreeBSD image[/TD]

[TD] [/TD]
[/TR]
[TR]
[TD]The rEFInd boot manager[/TD]

[TD] [/TD]
[/TR]
[TR]
[TD]The UFS EFI driver[/TD]

[TD] [/TD]
[/TR]
[TR]
[TD]The katana drivers[/TD]

[TD] [/TD]
[/TR]
[TR]
[TD]The following packages[/TD]

[TD]rsync, 7-zip, fuse, fusefs-ntfs[/TD]
[/TR]
[/TABLE]

For the katana drivers, we only really need the Intel Rapid Storage Technology F6 Driver,
but believe me, if you want the wifi to work on Windows 11, also download the wifi driver ;D

Step 1/2: Identify Your Storage Media:
If you're a newbie, or you just seem to forget these commands for some reason,
these can help you narrow down the device identifier that corresponds to a given disk:

Bash:
camcontrol devlist

Bash:
gpart show
Step 1: Prepare The FreeBSD Bootable Media!
This one is easy; In fact, if you're anything like me, you probably already have and always keep a FreeBSD USB stick on you;
But in case you don't:

FIRST VERIFY THAT THE CHECKSUMS MATCH!!!

I am not kidding! You would not believe how unobvious a corrupted live image can be,
and If you haven't already, you do not want to find out!
So just in case, download the checksum file, and compare the sha256 image checksum with the output of

Bash:
sha256sum /path/to/freebsd/image

Once, you've verified everything is in order, proceed to write the image onto the smaller of the two USB sticks.

Bash:
dd if=/path/to/freebsd/image of=/path/to/bsdusb status=progress

I shall refer to this USB stick as bsdstick hence forth!
That's it! It's really that simple.
I want you to remember this moment the next time you're about to say something about how complicated FreeBSD is.

Step 2: Prepare The Windows 11 Bootable Media!
A lot of tutorials, say something like, preparing the bootable media is out of the scope of this tutorial...

e7633bedf897bb24ce668ac9c5df6bf88a58ff7e114d27606a756f4c4888a3f1.jpg


We are not going to take the easy route, moi cherry!
Fasten your seat belt, it's going to be a bumpy ride...
I've mentioned in the previous section it's important to verify checksums.
Keep this in mind, the Windows 11 image is a couple times larger than the FreeBSD image,
so if you checked the checksum for FreeBSD, then I shouldn't have to convince you to do it in this case.
But I will say it again, anyway, VERIFY THAT THE CHECKSUMS MATCH!!!

One thing worth mentioning, is that the checksums for Windows image files tend to be in all caps.
Do not fear, you can easily convert to lowercase using awk:

Bash:
echo "$expected_hash" | awk '{print tolower($0)}'

Okay, let's do this!
First, we shall wipe the drive and create a master boot record partitioning scheme on it:
Bash:
gpart destroy -F /path/to/winusb
gpart create -s mbr /path/to/winusb

Next, we will create the first partition that will hold just enough data to kick start the installer,
so that it can read the big file over 4gs in size from the partition to come:
Bash:
gpart add -t fat32 -a 1m -s 1G /path/to/winusb
newfs_msdos /path/to/winusbs1

As mentioned, the install.wim file is over 4Gs so it has to be kept on a second, NTFS partition:
Bash:
gpart add -t ntfs -a 1m -s 7G /path/to/winusb

Right, so before we can create a file system on the partition, make sure to load the fusefs module;
You can check if it's loaded using the kldstat command.
If it isn't, you can load it like so:

Code:
kldload fusefs

Once that is taken care of, proceed to create the filesystem:

Bash:
mkntfs /path/to/winusbs2

If it's not in your path, search for it using whereis, or the find utility;

Now, we still need to create one more partition to carry our drivers, etc...
Bash:
gpart add -t fat32 -a 1m /path/to/winusb
newfs_msdos /path/to/winusbs3
I hope this goes without saying, but just in case, /path/to/winusbsX should be something like /dev/da1sX
Since we are using the MBR partitioning scheme, make sure to use 's' instead of 'p'.

Alright, so we have three partitions with file systems created on them;
Let's mount them:
Bash:
mkdir mnt1 mnt2 mnt3
mount -t msdosfs /path/to/winusbs1 ./mnt1
ntfs-3g /path/to/winusbs2 ./mnt2
mount -t msdosfs /path/to/winusbs3 ./mnt3

Can you guess what the next step will be?
I bet if you're a linux user that's done this procedure before, you'd say something like:

"We should mount the ISO image to copy from it"

HAHAHAHA, WRONG! You're in FreeBSD land now, can you hear the wilderness calling?

Seriously, though, you're not technically wrong; In fact you would run mdconfig to configure a virtual device
and then mount the ISO image using that.
But let me tell you a little secret, only a readme file would be waiting for you, saying that your ISO file is not really
an ISO file, but more of an ISO-13346 file.
So what does that mean for us?
As usual, we must resort to trickery!
Behold, the genius solution to our problem:
Use 7-zip, which for some reason is capable of extracting the contents of the Windows ISO file:
Bash:
7z x /path/to/win.iso -oiso_dir

Once that is done, we must copy the files in the following manner:
Bash:
rsync -r --progress --exclude sources ./iso_dir/ ./mnt1/
mkdir ./mnt1/sources
cp ./iso_dir/sources/boot.wim ./mnt1/sources/
rsync -r --progress ./iso_dir/ ./mnt2/

Perfect, but don't forget about the katana drivers, and our EFI associated friends;
Copy them over to ./mnt3
And do yourself a favor by unpacking the disk driver...

Now it is time to unmount the partitions and do a little cleaning:
Bash:
umount ./mnt1 ./mnt2 ./mnt3
sync
rmdir ./mnt1 ./mnt2 ./mnt3
rm -R ./iso_dir

From this point on, I shall refer to this USB stick as winstick!
 
Step 2 And 1/2: Prepare The UEFI:

This may, or may not be obvious to you, depending on whether, or not you've dual booted before with Windows.
Secure, and fast boot must be switched off in the UEFI settings.
On the katana, you can enter them by pressing the DEL key during startup.
Also, make sure booting via USB is enabled, and that the VMD controller is enabled.
Otherwise, there will be a problem when installing the disk driver.

Step 3: Install Windows 11:

No, I am not insulting your intelligence, by holding your hand through the Windows 11 installation process.
There are some things you should know that are not obvious at all.
First of all, to enter the boot menu on the katana, press F11
Select the winstick; Once you start the installation you will be greeted with a message saying you are missing

drivers and that a disk cannot be detected. Press the browse button and select the directory that holds the unpacked
driver. Then proceed. Select all the available drivers, and click next.
With a bit of luck your disk should now be detected.
Second of all, just make sure to leave like 66GBs free.
We will use 16GBs as swap, and 50GBs for the ZFS root partition.
The rest can be consumed by Windows, so you have more space for video games, or whatever...
Okay, so the installation has rebooted the PC and you've eventually realized it's a trap!
The installer requires you to have a microsoft account! What a load of crock.
Luckily, even though you can no longer click I don't have an internet connection, like you could when installing Windows 10,
you can once again resort to trickery!
Open up a terminal emulator using SHIFT+F10
Enter:
Code:
OOBE\BYPASSNRO
This will reboot the PC and you will be able to select the I don't have an internet connection button, like in the good old days.
Remember that you have the wifi driver on the winstick, and good luck with the rest of the installation / updating, etc...
 
Step 4: Install rEFInd - Part 1:

Because the Windows 11 installer, so conveniently created an EFI partition for us, we can now hack it.
Boot up using the bsdstick, and select the Live CD option. Log in as root.

It's time to locate the EFI partition. A simple gpart show, as previously mentioned,
allows us to see, the partition we are interested in is /dev/nda0p1

Before we do anything with that, though, first plug in the winstick and mount its third partition;
Again, make sure you use the device ID you see on your screen!
It should be the same as mine (/dev/da1), if you have only two usb sticks plugged in, but you never know:

Bash:
mkdir /tmp/drivers
mount -t msdosfs /dev/da1s3 /tmp/drivers

Now, let's mount the EFI partition:

Bash:
mkdir /tmp/efi
mount -t msdosfs /dev/nda0p1 /tmp/efi

Moving on, it is recommended to modify the contents of this partition in the following way:

Bash:
cp /tmp/drivers/refind-bin-0.14.0.2.zip /tmp/efi/EFI/
cp /tmp/drivers/ufs2_x64.efi /tmp/efi/EFI/
umount /tmp/drivers
cd /tmp/efi/EFI
unzip ./refind-bin-0.14.0.2.zip
rm ./refind-bin-0.14.0.2.zip
mv ./refind-bin-0.14.0.2/refind ./
rm -R ./refind-bin-0.14.0.2
cd ./refind
rm -R ./drivers_aa64 ./drivers_ia32 ./refind.conf-sample ./refind_aa64.efi ./refind_ia32.efi ./tools_aa64 ./tools_ia32
rm ./drivers_x64/*
mv ../ufs2_x64.efi ./drivers_x64/

Were are removing all but the bare minimum, to minimize any potential interference from optional drivers.
We have also removed the sample config file.
We must now create a new one (make sure you are still in the refind directory):

Bash:
touch ./refind.conf

Inside we want to have the following:

Code:
timeout 10
scanfor manual

menuentry "Windows 11" {
    icon /EFI/refind/icons/os_win8.png
    loader /EFI/Microsoft/Boot/bootmgfw.efi
}

The first line sets the timeout, before a selection is made for you.
I set this to 10 seconds, but feel free to change it to your liking.

The second line says we don't want to be polluted with automatically detected trash.
But, again, if that is your style, feel free to change it.
I am not going to go into details here, but I will post a couple of links to some bibliography
that should help you achieve any modifications.

Finally, we have the menu entry, consisting of an icon (feel free to change that as well; there are quite a few icons available, actually),
and a path to the loader. We are not quite done yet, though.

We still have to update the UEFI boot registers;
For this task we will use efibootmgr.
The first thing we are gonna do, is list what we have.
Simply issue efibootmgr. It will print out a list of boot options.
We want to add our new boot manager to the list:

Bash:
cd
efibootmgr -c -l /tmp/efi/EFI/refind/refind_x64.efi -L rEFInd

We are specifying the create option combined with the path to the loader, and a label.
This command should print out the updated list at the end.

I can see that 'Boot0000 rEFInd' has appeared.
After that we should mark the new entry as active:

Bash:
efibootmgr -a -b 0

Finally, reboot. Technically, this might be enough, but for me, it wasn't;
I still had to enter the UEFI settings, navigate to the Boot tab and:

(a): Set Boot Option #1 in 'FIXED BOOT ORDER Priorities' to 'Hard Disk:rEFInd'.
(b): Inside 'UEFI Hard Disk Drive BBS Priorities' I had to mark Boot Option #1 as rEFInd.

If everything went according to plan, during the next boot you should be greeted by the rEFInd menu,
and upon selecting the windows icon and pressing enter, the Windows loader should be executed.
This concludes the first part of configuring rEFInd.
 
Step 5: Install FreeBSD:

If you've made it this far, I congratulate you! But only if you actually managed to get everything to work.
Otherwise, I am not impressed. Anyway, moving on, we are going to put the cherry on top of the cake now.
We are going to get off the beaten track and perform a manual installation. Are you excited yet?
Boot up from the bsdstick, select Live CD, and once again log in as root.

Let's take a look at our partition table for the disk where Windows 11 was installed (/dev/nda0).
When I created a new partition during the Windows installation, I specified 900,000MB.
Now, gpart tells me I have 75GBs left. Perfect!

The time has come to create our encrypted ZFS partition.
Load the ZFS module and set the ashift property to 12.

It'll be better that way - source: trust me bro

But seriously, it will most likely be better that way, even if you have a disk with native 512 alignment.
On the katana you should definitely do it. You can find more info on this in the bibliography below.

Anyway, here are the commands:

Bash:
kldload zfs
sysctl vfs.zfs.min_auto_ashift=12

Now, that we have that out of the way, let's create our kiddie pool!
First let's create a swap partition, so that we don't forget later on:

Bash:
gpart add -t freebsd-swap -s 16g -a 4k -l swap /dev/nda0

The laptop has 16GBs of ram by default, so it's wise to create a swap partition of at least the same size.
What I am about to say, is very important, so pay extra attention now!
Otherwise, you might crash your live CD.
When creating a zpool, and especially when importing one, you have to be careful, otherwise you might

accidentally mount the resources at /
I hope I don't have to say this. That will cause problems. So be CAREFUL!
Before we create the pool, we still have to add a partition for it:

Bash:
gpart add -t freebsd-zfs -a 4k -l root /dev/nda0

Okay, fantastic. Next up, we must generate a key file.
(Actually, we don't have to, but for extra protection, you should).
The thing is, if we generate the key right now, we will have to remember to copy it over to the keystick
later on. And I might not remember to do that, so we will postpone the ZFS pool creation for a moment
and focus on creating the keystick. That is what I call the third usb stick that will hold the FreeBSD boot
partition (including the kernel), and the key file.

So go ahead, and plug in the third USB stick. On my PC it says it's /dev/da1.
If yours is different, modify the following commands accordingly:

Bash:
gpart destroy -F /dev/da1
gpart create -s gpt /dev/da1
gpart bootcode -b /boot/pmbr /dev/da1
gpart add -t efi -s 50m -l EFI /dev/da1
gpart add -t freebsd-ufs -l KEY /dev/da1
gpart set -a bootme -i 2 /dev/da1
newfs_msdos /dev/da1p1
newfs -j /dev/da1p2

We've resorted to trickery, again.
Notice how we created an efi partition (really, it's a fat32 partition under cover) for the loader.
We want to store the kernel on the same partition as the loader.
However, we also want to be able to keep unix links without them breaking, so the trick is to

create a small EFI partition, place the loader on it, and mark the next partition with the 'bootme' attribute.
We will store the /boot goodies on the ufs partition and because it's marked 'bootme', the loader will jump to it
automatically, should we choose to boot from the stick (we won't (we will be using rEFInd, remember xD?), but it's still good practice).

Okay, so let's first place the loader in the EFI partition:

Bash:
mkdir /tmp/efi
mount -t msdosfs /dev/da1p1 /tmp/efi
mkdir -p /tmp/efi/EFI/Boot
cp /boot/loader.efi /tmp/efi/EFI/Boot/bootx64.efi
umount /tmp/efi

Next, let's initialize the ufs partition:

Bash:
mkdir /tmp/key
mount /dev/da1p2 /tmp/key
cp -Rp /boot /tmp/key/

Great, since we've got this, we can return to the key file creation:
Code:
dd if=/dev/random of=/tmp/key/boot/katana.geli.key0 bs=64 count=1

Splendid! Now, that we have our key, we may proceed with the zpool creation.
I'm going to do something that is generally frowned upon by the FreeBSD overlords,
but I am doing it to keep things simple for the purpose of this tutorial.
I am going to perform the installation on a single partition.
Feel free to modify the dataset creation part as you wish.
But before we get to that, we must create the encrypted device, and for that
we shall use the generated key file:

Bash:
geli init -b -s 4096 -K /tmp/key/boot/katana.geli.key0 /dev/nda0p6

We are basically specifying the alignment, the key file, and most importantly, the -b flag,
which means, we want the loader to decrypt the partition during boot.
It is very important that you set this flag at this point. You cannot append it later!
Also make sure the partition number represents the ZFS partition.

If everything goes as expected, you will be asked for a passphrase.
Once entered, there will be a message about a backup file.
You can copy it over to somewhere off the system, If you like:

Alright, so we have created the encrypted geli device. But before we can use it, we must attach it:

Bash:
geli attach -k /tmp/key/boot/katana.geli.key0 /dev/nda0p6

You will be prompted for the passphrase again.

Once that is entered, you can notice a new device appears, the /dev/nda0p6.eli device.
And this is the device which we are interested in for pool creation.
Before we create the pool, though, we want to randomize the bits on the partition,
so that breaking the encryption is even harder:

Code:
dd if=/dev/random of=/dev/nda0p6.eli bs=1m

Next, we create the basic zpool and some datasets on it:

Bash:
mkdir /tmp/root
zpool create -o altroot=/tmp/root -O canmount=off -O mountpoint=none zroot /dev/nda0p6.eli
zfs create -o canmount=off zroot/ROOT
zfs create -o mountpoint=/ zroot/ROOT/default

Remember how I said, you have to watch out when creating datasets?
If we hadn't set the altroot, the zroot/ROOT/deafult dataset would have caused our live CD to crash.
That's because unless you explicitly specify that a newly created dataset isn't supposed to be mounted, it will be!
Same goes for importing, remember about the altroot option.

Okay, so if you issue zfs mount you will be able to see that instead of being mounted at / the default dataset is mounted at /tmp/root instead.
That is a good thing. Anyway, if you want to customize your datasets, now would probably be the time to do so.
If not, let's move on. It's time to install the base system:

Bash:
tar --unlink -xpf /usr/freebsd-dist/base.txz -C /tmp/root

This doubles our /boot, so we will remove it:

Bash:
rm -Rf /tmp/root/boot

As a final touch to the ZFS partition, we will configure the /etc/fstab file.
We want to have encrypted swap, and we want to automatically mount the /boot (ufs) partition during startup.
The file should look like this:

Bash:
/dev/nda0p5.eli none swap sw 0 0
/dev/gptid/294fe23e-d33c-11ee-a245-2cf05dff819c /stage3 ufs rw 0 1

Here, we are setting the swap to be encrypted by adding the .eli suffix. The swapon program will understand this automatically.
The next line has the GUID of the ufs partition of our keystick. We are mounting it in a directory called stage3, because
it is the third stage loader that resides in it. Normally, the root partition should have 1 in the last field, but because we have
a ZFS root partition, fsck is not welcome, so our ufs partition takes its place.

Of course, the stage3 directory doesn't exist on its own. We must create it,
and let's create a soft link to /boot while we are at it:

Bash:
mkdir /tmp/root/stage3
ln -s /stage3/boot tmp/root/boot

By the way, if you are wondering, how I found the GUID, I did it using gpart list
Notice, how the link is a little bit tricky. We are not linking to /tmp/root/stage3/boot,
because once we boot into the system, the path will have changed. But not the location of the link.

Now, this is all swell and well, but there is something missing, can you guess what that is?
Of course, yes it's setting up the loader.conf file.
We have to somehow specify where the loader is to look for the key file, etc...
The /tmp/key/boot/loader.conf file should look like this:

Bash:
vfs.mountroot.timeout="10"
loader_menu_multi_user_prompt="Best System Distribution Loader"

zfs_load="yes"
geom_eli_load="yes"

geli_nda0p6_keyfile0_load="yes"
geli_nda0p6_keyfile0_type="nda0p6:geli_keyfile0"
geli_nda0p6_keyfile0_name="/boot/katana.geli.key0"

vfs.root.mountfrom="zfs:zroot/ROOT/default"

Okay, and that would be it for the FreeBSD portion, though not entirely...
Anyway, you should familiarize yourself with the commands that are issued for you automatically when the system is reboot.
You can unmount and detach the devices yourself like so:

Bash:
zfs umount zroot/ROOT/default
zpool export zroot
geli detach /dev/nda0p6.eli
umount /tmp/key
 
Step 6: Install rEFInd - Part 2:

That's it, you're so close!
Before we reboot, we still have to add an entry to our refind.conf file.
Let's mount the EFI partition once more:
Bash:
mount -t msdosfs /dev/nda0p1 /tmp/efi
And add the following to the existing /tmp/efi/EFI/refind/refind.conf file:

Code:
menuentry "FreeBSD 14" {
    icon /EFI/refind/icons/os_freebsd.png
    volume 294fe23e-d33c-11ee-a245-2cf05dff819c
    loader /boot/loader.efi
}

Do you recognize the GUID? If you do that's because it's the same one we used in the /etc/fstab file.
So, what we are saying here, is that we want rEFInd to jump to the ufs partition on our keystick;
From there, the bsd loader takes over, loads the kernel and mounts the ZFS root partition, and then the
init script is run. Now you can unmount the partition and reboot. Finally!

Bash:
umount /tmp/boot
reboot

Tips And Tricks!

Chances are you made a typo somewhere in /etc/fstab and you're dropped into a shell in single user mode.
You try to update the file, but alas it turns out the root partition has been mounted in read-only mode!
An easily solvable problem, that nevertheless might be startling to those who are new to ZFS;
Simply remount the dataset:

Bash:
zfs mount -o remount,rw zroot/ROOT/default

Bonus Round: Connecting To The WIFI!
The installer will automatically setup the wifi interface during the installation, if you choose to.
Because we skipped the installer entirely, we have to bring up the interface manually.
Luckily, this gives us a fantastic opportunity to learn more about networking!
Here is how you would connect to the wifi:

The first thing you would probably do, is look at the output of ifconfig,
but it turns out the wifi card won't be there by default.
You need to first query the available hardware like so:

Bash:
sysctl net.wlan.devices

On the katana the output will include 'iwlwifi0';
This gives enough to create the missing interface:
Bash:
ifconfig wlan0 create wlandev iwlwifi0

Once the interface is created, set it up, and scan for available networks:

Bash:
ifconfig wlan0 up
ifconfig wlan0 scan

When you have the SSID (network name) you are interested in, and the PSK (password) to connect to it,
you can create the /etc/wpa_supplicant.conf file, with the following contents:

Code:
network={
    ssid="name-of-you-network"
    psk="password"
}

Then start wpa_supplicant and acquire an IP address like so:

Code:
wpa_supplicant -B -i wlan0 -c /etc/wpa_supplicant.conf
dhclient wlan0


BIBLIOGRAPHY:

There are a bunch of resources where you can learn more about this whole process,
but the most valuable by far, were the books from the FreeBSD Mastery series:

FreeBSD Mastery: Storage Essentials - Michael W Lucas
FreeBSD Mastery: ZFS - Michael W Lucas


If you are considering, but are not sure.
Let me tell you, they are definitely worth the money!
They provide the clarifications and examples that the man pages / handbook often lack.

 
Back
Top