bhyve A solid HOWTO with BHYVE on FreeBSD 14 and Windows 11

pathiaki

pathiaki

So, now that I've tried Vbox and found it's performance lacking, I've got a lot of feedback that I should try BHYVE. However, I'm finding a lot 'half' instructional documents. I'd like to get BHYVE running on FreeBSD14 HOST and Windows 11 running as a guest.

Some things that I've found is that there doesn't seem to be VNC server that is settled on and a few don't even have startup scripts so you can enable it from /etc/rc.d. Is this correct? Is there a preferred one?
(Everything seems to point to having a VNC server running on default port 5900 so you can start the 'install' of Windows with a VNC viewer by connecting to the VM with the viewer)

I'm a BHYVE newbie if that's not obvious.... be gentle. :D

Paul
 
Problem is, I never used Windows, and my bhyve instances are mainly automated containers for isolation only.
But indeed it seems that the bhyve docs are adressing somebody who already knows weill about the required things for virtualization.

I am not sure how important that VNC thing is, but in my understanding some of the important questions are (from bottom up):
  • how do you wire your networks?
  • how do you configure your virtual hardware (pci-slots, interfaces, etc.)
For this I didn't find configuration tools that were to my liking (commandline is a pain, packages are already too much), so I wrote shell scripts for rc.d, so that all the configuration can be put into rc.conf[.local].
 
I know I'm not addressing the question of "please give me a hand-holding guide" - but what I can say is that I have no problems with Windows 11 under bhyve. I have several Windows 11 bhyve VMs that I use daily and it works pretty well.

I wouldn't recommend using VNC, I usually use RDP. Definitely always with Windows clients but all of my graphical Linux bhyve VMs are using RDP too.

If you're not comfortable managing your VMs manually, I can recommend having a look at sysutils/vm-bhyve.
 
Well, so far, I've followed the Handbook on Bhyve. (I know, crazy, right? ;))

I did the following:

kldload vmm

# ifconfig tap0 create
# sysctl net.link.tap.up_on_open=1
net.link.tap.up_on_open: 0 -> 1
# ifconfig bridge0 create
# ifconfig bridge0 addm em1 addm tap0
# ifconfig bridge0 up

(exactly from Handbook except my interface on my internal network is em1)

From my /zroot/bhyve directory:

# bhyve -AHP -s 0:0,hostbridge -s 31:0,lpc \
-s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,./disk0.img \
-s 4:0,ahci-cd,./Win11_23H2_English_x64v2.iso -c 4 -m 1024M \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1920,h=1080,wait \
-s 30,xhci,tablet \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
windows11

I believe what I'm doing above is calling bhyve with various flags to talk across the bridge on tap0, block interface to disk image from iso on Windows with 4 cores and 1GB of ram and a framebuff that is 1920x1080 that can be connected at tcp port 5900... the UEFI boot interface is required for gfx interfaces OS like Windows.

Paul
 
Thank you both. I'll try the graphical for now but I really want to know the ins and outs of Bhyve. :)

I'm a sysadmin by trade and I like knowing what's under-the-covers. :D

I also wouldn't mind creating a document going forward (aka hand-holding HOWTO). I know we're usually of this type of people but this project should have taken off faster and farther than it has in 30+ years. I believe braindead HOWTOs and GUIs would help that.
 
algi.... definitely a large step forward.

# If you want to pull a graphical console, you'll need the UEFI loader,
# no matter what OS you're installing on the guest.

loader="uefi"
graphics="yes"
xhci_mouse="yes"

# If not specified, cpu=n will give the guest n discrete CPU sockets.
# This is generally OK for Linux or BSD guests, but Windows throws a fit
# due to licensing issues, so we specify CPU topology manually here.

cpu=2
cpu_sockets=1
cpu_cores=2

# Remember, a guest doesn’t need extra RAM for filesystem caching--
# the host handles that for it. 4G is ludicrously low for Windows on hardware,
# but it’s generally more than sufficient for a guest.
memory=4G

# put up to 8 disks on a single ahci controller. This avoids the creation of
# a new “controller” on a new “PCIe slot” for each drive added to the guest.

ahci_device_limit="8"

# e1000 works out-of-the-box, but virtio-net performs better. Virtio support
# is built in on FreeBSD and Linux guests, but Windows guests will need
# to have virtio drivers manually installed.

#network0_type="e1000"

network0_type="virtio-net"
network0_switch="public"

# bhyve/nvme storage is considerably faster than bhyve/virtio-blk
# storage in my testing, on Windows, Linux, and FreeBSD guests alike.

disk0_type="nvme"
disk0_name="disk0.img"

# This gives the guest a virtual "optical" drive. Specifying disk1_dev=”custom”
# allows us to provide a full path to the ISO.

disk1_type="ahci-cd"
disk1_dev="custom"
disk1_name="/zroot/bhyve/virtio-win.iso"

# windows expects the host to expose localtime by default, not UTC

utctime="no"

passthru0="4/0/0"
 
jbo@ said:
I know I'm not addressing the question of "please give me a hand-holding guide" - but what I can say is that I have no problems with Windows 11 under bhyve. I have several Windows 11 bhyve VMs that I use daily and it works pretty well.

I wouldn't recommend using VNC, I usually use RDP. Definitely always with Windows clients but all of my graphical Linux bhyve VMs are using RDP too.

If you're not comfortable managing your VMs manually, I can recommend having a look at sysutils/vm-bhyve.
Click to expand...
Have you installed them from scratch or were they P2V (physical to virtual) migrations? I'm considering a P2V migration.
 
I have official ISOs (plural) as well but the pain of reinstalling everything is putting me off. It's currently a separate partition on my laptop. I want to move it to a zvol and later reclaim the space for an additional FreeBSD image.

I did the P2V thing with XP from a physical partition to VirtualBox back in the day. That required a binary edit of the XP boot record and two XP O/S files. The edits removed the disk geometry from the boot record and the two files so XP didn't expect certain data to be at specific addresses on disk. I don't know if Windows 10 or 11 will work. My first attempt at Windows 10 P2V failed to boot.
 
HI all!!!

So, going back to my template file that I pasted above.... I did the following before running it from scottro's page:

vm_enable="YES"
vm_dir="zfs:zroot/bhyve

I copied my template to .templates/windows11.conf

vm create -t windows11 -s 256G windows11

I ran 'vm list' and it shows me:

NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE
windows11 default uefi 2 4G - No Stopped

I thought that seemed ok. I then decided (still in zroot/bhyve) to install my ISO.

vm install windows11 ./Win11_23H2_English_x64v2.iso

It came back with:

Starting windows11
* found gues in /zroot/bhyve/windows11
! guest appears to be running already

I ran 'vm list' and it shows me:

NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE
windows11 default uefi 2 4G - No Stopped

Any ideas all? (If I need to 're-initialize', and try again, how do I?) How does the template file look?
 
I had some trouble running my W10 but now it works fine. I tried first with vm-bhyve but I found out that it was MUCH easier (to me, personal preference) using bhyve directly, so I simply use a script.

I followed the handbook, some threads here and tada 😄.
The logic is very KISS :
Enable bhyve, ifconfig to bridge your physical LAN/WAN to a virtual interface, truncate a hard drive file if it does not exist (zfs or in any local or remote location), run a single bhyve command according to the host needs (for Windows, uefi fd firmware file, 4GB or more of RAM, lpc to 31, keyboard layout if you use a non US keyboard, and virtio drivers for drive/network controller, don't play with sockets because Microsoft doesn't like running multi-socket if you didn't pay for), connect to it using vncviewer then destroy the VM when you turned it off.

PS : Don't forget your firewall if you use one, pf was blocking vnc in my case.
 
Did you, in your config file, have
bhyve_options="-A"
(I usually put at the end, of the file, don't know if that matters).
I believe you are going to need that line, and I don't see it in your post above, unless I overlooked it.
 
scottro said:
Did you, in your config file, have
bhyve_options="-A"
(I usually put at the end, of the file, don't know if that matters).
I believe you are going to need that line, and I don't see it in your post above, unless I overlooked it.
Click to expand...
For me yes, but I pass it as a command argument like the handbook does :
bhyve -AHP (rest of the command).
I'm not on my computer now so I can't send you the full command...

Edit : here it is (I use variables; I'm not expert so it can be greatly enhanced ; run as root)
Code: 
bhyve -AHP \
-s 0:0,hostbridge \
-s 1:0,virtio-net,tap0 \
-s 2:0,ahci-cd,$CD1 \
-s 3:0,ahci-cd,$CD2 \
-s 4:0,virtio-blk,$HD \
-c 4 -m 4096M \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1600,h=900,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,$UEFI \
$VM
My variables point all in one directory on my NAS, containing every required file (the ISOs, the disk IMG, and the UEFI firmware FD file), no ZFS used.
I use 2 CD drives to run both Windows ISO for install and VirtIO ISO for drivers (required to install the SATA drive, as Windows setup will not find any fixed drive + network driver once Windows is setup).
This setup only requires on FreeBSD for vmm to be enabled (vm_path is useless) and the network bridge, if required, and vncviewer. vm-bhyve is NOT used. As I said, some people prefer this way, some prefer vm-bhyve.
On my side, I'll have to add -K to add a French keyboard layout. If you're not using US keyboard, you'll have to do the same because you'll get QWERTY keyboard even if region settings are correct on your FreeBSD and in Windows.

To access the vm, simply run as root or regular user
Code: 
vncviewer 0:5900
(watch for firewalls !)
To check if something is running, you can test
Code: 
sysctl kern.vm_guest
and
Code: 
ls -al /dev/vmm
Once you stopped the VM, run
Code: 
bhyvectl --destroy --vm=$VM
. To restart the VM, just run once again the bhyve -AHP etc...
 
Hello, My system is FreeBSD 14.0-RELEASE-p6, I got
Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz (2000.29-MHz K8-class CPU)
and VT-x: HLT,PAUSE VT-x, enabled by modified the bios.

It was enabled before upgrading the bios to latest. The lasted bios does not have the option and vt is disable by default. Somehow I managed to flash a mod bios to re-enable the VT-x. I also use symcmos tool to modify the register of bios setting from 000 to 001 which enabled the VT.

My dmesg shows VT-x: HLT,PAUSE
I can run virtualbox with no problem.

However when I try to run vm-bhyve,
sh: 
service vm start
. I got
sh: 
/usr/local/sbin/vm: ERROR: kernel vmm not initialised (no VT-x / AMD SVM cpu support?)

I put vm_disable_host_checks="YES" in rc.conf
vm starts however whenever I run vm command it give me error like this
sh: 
/usr/local/sbin/vm: ERROR: unable to load vmm.ko!
I tired sudo kldunload vmm.
I manged to this step
sh: 
sudo vm install void void-live-x86_64-20240314-base.iso
Starting void
  * found guest in /vm/void
  * booting...
sh: 
sudo vm list
NAME  DATASTORE  LOADER  CPU  MEMORY  VNC  AUTO  STATE
void  default    uefi    2    1G      -    No    Stopped
vm is not running. I guest there is something wrong with the VT-x I could not run vm-BHYVE. I follow this https://vermaden.wordpress.com/2023/08/18/freebsd-bhyve-virtualization/
and this https://srobb.net/vm-bhyve.html and my vm is not running I could not connect to it using vnc viewer

Any ideas? Thanks
 
You must log in or register to reply here.
Back
Top