Bhyve + Windows 10 + passthough + ... = tips & tricks

F

free-and-bsd

OK, it seems now time we wrote a bhyve + Win10 + ... HOTWO :). So far it can include:

1) This patch providing improved performance of virtio-blk disk for Windows installation (adopted already, just use the functionality).
2) That other bhyve patch (not yet adopted) against multple-bar PCIe cards passthrough causing Win 10 guest boot failure
3) Yet another patch against bhyve freezing with passthough enabled on some motherboards (not yet adopted, it seems).
4) Switching the existing installation from ahci-hd disk image to virtio-blk virtual controller (you learned about this advanced feature too late and now wonder if it is possible to switch over).
5) And yes, my last discovery: about VIA VL805 based USB 3.0 host controller being NOT recommended for passthrough (causes all sorts of weird things to happen!). Somehow I have a kind of history of having problems with VIA hardware...
6) OK, what else?... EDIT: some optimal bhyve command line options to use with Win 10 guest.
7) How to connect? net/freerdp connection parameters that work well for me.
8) Virtual NAT/bridge networking for guest. I found online sources mostly instructing one to use "bridged" type of virtual networking for guest. But whoever for some reason doesn't like the idea of bridging his real NIC with the virtual one will want NATed network. I didn't find precise and exact instructions to that effect, though. But the setup is actually VERY simple.

In my next posts I will explain each portion in more detail...

EDIT: I guess, by the time this HOWTO takes its shape, the patches above won't be needed as they seem to be gradually adopted into STABLE. Including, BTW, the HDA sound patch.
 
Last edited:
Ok, this will not be a HOTWO as in step-by-step instruction. Rather, these notes are meant for those who have read online guides for starters. This will be a collection of practical advice regarding some problems one may encounter and solutions that have proved to work.
Maybe it would be logical, then, to start with what's needed to make Windows 10 guest experience be smooth enough. Apart from PCI passthrough, which I'll discuss later on, the following steps will help much:

(1) We want to use Win 10 PRO to be able to use RDP connection to virtual machine, that will be faster than VNC.​
(2) Going further about speed, it will be advisable to use not ahci-hd, but virtio-blk type of disk adapter for the installation. No need for patching here as all the patches needed for that to work are already in 12.1.​
NB: don't forget to include virtio-blk ISO into your bhyve config. It will be needed not only for virtio-blk, but also for vritio-net, e.g.:​
Code: 
-s 6,ahci-cd,/path/to/virtio-win-0.X.XXX.iso \
(3) Use a reasonable CPUs command line option to bhyve. If we just use bhyve -c 4 ..., it will not understand that we actually meant a 2-core x 2-thread single processor. Rather, it will add 4 processors, which Windows will not be able to use -- or so I gather from personal experience.​
​
Therefore, we need to use sockets,cores,threads:
Code: 
bhyve -c sockets=1,cores=2,threads=2 -m 6G...
At least, when I switched to this syntax from just "-c 4", performance immediately showed I guessed right. BTW, VMPlayer doesn't seem to offer this functionality.

(4) And finally, snd_hda patch doesn't seem to be needed that much. Because both output and input sound is easily added via net/freerdp command line.​
EDIT: however, the patch with additional adjustments seems to be included nevertheless. As is mentioned here, two very small patches make it work in STABLE. Though for some reason I don't hear any sound on my host. But Win 10 guest shows (in VNC) working sound devices with volume levels showing how it really works... will look into it later. It is reported to work if added to bhyve command line like this then:​
Code: 
-s 6,hda,play=/dev/dsp,rec=/dev/dsp
 
Last edited:
Now, as mentioned in my item (8) of the OP, while preparing a bhyve guest one prepares some networking. Without VT-d & passthrough it will be virtual networking.

As I mentioned, all the online guides available assume you need bridged type of networking and explain how to implement it. Good!
But whoever wants to implement NAT (separate virtual network) can do it by simply adding these line to /etc/rc.conf:
Code: 
cloned_interfaces="bridge0 tap0"
ifconfig_bridge0="inet 192.168.7.1/24 addm tap0"
Almost done! But for that to work, one needs to load the needed modules at boot time. As I boot with GRUB2, I use this in my /boot/grub/grub.cfg:
Code: 
kfreebsd_module_elf /ROOT/freebsd-12.0/@/boot/kernel/if_bridge.ko
kfreebsd_module_elf /ROOT/freebsd-12.0/@/boot/kernel/bridgestp.ko
kfreebsd_module_elf /ROOT/freebsd-12.0/@/boot/kernel/if_tap.ko
But if one uses traditional FreeBSD loader, one will have to add the lines to /boot/loader.conf accordingly:
Code: 
if_bridge_load="YES"
bridgestp_load="YES"
if_tap_load="YES"
This virtual network 192.168.7.0/24 gets active once the guest that uses tap0 interface activates its networking functionality. For that matter, I also added corresponding PF rules to skip on these interfaces. But I think the system complained about non-existent interfaces, as these were not yet active at the time PF rules are loaded...

Lastly, the guides tell you need to add net.link.tap.up_on_open=1 sysctl to your /etc/sysctl.conf. I did it and got complains at boot time about an "unknown sysctl". So I don't remember if that was actually needed for this type of setup.

EDIT: It just dawned on me that using NAT is precisely what caused me slow networking speed on my VM. Because in case you follow the guides and use NIC+tap0 bridge type virtio, you're as good as using your NIC directly. Hence the speed. Which may be another reason why the method above is never mentioned in bhyve guides... So, apart from a purely scientific satisfaction from the fact that virtual NAT is possible, the whole idea turns out to be eh.... not so smart, after all.

Just as the saying goes which says, sometimes the road less traveled is less traveled for a reason :)
 
Last edited:
hello free-and-bsd,

I'm following online guides and your post to install windows 10 on my FreeBSD 12.1 RELEASE.

by now I'm using this for installing using freevnc to connect:

kldload vmm
ifconfig tap0 create
sysctl net.link.tap.up_on_open=1
ifconfig bridge0 create
ifconfig bridge0 addm ue0 addm tap0
ifconfig bridge0 up

this configuration does not works, because the C: drive is not visible in the installation : (using virtio-blk)
bhyve -c sockets=1,cores=2,threads=2 -m 4G -w -H \
-s 0,hostbridge \
-s 3,ahci-cd,/usr/home/scerruti/virtualizing/Win10_1909_EnglishInternational_x64.iso \
-s 4,virtio-blk,/usr/home/scerruti/virtualizing/windows_10.img \
-s 5,virtio-net,tap0 \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1920,h=1080,wait \
-s 30,xhci,tablet \
-s 31,lpc -l com1,stdio \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
vm0

this configuration works instead and C: drive show and I can install windows: (using ahci-hd)

bhyve -c sockets=1,cores=2,threads=2 -m 4G -w -H \
-s 0,hostbridge \
-s 3,ahci-cd,/usr/home/scerruti/virtualizing/Win10_1909_EnglishInternational_x64.iso \
-s 4,ahci-hd,/usr/home/scerruti/virtualizing/windows_10.img \
-s 5,virtio-net,tap0 \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1920,h=1080,wait \
-s 30,xhci,tablet \
-s 31,lpc -l com1,stdio \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
vm0


I haven't used any patch of sort, do you have any clue why the first configuration didn't work out?

BTW I'm using a standard .iso downloaded from microsoft site and a legit W10 pro key.
I tried to use virtualbox but it hanged everytime during installation... and I'm curious about bhyve ^_^
 
Hi there! Because Win 10 doesn't have the drivers for virtio-blk. For that stuff to work you add another ISO image in addition to the windows installation ISO. You need virtio-win driver ISO, can download it from here:
Code: 
...
-s 6,ahci-cd,/path/to/virtio-win-0.X.XXX.iso \
...
BTW, you need that ISO anyway to install your networking driver for virtio-net adapter once Windows installation is ready or you won't have networking.
If you're installing on a virtio-blk device, Windows installation will complain of not being able to see any HDD. You then click "upload driver" or suchlike button there in Win 10 install window and choose your virtio-win ISO (will appear as CDROM), then choose from the folders the needed driver.
NB: don't give it just whole CDROM to scan for drivers because it won't be able to find the right one. Very Windows-like.

I didn't include that because it's well described in online guides. Sorry Included that into my OP :)
 
DusTech said:
BTW I'm using a standard .iso downloaded from microsoft site and a legit W10 pro key.
Click to expand...
Hope you didn't use your key yet :). I'd recommend not to use it until you have finished ALL the tweaks and modifications. Or else it will tell you the system is NOT activated because you changed some hardware detail there. I mean, install, configure, install ALL the updates etc. Only THEN activate.

EDIT: it will tell you your key is already used on another computer and you need a new one. The only fix for that will be a call to Microsoft where they will give you a new activation code after you explain the situation. Thankfully, they never ask for detailed explanations there. But first you'll need to conquer the robot ??
 
Last edited:
hi!

I'm experimenting further.
I did as you said to use the Red Hat driver to detect the hard disk and it worked out, but I wasn't able to use the network during installation... should I load also the virtio-net driver during the installation?
Also I'm running FBSD 12.1 and it should support nvme for the .img, do you think it should provide some benefit?
Also, why aren't you using vm(8)? just asking!
 
"Normal" Windows problem detecting NIC during installation, it happens all the time :confused: in real hardware installations.
DusTech said:
should I load also the virtio-net driver during the installation?
Click to expand...
Windows never asks for that, I think. It just lets you leave networking setup until after the installation. So you keep that CDROM in the VM for the second & third boot -- until you load your desktop session in Windows. Then you install virtio-net driver much the way you install drivers during "normal" Win 10 installation.

BTW, didn't add this to my HOWTO tricks, but I always keep the VNC line
Code: 
-s 29,fbuf,tcp=0.0.0.0:5900,w=1920,h=1080,wait \
in my bhyve startup script -- just in case Windows will decide to do one of its reboot & reconfigure tricks. It costs nothing yet you're able to view the boot process before the NIC is ready. Then you can login through RDP. Or it will drive you mad keeping you waiting until (never know when exactly!) you're able to login via RDP, which you can't until the desktop session is ready.
 
DusTech said:
Also, why aren't you using vm(8)? just asking!
Click to expand...
Because until my VM works the way I want it, there is always need to reboot & restart. Occasionally it won't start after shutdown, so I need to bhyvectl -- destroy. I may be mistaken about the way vm-bhyve works, but it seems to me I save much time using bhyve binary directly.

Like when you're setting up FreeRADIUS or OpenVPN, or any other one requiring fine-tuning. You start/stop from command line until the setup is complete. Only then it's ready to be used as service, which is what vm-bhyve does, unless I'm very much mistaken.
 
Ok thanks...

And I'm here again installing W10 on bhyve...
I' configured with 4 core, 4 threads, 8 GB Ram and virtio-blk for the disk.
I don't know why it's so painfully slow... and my CPU is working like mad.
When I tried NVMe my CPU wasn't so under pressure, but the VM hanged during the installation (if I remember correctly)

1587240474993.png


I'll try to endure through a full installation without network and see how it will run, if it will run.

Btw my CPU is a i5-2500k OC @4500 GHz.
 
That background looks a lot like Windows. And so does that CPU history program...
I hope you're not running bhyve in FreeBSD which, in turn, is installed inside a VM hosted by Windows?
 
free-and-bsd said:
That background looks a lot like Windows. And so does that CPU history program...
I hope you're not running bhyve in FreeBSD which, in turn, is installed inside a VM hosted by Windows?
Click to expand...

:D no I'm not .
it's FreeBSD on metal with Plasma Desktop.
Anyway I think i gave too many resources to the VM... I'll retry with only one core, two threads and 4 GB of RAM...
 
mmm
as it's mentioned here
Only one or two VCPUs should be used during installation but this number can be increased once Windows is installed.
Click to expand...

Indeed it worked out better with only 1 core.
Now I've installed but there is no network connection... I'll figure it out next session :)
 
Just so you know there is fix regarding TPR shadowing that was disabled for non-Xeon CPUs. This has been at least fixed in -CURRENT and probably in -STABLE by now.

I'm watching this thread too as I will also soon be trying to virtualize Win10 in Bhyve as well.
 
DusTech said:
mmm
as it's mentioned here

Indeed it worked out better with only 1 core.
Now I've installed but there is no network connection... I'll figure it out next session :)
Click to expand...
So did you figure that out yet? I'm trying this now on another machine with tap0 device -- it DOES NOT work. Nor with Red Hat 118 driver, nor 139 nor any other driver out there. It's "identifying..." then ending up with "unknown network" with funny addresses. Under FreeBSD host tap0 is shown as `active`, but Windows 10 guest doesn't know what to make of it.
Don't know what to think of it.
 
It only works for me when I use NAT configuration: tap0 is part of bridge0 network. But then one needs to add PF rules to enable traffic between the bridge0 network and the rest of the world.
So, one would argue, the bridged tap0 is the best solution and the simplest... but it doesn't work for me.
 
You must log in or register to reply here.
Back
Top