bhyve Current state of bhyve Nvidia passthrough?

rem06bor · Dec 27, 2023

I have two nvidia cards, 1080ti and tesla p100. They worked somehow on 13.2, I was able to passthru these cards to VM with ubuntu and nvidia drivers. In my case the system with 1080ti and p100 on 13.2 is not stable, sometimes happens that whole system freez, or cpu soft lockup occures for infinite time inside VM but it works.

After upgrading to 14.0 (using exactly the same VM, the same drivers, the same bhyve configuration) on VM dmesg shows:

[ 77.208984] NVRM: Can't find an IRQ for your NVIDIA card!
[ 77.212697] NVRM: Please check your BIOS settings.
[ 77.212699] NVRM: [Plug & Play OS] should be set to NO
[ 77.212700] NVRM: [Assign IRQ to VGA] should be set to YES
[ 77.212702] nvidia: probe of 0000:00:07.0 failed with error -1
[ 77.212733] NVRM: The NVIDIA probe routine failed for 1 device(s).
[ 77.212734] NVRM: None of the NVIDIA devices were initialized.
[ 77.213269] nvidia-nvlink: Unregistered Nvlink Core, major device number 237

and repeating system messages:

[ 206.437547] NVRM: Can't find an IRQ for your NVIDIA card!
[ 207.439361] NVRM: Can't find an IRQ for your NVIDIA card!
[ 208.455424] NVRM: Can't find an IRQ for your NVIDIA card!
[ 209.451508] NVRM: Can't find an IRQ for your NVIDIA card!
[ 210.464152] NVRM: Can't find an IRQ for your NVIDIA card!
[ 211.475571] NVRM: Can't find an IRQ for your NVIDIA card!

I can reproduce this on two different computers (both based on Intel) and two different Nvidia cards (1080ti, p100).
I am looking for any hint.

roswellian · Dec 27, 2023

I've been successfully utilizing Corvin's Nvidia passthrough code since version 13.0, specifically with my Quadro K2200 card and Linux (Debian) as the guest OS. Remarkably, I haven't encountered any issues yet. Detailed steps were previously shared by someone in the forum. Essentially, the process involves fetching the codes from https://github.com/Beckhoff/freebsd-src/tree/phab/corvink/14.0/nvidia-wip for FreeBSD 14.0 compatibility. If targeting version 13.1, simply locate the 13.1/nvidia-wip branch. Then, compile the source code following the instructions outlined in the FreeBSD manual. Finally, proceed with standard passthrough steps as outlined in the bhyve manual. Following these steps should result in a functional Nvidia GPU setup!

rem06bor · Dec 28, 2023

The code for 14.0 with Corvin patches seems a little bit broken. I needed to do some minor fixes to compile it:

Diff:

diff --git a/usr.sbin/bhyve/pci_passthru.c b/usr.sbin/bhyve/pci_passthru.c
index 379b1255a79f..f74adfa2f84b 100644
--- a/usr.sbin/bhyve/pci_passthru.c
+++ b/usr.sbin/bhyve/pci_passthru.c
@@ -1171,7 +1171,7 @@ passthru_write(struct pci_devinst *pi, int baridx, uint64_t offset, int size,
        } else if (baridx == 0 &&
            pci_get_cfgdata16(pi, PCIR_VENDOR) == PCI_VENDOR_NVIDIA &&
            pci_get_cfgdata8(pi, PCIR_CLASS) == PCIC_DISPLAY) {
-               passthru_cfgwrite(ctx, vcpu, pi, offset - 0x88000, size, value);
+               passthru_cfgwrite(pi, offset - 0x88000, size, value);
        } else {
                assert(pi->pi_bar[baridx].type == PCIBAR_IO);
                assert(size == 1 || size == 2 || size == 4);
@@ -1206,7 +1206,7 @@ passthru_read(struct pci_devinst *pi, int baridx, uint64_t offset, int size)
                /* dummy read to MMIO because hw might depend on it */
                memcpy(&val, nvidia_bar0 + offset, size);
 
-               passthru_cfgread(ctx, vcpu, pi, offset - 0x88000, size, (uint32_t *)&val);
+               passthru_cfgread(pi, offset - 0x88000, size, (uint32_t *)&val);
        } else {
                assert(pi->pi_bar[baridx].type == PCIBAR_IO);
                assert(size == 1 || size == 2 || size == 4);
@@ -1299,14 +1299,14 @@ passthru_mmio_addr(struct pci_devinst *pi, int baridx, int enabled,
                    hpa, gpa, len,
                    enabled ? "map" : "unmap");
                if (!enabled) {
-                       if (vm_unmap_pptdev_mmio(ctx, sc->psc_sel.pc_bus,
+                       if (vm_unmap_pptdev_mmio(pi->pi_vmctx, sc->psc_sel.pc_bus,
                                sc->psc_sel.pc_dev, sc->psc_sel.pc_func, gpa,
                                len) != 0) {
                                warnx(
                                    "pci_passthru: vm_unmap_pptdev_mmio nvidia low failed");
                        }
                } else {
-                       if (vm_map_pptdev_mmio(ctx, sc->psc_sel.pc_bus,
+                       if (vm_map_pptdev_mmio(pi->pi_vmctx, sc->psc_sel.pc_bus,
                                sc->psc_sel.pc_dev, sc->psc_sel.pc_func, gpa,
                                len, hpa) != 0) {
                                warnx(
@@ -1324,14 +1324,14 @@ passthru_mmio_addr(struct pci_devinst *pi, int baridx, int enabled,
                    hpa, gpa, len,
                    enabled ? "map" : "unmap");
                if (!enabled) {
-                       if (vm_unmap_pptdev_mmio(ctx, sc->psc_sel.pc_bus,
+                       if (vm_unmap_pptdev_mmio(pi->pi_vmctx, sc->psc_sel.pc_bus,
                                sc->psc_sel.pc_dev, sc->psc_sel.pc_func, gpa,
                                len) != 0) {
                                warnx(
                                    "pci_passthru: vm_unmap_pptdev_mmio nvidia low failed");
                        }
                } else {
-                       if (vm_map_pptdev_mmio(ctx, sc->psc_sel.pc_bus,
+                       if (vm_map_pptdev_mmio(pi->pi_vmctx, sc->psc_sel.pc_bus,
                                sc->psc_sel.pc_dev, sc->psc_sel.pc_func, gpa,
                                len, hpa) != 0) {
                                warnx(
(END)

I installed the new kernel and bhyve (only this one file, not the whole world). Now nvidia 1080ti backed to life on 14.0. My guest is ubuntu 22.04 with driver 535.129.03.
I really hope it will be more stable than on 13.2 (without patches) but it needs to do more long term tests.

sfourman · Jan 2, 2024

Thank you to everyone involved in developing and testing this work. It is very much needed and appreciated.
My question is: does anyone have a single patch that can be applied to 14.0/STABLE?
I've been a bit disconnected therefore am wondering why (if something of this magnitude exists) it never made it into FreeBSD 14? Thank you.

rem06bor · Jan 12, 2024

As far as I know there were more patches, and most of them have been applied to 14.0.
For now there are two patches indeed. One is for bhyve (it refers to "dummy bar 0" issue) and the second one is a kernel patch that changes hypervisor signature from bhyve to kvm signature.

I do not know much about "bar 0" and gpu, but for obvious reason you cannot change hypervisor signature permanently. There is a suspicious that nvidia drivers can work differently for different hypervisor.

I tested 1080ti a little more, and it seems to work stable. There is only a problem after restarting VM. That means if I want to restart VM with gpu attached to it, I need to restart the hole computer otherwise the gpu not work.

syllopsium · Feb 21, 2024

Thanks for the patch - but you can easily see why it didn't make it into 14.0. With it applied although Linux passthrough now works, attempting another VM start afterwards almost always results in a hard lock of the entire system at least here (Ivy Bridge, C602 chipset).

The hypervisor signature change isn't required, only the modifications in the bhyve directory. I don't remember it being this unstable in some pre 14.0 builds. It's also currently only Linux that's affected, without the patch Windows and FreeBSD VMs are quite happy to switch between each other, at least in the case of an Nvidia CPU with proprietary (Windows) and open source (FreeBSD) drivers on my system.

Unfortunately Linux passthrough is what I need, because I'm trying to compare FreeBSD WINE to Linux WINE, as I suspect Linux WINE is more compatible than FreeBSD (even for the same version).

First_Law_of_Unix · Aug 24, 2024

I got Ubuntu 24.04 working on FreeBSD 14.2 via Bhyve and I'm amazed how it consumes less 1% CPU resources while idling.
Anyone knows if "RTX 3080 Ti" GPU is supported for Bhyve GPU Pass through, where guest is Ubuntu?

Thanks.

Lincas · Aug 24, 2024

First_Law_of_Unix said:
I got Ubuntu 24.04 working on FreeBSD 14.2 via Bhyve and I'm amazed how it consumes less 1% CPU resources while idling.
Anyone knows if "RTX 3080 Ti" GPU is supported for Bhyve GPU Pass through, where guest is Ubuntu?

Thanks.

It is possible https://dflund.se/~getz/Notes/2024/freebsd-gpu/ this is for 14.0 but it is working on 14.1

First_Law_of_Unix · Aug 25, 2024

Lincas said:
It is possible https://dflund.se/~getz/Notes/2024/freebsd-gpu/ this is for 14.0 but it is working on 14.1

Thanks.
This is great. Will delve into it.

Lincas · Aug 25, 2024

**That's how this workaround should look like:**

At this moment I do not think 15-CURRENT will work because Corvin did some updates and it is totally not working unless he fixed it.

A and B steps were enough to run Windows 10 with an Nvidia GPU for me. 11 hangs with more than one CPU. If you later only need to update the system, it may be enough to run A, B, and G.

A.1) If you do not have the source, you can clone:

Code:

git clone -b releng/14.1 https://git.freebsd.org/src.git /usr/src
cd /usr/src

B.1) Next, edit the file sys/amd64/vmm/x86.c and find the following line:

Code:

static const char bhyve_id[12] = "Bhyve Bhyve ";

B.2) Change it to:

Code:

static const char bhyve_id[12] = "KVMKVMKVM\0\0\0";

B.3) Then, build and install the kernel:

Code:

make -j$(nproc) buildkernel
make -j$(nproc) installkernel

C.1) Now let’s build world:

Code:

make -j$(nproc) buildworld

D.1) Get Corvin’s patches:

Code:

git clone --depth 1 --branch phab/corvink/14.0/nvidia-wip https://github.com/Beckhoff/freebsd-src /usr/src-nvidia

E.1) We need to change the buildworld object directory:

Code:

export MAKEOBJDIRPREFIX=/usr/obj-nvidia

E.2) Now we need to install includes and make the rest:

Code:

cd /usr/src-nvidia
cd include
make -j$(nproc)
make install 
cd -
cd lib/libvmmapi
make -j$(nproc)
cd - 
cd sys/modules/vmm
make -j$(nproc)
cd - 
cd usr.sbin/bhyve
make -j$(nproc)
cd - 
cd usr.sbin/bhyvectl
make -j$(nproc)
cd - 
cd usr.sbin/bhyveload
make -j$(nproc)
cd -

F.1) For step F, you need to reboot the machine and in Boot Loader select the second variant 'Single User'. After the kernel boots, mount the disks:

For UFS:

Code:

mount -a

For ZFS:

Code:

zfs set readonly=off zroot
zfs mount -a

F.2) Let’s install World:

Code:

cd /usr/src
etcupdate -p
make -j$(nproc) installworld
etcupdate -B

F.3) Delete old stuff that's not needed anymore:

Code:

make delete-old
make delete-old-libs

Or if you know what you are doing:

Code:

yes | make delete-old
yes | make delete-old-libs

G.1) At the end, let’s install Corvin’s patched Bhyve:

Code:

cd /usr/src-nvidia
export MAKEOBJDIRPREFIX=/usr/obj-nvidia
cd lib/libvmmapi
make install
cd -
cd sys/modules/vmm
make install 
cd -
cd usr.sbin/bhyve
make install 
cd -
cd usr.sbin/bhyvectl
make install 
cd -
cd usr.sbin/bhyveload
make install

drookie · Oct 1, 2024

Guys, you do understand that all these how-tos suggest you install older kernel on a more recent world, right ? Like 14/0 on a 14/1 or 15. Just to clear things out.

c-c-c-c · Nov 3, 2024

Lincas said:
A.1) If you do not have the source, you can clone:
git clone -b releng/14.1 https://git.freebsd.org/src.git /usr/src cd /usr/src
B.1) Next, edit the file sys/amd64/vmm/x86.c and find the following line:
static const char bhyve_id[12] = "Bhyve Bhyve ";
B.2) Change it to:
static const char bhyve_id[12] = "KVMKVMKVM\0\0\0";
B.3) Then, build and install the kernel:
make -j$(nproc) buildkernel make -j$(nproc) installkernel

This is ok.

Lincas said:
D.1) Get Corvin’s patches:
git clone --depth 1 --branch phab/corvink/14.0/nvidia-wip https://github.com/Beckhoff/freebsd-src /usr/src-nvidia
E.1) We need to change the buildworld object directory:
export MAKEOBJDIRPREFIX=/usr/obj-nvidia
E.2) Now we need to install includes and make the rest:
cd /usr/src-nvidia cd include make -j$(nproc) make install cd - cd lib/libvmmapi make -j$(nproc) cd - cd sys/modules/vmm make -j$(nproc) cd - cd usr.sbin/bhyve make -j$(nproc) cd - cd usr.sbin/bhyvectl make -j$(nproc) cd - cd usr.sbin/bhyveload make -j$(nproc) cd -

Now you're installing code from 14.0 onto a 14.1 system, don't do that.

Lincas said:
F.1) For step F, you need to reboot the machine and in Boot Loader select the second variant 'Single User'. After the kernel boots, mount the disks:

For UFS:
mount -a
For ZFS:
zfs set readonly=off zroot zfs mount -a
F.2) Let’s install World:
cd /usr/src etcupdate -p make -j$(nproc) installworld etcupdate -B
F.3) Delete old stuff that's not needed anymore:
make delete-old make delete-old-libs
Or if you know what you are doing:
yes | make delete-old yes | make delete-old-libs

No reason to do this.

Lincas said:
G.1) At the end, let’s install Corvin’s patched Bhyve:
cd /usr/src-nvidia export MAKEOBJDIRPREFIX=/usr/obj-nvidia cd lib/libvmmapi make install cd - cd sys/modules/vmm make install cd - cd usr.sbin/bhyve make install cd - cd usr.sbin/bhyvectl make install cd - cd usr.sbin/bhyveload make install

This is identical to E.2 which you shouldn't be doing.

Instead take the 14.1 branch and apply the patches, it's ill advised to mix between minor versions like this even if it might work as the the ABI is stable between minor versions. You're missing out on some other improvements this way.
I'll attach a .patch file that you can apply to 14.1 later to avoid the mess.
git apply --verbose nvidia.patch

c-c-c-c · Nov 7, 2024

c-c-c-c said:
Instead take the 14.1 branch and apply the patches, it's ill advised to mix between minor versions like this even if it might work as the the ABI is stable between minor versions. You're missing out on some other improvements this way.
I'll attach a .patch file that you can apply to 14.1 later to avoid the mess.
git apply --verbose nvidia.patch

I forgot to rename read_config to pci_host_read_config, I'm not able to edit the previous post so here is a complete patch file to apply the same way. Just tested building 14.1-RELEASE-p6 and nvidia gpu is detected in bhyve.
The approach previously described really messes things up as 14.0 is no longer supported by ports and you'll get funny errors.

So get the patch, clone 14.1-release branch, apply patch, build and install kernel then build/install /include, vmm, bhyve, bhyvectl, bhyveload.

GlitchyDot · Jan 11, 2025

c-c-c-c said:
I forgot to rename read_config to pci_host_read_config, I'm not able to edit the previous post so here is a complete patch file to apply the same way. Just tested building 14.1-RELEASE-p6 and nvidia gpu is detected in bhyve.
The approach previously described really messes things up as 14.0 is no longer supported by ports and you'll get funny errors.

So get the patch, clone 14.1-release branch, apply patch, build and install kernel then build/install /include, vmm, bhyve, bhyvectl, bhyveload.

where i have to put this patch before i apply it ? i wanna test on 14.2-STABLE

bileslav · Jan 11, 2025

GlitchyDot said:
where i have to put this patch before i apply it ? i wanna test on 14.2-STABLE

Here are Corvin's patches for 14.0, unfortunately never updated for 14.1 and 14.2.

Like a month ago, I had successfully rebased the branch for 14.2-RELEASE by myself. It took me a few hours to understand how to do it. Not very difficult, but not easy either. I thought about publishing the repository, but alas, never did. However, I still have the resulting bhyve binary. This is the only file that the patches affect. It's embarrassing to share some privileged binary, but still. It should allow the NVIDIA driver to initialize in Linux VMs. Windows 10 and FreeBSD VMs works without the patches. Windows 11 is just broken.

https://drive.google.com/file/d/1ddMHHRbMxa-Q_oCIYVf9vYJVgtre57Ff/view?usp=drive_link

c-c-c-c · Jan 14, 2025

GlitchyDot said:
where i have to put this patch before i apply it ? i wanna test on 14.2-STABLE

put it in freebsd-src then run

Code:

git apply --verbose nvidia.patch

bileslav said:
Here are Corvin's patches for 14.0, unfortunately never updated for 14.1 and 14.2.

Like a month ago, I had successfully rebased the branch for 14.2-RELEASE by myself. It took me a few hours to understand how to do it. Not very difficult, but not easy either. I thought about publishing the repository, but alas, never did. However, I still have the resulting bhyve binary. This is the only file that the patches affect. It's embarrassing to share some superuser binary, but still. It should allow the NVIDIA driver to initialize in Linux VMs. Windows 10 and FreeBSD VMs works without the patches. Windows 11 is just broken.

https://drive.google.com/file/d/1ddMHHRbMxa-Q_oCIYVf9vYJVgtre57Ff/view?usp=drive_link

you'd likely need vmm.ko aswell, and downloading random binaries off a forum is a bad idea. Just rebuild it, just takes a few minutes.

ZioMario · Jan 14, 2025

bileslav said:
Windows 11 is just broken.

Never understood why it is not broken for me...

bileslav · Jan 15, 2025

c-c-c-c said:
you'd likely need vmm.ko aswell

I've just verified that /usr/sbin/bhyve is enough to fix the error:

Code:

[    2.260202] nvidia-nvlink: Nvlink Core is being initialized, major device number 238
[    2.260207] NVRM: Can't find an IRQ for your NVIDIA card!
[    2.260780] NVRM: Please check your BIOS settings.
[    2.260782] NVRM: [Plug & Play OS] should be set to NO
[    2.260783] NVRM: [Assign IRQ to VGA] should be set to YES
[    2.260784] nvidia 0000:00:0a.0: probe with driver nvidia failed with error -1
[    2.260808] NVRM: The NVIDIA probe routine failed for 1 device(s).
[    2.260809] NVRM: None of the NVIDIA devices were initialized.
[    2.260985] nvidia-nvlink: Unregistered Nvlink Core, major device number 238

When I built the patches, I ran diff on every produced file, and the only different file was the executable. I don't remember anything about vmm.ko specifically.

This is how I did it:

sh:

git clone https://github.com/beckhoff/freebsd-src /usr/src
cd /usr/src
*do the work on origin/phab/corvink/14.0/nvidia-wip for 14.2*
cd /usr/src/usr.sbin/bhyve
make

ZioMario said:
Never understood why it is not broken for me...

So far, you're the only person who reported that it works. I'd also like to know why. It seems no one currently works on that problem.

ZioMario · Jan 16, 2025

Can you write a tutorial for us where you explain how to rebase the corvin's patches for 14.2 ? And since it seems that Corvin will not rebase them anymore,we should hope that they will work even for 15.x ; anyway,even that your method worked even for 15.x. Thanks.

c-c-c-c · Jan 16, 2025

ZioMario said:
Can you write a tutorial for us where you explain how to rebase the corvin's patches for 14.2 ? And since it seems that Corvin will not rebase them anymore,we should hope that they will work even for 15.x ; anyway,even that your method worked even for 15.x. Thanks.

That is available if you scroll up in this thread.

bileslav said:
When I built the patches, I ran diff on every produced file, and the only different file was the executable. I don't remember anything about vmm.ko specifically.

You wont change the hypervisor identifier by just changing bhyve as it resides in
/sys/amd64/vmm/x86.c

ZioMario · Jan 16, 2025

c-c-c-c said:
That is available if you scroll up in this thread.

You wont change the hypervisor identifier by just changing bhyve as it resides in
/sys/amd64/vmm/x86.c

I have a question to ask you. If I want to passthru my nvidia Geforce RTX 2080 ti to a Windows vm,I need to change the string inside the file x86.c ; let's say that I don't want to change that string AND I want to passthru the nvidia gpu inside a Linux vm. It will work ? Can the GPU be passed inside a Linux vm using the original "bhyve" string and not the "KVM" string ?

I'm asking this because at the moment I'm still using two different executables (bhyve-win & bhyve-lin : depending on the operating system I want as a guest) and the same vmm.ko module (with the KVM string embedded inside),but if I want to passhtru my GPU to a Windows vm using the executable bhyve-lin (patched by Corvin),it will not work ; Windows will report Error 43,showing the same behaviour that it showed if the string hadn't been applied). Never understood why.

c-c-c-c · Jan 16, 2025

ZioMario said:
I want to passthru my nvidia Geforce RTX 2080 ti to a Windows vm,I need to change the string inside the file x86.c

Correct.

ZioMario said:
let's say that I don't want to change that string

That's odd, you need to change it if you want the nvidia driver to function properly.

ZioMario said:
Can the GPU be passed inside a Linux vm using the original "bhyve" string and not the "KVM" string ?

Yes but the driver wont work so the GPU wont be doing a whole lot. Other pci devices are able to be passed through without this issue.

ZioMario said:
I'm asking this because at the moment I'm still using two different executables (bhyve-win & bhyve-lin : depending on the operating system I want as a guest) and the same vmm.ko module (with the KVM string embedded inside),but if I want to passhtru my GPU to a Windows vm using the executable bhyve-lin (patched by Corvin),it will not work ; Windows will report Error 43,showing the same behaviour that it showed if the string hadn't been applied). Never understood why.

Sorry, I have no idea what you're trying to say here. Does windows not work with the bhyve identifier as "KVMKVMKVM"?

But please let me reiterate, if you want to patch 14.1 or above then use the patch file I attached above and apply it the way i described above. Might need changes if function names or line numbers change in the future but it shouldn't be difficult at all.

bileslav · Jan 16, 2025

c-c-c-c said:
You wont change the hypervisor identifier by just changing bhyve as it resides in
/sys/amd64/vmm/x86.c

Well, then I just don't have to mess with the identifier at all to make the NVIDIA driver happy on Linux VMs (as well as in every other case).

ZioMario said:
Can you write a tutorial for us where you explain how to rebase the corvin's patches for 14.2 ?

It involves interactive work while resolving a few merge conflicts using Git. Although the task isn't difficult overall, I won't write a detailed post about it. You know, I'm not a Git pro myself; I did a lot of googling along the way.

ZioMario said:
it seems that Corvin will not rebase them anymore

I hope this isn't the case. One of us should email him. I still have other things to do before I get around to it.

ZioMario · Jan 17, 2025

c-c-c-c said:
Does windows not work with the bhyve identifier as "KVMKVMKVM"?

I use bhyve-win to pass the gpu inside Windows. It doesn't have the Corvins patches inside. It works for the Windows vms,not for the Linux vms.
I use bhyve-lin to pass the gpu inside Linux. It does have the Corvin patches inside. It works for the Linux vms,not for the Windows vms.
vmm.ko has the string inside.
if I want to use bhyve-win for a Linux vm it does not work because inside it there aren't the Corvins patches.
If I want to use bhyve-lin for a Windows vm it does not work : Windows reports error 43. Don't know why.
I'm the only user that should use two different bhyve executables for passing his gpu,depending the OS to virtualize,if it is Linux or Windows.

ZioMario · Jan 17, 2025

bileslav : can u guess why bhyve-lin can't pass the GPU inside a Windows vm ?

bhyve Current state of bhyve Nvidia passthrough?

rem06bor

roswellian

rem06bor

sfourman

rem06bor

Attachments

syllopsium

First_Law_of_Unix

Lincas

First_Law_of_Unix

Lincas

drookie

c-c-c-c

Attachments

c-c-c-c

Attachments

GlitchyDot

bileslav

c-c-c-c

ZioMario

bileslav

ZioMario

c-c-c-c

ZioMario

c-c-c-c

bileslav

ZioMario

ZioMario