bhyve Current state of bhyve Nvidia passthrough?

S

syllopsium

From what I can see the current state is Nvidia GPU passthrough is working with patches to at least Linux and also Windows in some cases, probably with patches against 13.1. Has this changed?

Before I start messing with 13.1 with patches, 13.2, or 14.0 or looking at source code trees I thought I'd check if someone has up to date info. I have AMD passthrough working 'fine' on a dual processor Ivy Bridge system, but it's fairly useless due to the fact AMD GPUs don't like to reset after a VM exit.

Ideally looking to get FreeBSD, Linux, and Windows 10 working with passthrough. Currently using a Quadro 6000, but I have a Quadro P4000 arriving soon.
 
To update so far - FreeBSD 14.0 works well with the Quadro P4000 with Linux (Manjaro) and FreeBSD 14.0 CURRENT VMs. Windows next. Didn't have as much luck with the Quadro 6000.

This is possibly BIOS related. The 6000 does not have EFI support, but the P4000 does. Whether a CoreBIOS BIOS instead of a custom EDK2 EFI would change matters is an interesting question.
 
Further update - Windows 10 is also 'fine' provided the Nvidia drivers are installed rather than the ones supplied by Windows update. Windows Update drivers placed the GPU into a state where it wouldn't work with any VM and a devctl reset did not resolve the issue.

However, a cold reboot fixed this and the GPU is now passing between Manjaro, FreeBSD 14.0, and Windows 10 (22H2). 12GB memory allocated to the Windows VM worked fine too.

It looks like FreeBSD and Windows need vnc to be enabled to provide a VGA framebuffer in order for the Nvidia drivers to work. This is not necessary for Manjaro Linux for some reason. All three are now using the closed source Nvidia drivers, so it's not an open source vs closed source issue.

Still, this is really promising. The whole reason for this is to compare against WINE and have VM and passthrough as a fall back, so at least the fall back is now in place.
 
I have read a lot of disparate topics on this forum, but did not quite understand the following. Please someone answer me briefly to simple questions:

(1) Can I passthrough NVIDIA RTX a2000 to Debian Bookworm from FreeBSD 13.2?
(2) Do I understand correctly that in FreeBSD 14.0 I can do this out of the box?
 
dnb said:
(1) Can I passthrough NVIDIA RTX a2000 to Debian Bookworm from FreeBSD 13.2?
(2) Do I understand correctly that in FreeBSD 14.0 I can do this out of the box?
Click to expand...
1. In theory yes, by using = Bhyve_pci_passthru (the link says GPU not supported that's why you need to test or do patches) you will need to test with RTX a2000 and 13.2.
2. What OP stated its that for him FBSD 14.0 + Passthrough with his NVIDIA Quadro P4000 is working, you will have to test with RTX a2000 (hopefully it works out the box like your question implies).

(My journey) I will be testing with some old Nvidia cards (1080ti, 2080ti) but it seems functional now in 13.2/13.1 and in FreeBSD 14.0 should increase the support (if patches are added).
 
GoNeFast_01 said:
1. In theory yes, by using = Bhyve_pci_passthru (the link says GPU not supported that's why you need to test or do patches) you will need to test with RTX a2000 and 13.2.
2. What OP stated its that for him FBSD 14.0 + Passthrough with his NVIDIA Quadro P4000 is working, you will have to test with RTX a2000 (hopefully it works out the box like your question implies).

(My journey) I will be testing with some old Nvidia cards (1080ti, 2080ti) but it seems functional now in 13.2/13.1 and in FreeBSD 14.0 should increase the support (if patches are added).
Click to expand...

Here is a link to a recent presentation at the EuroBSDcon 2023 conference. The link points precisely to the point about support for video cards from various manufacturers. And the speaker states that NVIDIA is not supported:

View: https://youtu.be/eurBCPj65oI?t=439


As far as I understand (and correct me if this is not so), this speaker is practically the only one currently working on this problem. How did NVIDIA cards still work? :-/ I'd like to know more details, but it's not clear where to look. Details I'm talking about:

1 - specific versions of FreeBSD code
2 - specific version of Linux distribution

Maybe someone here who was able to get NVIDIA to work will give a report that will allow me to reproduce the work.
 
What am I doing wrong when trying to passthru RTX A2000 in FreeBSD 14.0?
Code: 
% pciconf -lv | grep -A3 ppt
ppt0@pci0:33:0:0:       class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x2531 subvendor=0x10de subdevice=0x151d
    vendor     = 'NVIDIA Corporation'
    device     = 'GA106 [RTX A2000]'
    class      = display
The bhyve launching script:
Code: 
....
GPU="33/0/0"
....
-s 11,passthru,$GPU
....
and bhyve failes with:
Code: 
bhyve: PCI device at 33/0/0 is not using the ppt(4) driver
device emulation initialization error: Device busy
Thanks for ideas!
 
You need to mask the GPU with the ppts driver in /boot/loader.conf, this way the nvidia graphics card does not attach to the host.

vmm(4)

man.freebsd.org man.freebsd.org
 
driesm said:
You need to mask the GPU with the ppts driver in /boot/loader.conf, this way the nvidia graphics card does not attach to the host.
Click to expand...
Well, the fact that pciconf's output about shows ppt is an evidence of that I already have it in /boot/loader.conf:
Code: 
% grep ppt /boot/loader.conf
pptdevs="33/0/0"
 
No tot sure whether the following can shed some light, and how is this related to pciconf's output of "ppt0@pci0:33:0:0" in terms of numbers?
Code: 
 % dmesg | grep -B2 -A4 ppt
pcib4: <ACPI PCI-PCI bridge> at device 3.1 on pci3
pci4: <ACPI PCI bus> on pcib4
ppt0 port 0x6000-0x607f mem 0xe0000000-0xe0ffffff,0xc0000000-0xcfffffff,0xd0000000-0xd1ffffff irq 86 at device 0.0 on pci4
hdac0: <NVIDIA (0x228e) HDA Controller> mem 0xe1080000-0xe1083fff irq 87 at device 0.1 on pci4
pcib5: <ACPI PCI-PCI bridge> at device 7.1 on pci3
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 8.1 on pci3
 
Passing through both Nvidia devices doesn't help, I still get "busy" on the main one:
Code: 
bhyve: PCI device at 33/0/0 is not using the ppt(4) driver
I wonder if it's related to the fact that Nvidia is the only video adapter installed: the CPU is AMD Ryzen Threadripper 3960X and doesn't have integrated graphics.
 
aragats said:
I wonder if it's related to the fact that Nvidia is the only video adapter installed
Click to expand...
Could be, I wonder if the console is attached to it anyway despite the gpu being ppt'ed. May be try forcing serial or null console (if you don't need boot output and/or system console)?

loader.conf:
Code: 
console="nullconsole"
 
yuripv79 said:
Code: 
console="nullconsole"
Click to expand...
Doesn't help. I wonder why I see 8 getty running with nullconsole though:
Code: 
% ps aux | grep getty
root    2115    0.0  0.0  12832   2372 v0  Is+  22:22     0:00.00 /usr/libexec/getty Pc ttyv0
root    2116    0.0  0.0  12832   2368 v1  Is+  22:22     0:00.00 /usr/libexec/getty Pc ttyv1
root    2117    0.0  0.0  12832   2368 v2  Is+  22:22     0:00.00 /usr/libexec/getty Pc ttyv2
root    2118    0.0  0.0  12832   2372 v3  Is+  22:22     0:00.00 /usr/libexec/getty Pc ttyv3
root    2119    0.0  0.0  12832   2372 v4  Is+  22:22     0:00.00 /usr/libexec/getty Pc ttyv4
root    2120    0.0  0.0  12832   2372 v5  Is+  22:22     0:00.00 /usr/libexec/getty Pc ttyv5
root    2121    0.0  0.0  12832   2368 v6  Is+  22:22     0:00.00 /usr/libexec/getty Pc ttyv6
root    2122    0.0  0.0  12832   2368 v7  Is+  22:22     0:00.00 /usr/libexec/getty Pc ttyv7
 
A few months ago I did tried nvidia passthru on a laptop with intel integrated and dedicated nvidia.
I remember that to use ppt on nvidia I must boot OS with intel selected (leaving nvidia unnused).

At that time I passthru nvidia to win10 and it was recognized but installation of nvidia drivers failed at some point.

Any tips or aditional information about:
vgapci0@pci0:1:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1f99 subvendor=0x17aa subdevice=0x3fa5
vendor = 'NVIDIA Corporation'
device = 'TU117M [GeForce GTX 1650 Mobile / Max-Q]'
class = display
subclass = VGA
?
 
/boot/loader.conf.local:
vmm_load="YES"
pptdevs="1/0/0 1/0/1"

ppt0@pci0:1:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1f99 subvendor=0x17aa subdevice=0x3fa5
vendor = 'NVIDIA Corporation'
device = 'TU117M [GeForce GTX 1650 Mobile / Max-Q]'
class = display
subclass = VGA
ppt1@pci0:1:0:1: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x10fa subvendor=0x17aa subdevice=0x3ffb
vendor = 'NVIDIA Corporation'
class = multimedia
subclass = HDA

bhyve:
-s 6:0,passthru,1/0/0 -s 6:1,passthru,1/0/1

- Installed nvidia drivers without problems
- graphic card is detected but is deactivated with a triangle and a "!":
"Windows stopped this device... code 43"
- hda is detected and shows no problem
 
I installed another graphics card in the same box. I'm not sure whether there is a setting which one is the "default", but the new one does display the console at boot.
However, I'm still getting "device is busy" error when trying to passthru the original one.
Another question: the one I want to passthru is pci0:33:0:0 and the new one pci0:1:0:0. Why those numbers are so different? I'm not really familiar with PCIe, but it seems strange to me that the devices from the same vendor have so different IDs:
Code: 
 % pciconf -lv | grep -A2 vga
vgapci0@pci0:1:0:0:    class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x06fd subvendor=0x10de subdevice=0x062e
    vendor     = 'NVIDIA Corporation'
    device     = 'G98 [Quadro NVS 295]'

% pciconf -lv | grep -A2 ppt
ppt0@pci0:33:0:0:    class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x2531 subvendor=0x10de subdevice=0x151d
    vendor     = 'NVIDIA Corporation'
    device     = 'GA106 [RTX A2000]'
--
ppt1@pci0:33:0:1:    class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x228e subvendor=0x10de subdevice=0x151d
    vendor     = 'NVIDIA Corporation'
    device     = 'GA106 High Definition Audio Controller'
 
You must log in or register to reply here.
Back
Top