bhyve pci_paththru. bge0: attaching PHYs failed on the guest side

kaN5300

kaN5300

Hello! Having troubles with passing pci device to the guest. Here is what we have:

Host:
Code: 
FreeBSD web2 11.1-RELEASE FreeBSD 11.1-RELEASE #0 r321309: Fri Jul 21 02:08:28 UTC 2017     root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

cat /boot/loder.conf | grep ppt

pciconf -vl | grep bge
bge0@pci0:3:0:0:        class=0x020000 card=0x2133103c chip=0x165f14e4 rev=0x00 hdr=0x00

vm passthru | grep ppt
ppt0       3/0/1        Yes          NetXtreme BCM5720 Gigabit Ethernet PCIe

bhyveload -c /dev/nmdm5A -S -m 256M -e autoboot_delay=3 -d /usr/vm/vm4/FreeBSD-11.1-RELEASE-amd64.raw vm4
bhyve -l com1,/dev/nmdm5A -c 1 -m 256M -S -AHP -U 43157865-e6ed-11e7-8139-6451060e7679 -s 0,hostbridge -s 31,lpc -s 4:0,virtio-blk,/usr/vm/vm4/FreeBSD-11.1-RELEASE-amd64.raw -s 5,passthru,3/0/1 vm4

Guest:
Code: 
dmesg | grep bge
bge0: <HP Ethernet 1Gb 2-port 332i Adapter, ASIC rev. 0x5720000> mem 0xc0010000-0xc001ffff,0xc0020000-0xc002ffff,0xc0030000-0xc003ffff irq 18 at device 5.0 on pci0
bge0: APE FW version: NCSI v1.2.46.0
bge0: CHIP ID 0x05720000; ASIC REV 0x5720; CHIP REV 0x57200; PCI-E
bge0: Try again
bge0: Try again
bge0: Try again
bge0: Try again
bge0: attaching PHYs failed
device_attach: bge0 attach returned 6

pciconf -l -v
hostb0@pci0:0:0:0:      class=0x060000 card=0x00000000 chip=0x12751275 rev=0x00 hdr=0x00
    vendor     = 'Network Appliance Corporation'
    class      = bridge
    subclass   = HOST-PCI
virtio_pci0@pci0:0:4:0: class=0x010000 card=0x00021af4 chip=0x10011af4 rev=0x00 hdr=0x00
    vendor     = 'Red Hat, Inc'
    device     = 'Virtio block device'
    class      = mass storage
    subclass   = SCSI
none0@pci0:0:5:0:       class=0x020000 card=0x2133103c chip=0x165f14e4 rev=0x00 hdr=0x00
    vendor     = 'Broadcom Limited'
    device     = 'NetXtreme BCM5720 Gigabit Ethernet PCIe'
    class      = network
    subclass   = ethernet
isab0@pci0:0:31:0:      class=0x060100 card=0x00000000 chip=0x70008086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82371SB PIIX3 ISA [Natoma/Triton II]'
    class      = bridge
    subclass   = PCI-ISA

ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128 
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
        inet 127.0.0.1 netmask 0xff000000 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo

I read some old info about bge driver and matching guest and host slot numbers: https://lists.freebsd.org/pipermail/freebsd-virtualization/2014-January/001976.html

Maybe i understood it wrong... In my case we have one multifunction pci device (NIC). First function (3/0/0) is for the host management (bge0). Second function (3/0/1) is passthru for the guest. If i am trying to match params with bhyve (e.g. -s 3:1,passthru,3/0/1) i am getting no network devs inside vm:

Host:
Code: 
bhyve -l com1,/dev/nmdm5A -c 1 -m 256M -S -AHP -U 43157865-e6ed-11e7-8139-6451060e7679 -s 0,hostbridge -s 31,lpc -s 4:0,virtio-blk,/usr/vm/vm4/FreeBSD-11.1-RELEASE-amd64.raw -s 3:1,passthru,3/0/1 vm4
Guest:
Code: 
pciconf -l -v
hostb0@pci0:0:0:0:      class=0x060000 card=0x00000000 chip=0x12751275 rev=0x00 hdr=0x00
    vendor     = 'Network Appliance Corporation'
    class      = bridge
    subclass   = HOST-PCI
virtio_pci0@pci0:0:4:0: class=0x010000 card=0x00021af4 chip=0x10011af4 rev=0x00 hdr=0x00
    vendor     = 'Red Hat, Inc'
    device     = 'Virtio block device'
    class      = mass storage
    subclass   = SCSI
isab0@pci0:0:31:0:      class=0x060100 card=0x00000000 chip=0x70008086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82371SB PIIX3 ISA [Natoma/Triton II]'
    class      = bridge
    subclass   = PCI-ISA

What options do i have in my case? Is it possible to passthru second NIC function to vm?
 
There are a few requirements for pass-through to work, the host needs to support IOMMU (VT-d) and the device needs to support MSI/MSI-x.

Have a look through here: https://wiki.freebsd.org/bhyve/pci_passthru

As far as I understood it you don't need to match up the host's bus/slot/function but you may need to pass-through each individual function.
 
You must log in or register to reply here.
Back
Top