proxmox Problems with FreeBSD on Proxmox with Chelsio T520-LL-CR SR-IOV

T

TheNetStriker

I'm having problems getting my Chelsio T520-LL-CR SR-IOV nic to work in FreeBSD 14 and 14.1 VM's running on a Proxmox server with SR-IOV pci passthrough.

I got the card to show up in ifconfig and it also can aquire an ip address via DHCP. I can ping the ip from other pc's and I can also ping other pc's from the VM. But I cannot get other protocols to work (e.g. an SSH connection) even with the firewall completely deactivated.
I tried to analyze the problem with tcpdump and Wireshark. SSH packets are showing up in tcpdump on the VM and there are also outgoing SSH packets, but they just don't arrive at the pc that tries to open the SSH connection.
I also tried the same SR-IOV pci device on a linux VM and there everything works perfectly.

Here is my /boot/loader.conf
Code: 
t4fw_cfg_load="YES"
t5fw_cfg_load="YES"
t6fw_cfg_load="YES"
if_cxgbe_load="YES"
if_cxgbev_load="YES"

And here is my /etc/rc.conf. (vtnet0 is a VirtIO network card I've added because I cannot connect via the Chelsio card)
Code: 
hostname="freebsd14"
keymap="ch.kbd"
ifconfig_vtnet0="DHCP"
ifconfig_cxlv0="DHCP"
sshd_enable="YES"
moused_nondefault_enable="NO"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"

This is the /var/log/messages output:
Code: 
Nov 17 13:22:53 freebsd14 syslogd: kernel boot file is /boot/kernel/kernel
Nov 17 13:22:53 freebsd14 kernel: ---<<BOOT>>---
Nov 17 13:22:53 freebsd14 kernel: Copyright (c) 1992-2023 The FreeBSD Project.
Nov 17 13:22:53 freebsd14 kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Nov 17 13:22:53 freebsd14 kernel:     The Regents of the University of California. All rights reserved.
Nov 17 13:22:53 freebsd14 kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Nov 17 13:22:53 freebsd14 kernel: FreeBSD 14.0-RELEASE #0 releng/14.0-n265380-f9716eee8ab4: Fri Nov 10 05:57:23 UTC 2023
Nov 17 13:22:53 freebsd14 kernel:     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
Nov 17 13:22:53 freebsd14 kernel: FreeBSD clang version 16.0.6 (https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a259152)
Nov 17 13:22:53 freebsd14 kernel: VT(vga): text 80x25
Nov 17 13:22:53 freebsd14 kernel: CPU: QEMU Virtual CPU version 2.5+ (2494.13-MHz K8-class CPU)
Nov 17 13:22:53 freebsd14 kernel:   Origin="GenuineIntel"  Id=0x60fb1  Family=0xf  Model=0x6b  Stepping=1
Nov 17 13:22:53 freebsd14 kernel:   Features=0x783fbfd<FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
Nov 17 13:22:53 freebsd14 kernel:   Features2=0x82b82201<SSE3,SSSE3,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,AESNI,HV>
Nov 17 13:22:53 freebsd14 kernel:   AMD Features=0x20100800<SYSCALL,NX,LM>
Nov 17 13:22:53 freebsd14 kernel:   AMD Features2=0x1<LAHF>
Nov 17 13:22:53 freebsd14 kernel: Hypervisor: Origin = "KVMKVMKVM"
Nov 17 13:22:53 freebsd14 kernel: real memory  = 2147483648 (2048 MB)
Nov 17 13:22:53 freebsd14 kernel: avail memory = 2043969536 (1949 MB)
Nov 17 13:22:53 freebsd14 kernel: Event timer "LAPIC" quality 100
Nov 17 13:22:53 freebsd14 kernel: ACPI APIC Table: <BOCHS  BXPC    >
Nov 17 13:22:53 freebsd14 kernel: random: unblocking device.
Nov 17 13:22:53 freebsd14 kernel: ioapic0 <Version 1.1> irqs 0-23
Nov 17 13:22:53 freebsd14 kernel: random: entropy device external interface
Nov 17 13:22:53 freebsd14 kernel: kbd1 at kbdmux0
Nov 17 13:22:53 freebsd14 kernel: vtvga0: <VT VGA driver>
Nov 17 13:22:53 freebsd14 kernel: kvmclock0: <KVM paravirtual clock>
Nov 17 13:22:53 freebsd14 kernel: Timecounter "kvmclock" frequency 1000000000 Hz quality 975
Nov 17 13:22:53 freebsd14 kernel: kvmclock0: registered as a time-of-day clock, resolution 0.000001s
Nov 17 13:22:53 freebsd14 kernel: smbios0: <System Management BIOS> at iomem 0xf5290-0xf52ae
Nov 17 13:22:53 freebsd14 kernel: smbios0: Version: 2.8, BCD Revision: 2.8
Nov 17 13:22:53 freebsd14 kernel: aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>
Nov 17 13:22:53 freebsd14 kernel: acpi0: <BOCHS BXPC>
Nov 17 13:22:53 freebsd14 kernel: acpi0: Power Button (fixed)
Nov 17 13:22:53 freebsd14 kernel: cpu0: <ACPI CPU> on acpi0
Nov 17 13:22:53 freebsd14 kernel: atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
Nov 17 13:22:53 freebsd14 kernel: atrtc0: registered as a time-of-day clock, resolution 1.000000s
Nov 17 13:22:53 freebsd14 kernel: Event timer "RTC" frequency 32768 Hz quality 0
Nov 17 13:22:53 freebsd14 kernel: hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Nov 17 13:22:53 freebsd14 kernel: Timecounter "HPET" frequency 100000000 Hz quality 950
Nov 17 13:22:53 freebsd14 kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
Nov 17 13:22:53 freebsd14 kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0x608-0x60b on acpi0
Nov 17 13:22:53 freebsd14 kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
Nov 17 13:22:53 freebsd14 kernel: pci0: <ACPI PCI bus> on pcib0
Nov 17 13:22:53 freebsd14 kernel: isab0: <PCI-ISA bridge> at device 1.0 on pci0
Nov 17 13:22:53 freebsd14 kernel: isa0: <ISA bus> on isab0
Nov 17 13:22:53 freebsd14 kernel: atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf080-0xf08f at device 1.1 on pci0
Nov 17 13:22:53 freebsd14 kernel: ata0: <ATA channel> at channel 0 on atapci0
Nov 17 13:22:53 freebsd14 kernel: ata1: <ATA channel> at channel 1 on atapci0
Nov 17 13:22:53 freebsd14 kernel: uhci0: <Intel 82371SB (PIIX3) USB controller> port 0xf040-0xf05f irq 11 at device 1.2 on pci0
Nov 17 13:22:53 freebsd14 kernel: usbus0 on uhci0
Nov 17 13:22:53 freebsd14 kernel: pci0: <bridge> at device 1.3 (no driver attached)
Nov 17 13:22:53 freebsd14 kernel: vgapci0: <VGA-compatible display> mem 0xfc000000-0xfcffffff,0xfea5a000-0xfea5afff at device 2.0 on pci0
Nov 17 13:22:53 freebsd14 kernel: vgapci0: Boot video device
Nov 17 13:22:53 freebsd14 kernel: virtio_pci0: <VirtIO PCI (legacy) Balloon adapter> port 0xf000-0xf03f mem 0xfd600000-0xfd603fff irq 11 at device 3.0 on pci0
Nov 17 13:22:53 freebsd14 kernel: vtballoon0: <VirtIO Balloon Adapter> on virtio_pci0
Nov 17 13:22:53 freebsd14 kernel: pcib1: <ACPI PCI-PCI bridge> mem 0xfea5b000-0xfea5b0ff irq 10 at device 5.0 on pci0
Nov 17 13:22:53 freebsd14 kernel: pci1: <ACPI PCI bus> on pcib1
Nov 17 13:22:53 freebsd14 kernel: virtio_pci1: <VirtIO PCI (legacy) SCSI adapter> port 0xe000-0xe03f mem 0xfe800000-0xfe800fff,0xfd400000-0xfd403fff irq 10 at device 1.0 on pci1
Nov 17 13:22:53 freebsd14 kernel: vtscsi0: <VirtIO SCSI Adapter> on virtio_pci1
Nov 17 13:22:53 freebsd14 kernel: t5vf0: <Chelsio T520-LL-CR VF> mem 0xfea5c000-0xfea5cfff,0xfea50000-0xfea57fff,0xfea58000-0xfea59fff at device 16.0 on pci0
Nov 17 13:22:53 freebsd14 kernel: cxlv0: <port 0> on t5vf0
Nov 17 13:22:53 freebsd14 kernel: cxlv0: Ethernet address: 06:44:2c:c6:80:00
Nov 17 13:22:53 freebsd14 kernel: cxlv0: 1 txq, 1 rxq (NIC)
Nov 17 13:22:53 freebsd14 kernel: t5vf0: 1 ports, 2 MSI-X interrupts, 3 eq, 2 iq
Nov 17 13:22:53 freebsd14 kernel: virtio_pci2: <VirtIO PCI (legacy) Network adapter> port 0xf060-0xf07f mem 0xfea5d000-0xfea5dfff,0xfd604000-0xfd607fff irq 10 at device 18.0 on pci0
Nov 17 13:22:53 freebsd14 kernel: vtnet0: <VirtIO Networking Adapter> on virtio_pci2
Nov 17 13:22:53 freebsd14 kernel: vtnet0: Ethernet address: bc:24:11:60:9e:a4
Nov 17 13:22:53 freebsd14 kernel: vtnet0: netmap queues/slots: TX 1/256, RX 1/512
Nov 17 13:22:53 freebsd14 kernel: 000.000759 [ 452] vtnet_netmap_attach       vtnet attached txq=1, txd=256 rxq=1, rxd=512
Nov 17 13:22:53 freebsd14 kernel: pcib2: <ACPI PCI-PCI bridge> mem 0xfea5e000-0xfea5e0ff irq 10 at device 30.0 on pci0
Nov 17 13:22:53 freebsd14 kernel: pci2: <ACPI PCI bus> on pcib2
Nov 17 13:22:53 freebsd14 kernel: pcib3: <ACPI PCI-PCI bridge> mem 0xfea5f000-0xfea5f0ff irq 11 at device 31.0 on pci0
Nov 17 13:22:53 freebsd14 kernel: pci3: <ACPI PCI bus> on pcib3
Nov 17 13:22:53 freebsd14 kernel: acpi_syscontainer0: <System Container> on acpi0
Nov 17 13:22:53 freebsd14 kernel: vmgenc0: <VM Generation Counter> on acpi0
Nov 17 13:22:53 freebsd14 kernel: acpi_syscontainer1: <System Container> port 0xaf00-0xaf0b on acpi0
Nov 17 13:22:53 freebsd14 kernel: acpi_syscontainer2: <System Container> port 0xafe0-0xafe3 on acpi0
Nov 17 13:22:53 freebsd14 kernel: acpi_syscontainer3: <System Container> port 0xae00-0xae17 on acpi0
Nov 17 13:22:53 freebsd14 kernel: atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
Nov 17 13:22:53 freebsd14 kernel: atkbd0: <AT Keyboard> irq 1 on atkbdc0
Nov 17 13:22:53 freebsd14 kernel: kbd0 at atkbd0
Nov 17 13:22:53 freebsd14 kernel: atkbd0: [GIANT-LOCKED]
Nov 17 13:22:53 freebsd14 kernel: psm0: <PS/2 Mouse> irq 12 on atkbdc0
Nov 17 13:22:53 freebsd14 kernel: psm0: [GIANT-LOCKED]
Nov 17 13:22:53 freebsd14 kernel: WARNING: Device "psm" is Giant locked and may be deleted before FreeBSD 15.0.
Nov 17 13:22:53 freebsd14 kernel: psm0: model IntelliMouse Explorer, device ID 4
Nov 17 13:22:53 freebsd14 kernel: fdc0: <floppy drive controller (FDE)> port 0x3f2-0x3f5,0x3f7 irq 6 drq 2 on acpi0
Nov 17 13:22:53 freebsd14 kernel: fdc0: does not respond
Nov 17 13:22:53 freebsd14 kernel: device_attach: fdc0 attach returned 6
Nov 17 13:22:53 freebsd14 kernel: orm0: <ISA Option ROM> at iomem 0xe7800-0xeffff pnpid ORM0000 on isa0
Nov 17 13:22:53 freebsd14 kernel: vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff pnpid PNP0900 on isa0
Nov 17 13:22:53 freebsd14 kernel: attimer0: <AT timer> at port 0x40 on isa0
Nov 17 13:22:53 freebsd14 kernel: Timecounter "i8254" frequency 1193182 Hz quality 0
Nov 17 13:22:53 freebsd14 kernel: Event timer "i8254" frequency 1193182 Hz quality 100
Nov 17 13:22:53 freebsd14 kernel: attimer0: non-PNP ISA device will be removed from GENERIC in FreeBSD 15.
Nov 17 13:22:53 freebsd14 kernel: fdc0: No FDOUT register!
Nov 17 13:22:53 freebsd14 kernel: Timecounter "TSC-low" frequency 1246994691 Hz quality 800
Nov 17 13:22:53 freebsd14 kernel: Timecounters tick every 10.000 msec
Nov 17 13:22:53 freebsd14 kernel: usbus0: 12Mbps Full Speed USB v1.0
Nov 17 13:22:53 freebsd14 kernel: ugen0.1: <Intel UHCI root HUB> at usbus0
Nov 17 13:22:53 freebsd14 kernel: uhub0 on usbus0
Nov 17 13:22:53 freebsd14 kernel: uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
Nov 17 13:22:53 freebsd14 kernel: Trying to mount root from ufs:/dev/da0p2 [rw]...
Nov 17 13:22:53 freebsd14 kernel: cd0 at ata1 bus 0 scbus1 target 0 lun 0
Nov 17 13:22:53 freebsd14 kernel: cd0: <QEMU QEMU DVD-ROM 2.5+> Removable CD-ROM SCSI device
Nov 17 13:22:53 freebsd14 kernel: cd0: Serial Number QM00003
Nov 17 13:22:53 freebsd14 kernel: cd0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes)
Nov 17 13:22:53 freebsd14 kernel: cd0: 4330MB (2217336 2048 byte sectors)
Nov 17 13:22:53 freebsd14 kernel: da0 at vtscsi0 bus 0 scbus2 target 0 lun 0
Nov 17 13:22:53 freebsd14 kernel: da0: <QEMU QEMU HARDDISK 2.5+> Fixed Direct Access SPC-3 SCSI device
Nov 17 13:22:53 freebsd14 kernel: da0: 300.000MB/s transfers
Nov 17 13:22:53 freebsd14 kernel: da0: Command Queueing enabled
Nov 17 13:22:53 freebsd14 kernel: da0: 32768MB (67108864 512 byte sectors)
Nov 17 13:22:53 freebsd14 kernel: uhub0: 2 ports with 2 removable, self powered
Nov 17 13:22:53 freebsd14 kernel: ugen0.2: <QEMU QEMU USB Tablet> at usbus0
Nov 17 13:22:53 freebsd14 kernel: intsmb0: <Intel PIIX4 SMBUS Interface> irq 9 at device 1.3 on pci0
Nov 17 13:22:53 freebsd14 kernel: intsmb0: intr IRQ 9 enabled revision 0
Nov 17 13:22:53 freebsd14 kernel: smbus0: <System Management Bus> on intsmb0
Nov 17 13:22:53 freebsd14 kernel: lo0: link state changed to UP
Nov 17 13:22:53 freebsd14 kernel: cxlv0: link state changed to UP
Nov 17 13:22:53 freebsd14 kernel: vtnet0: link state changed to UP
Nov 17 13:22:53 freebsd14 kernel: uhid0 on uhub0
Nov 17 13:22:53 freebsd14 kernel: uhid0: <QEMU QEMU USB Tablet, class 0/0, rev 2.00/0.00, addr 2> on usbus0

What could be the cause of this problem and is there anything else I could provide to analyze this problem?
 
Update: I tested this in various FreeBSD versions and also in the latest pfSense version and the problem was in all versions. Now I just tested the lastest OPNsense version 24.7 and there the virtual interfaces are working perfectly. Any idea what could be the difference in OPNsense?
 
For SR-IOV you must make a control file.

/etc/cxl0.conf
Code: 
PF {
    device : "cxl0";
    num_vfs : 5;
}
#DEFAULT {
#    passthrough : true;
#}
#VF-0 {passthrough : true;}
#VF-1 {passthrough : true;}

/boot/loader.conf
Code: 
#####  Chelsio  #####
t5fw_cfg_load="YES"
if_cxgbe_load="YES"
if_cxgbev_load="YES"
 
Phishfry said:
For SR-IOV you must make a control file.

/etc/cxl0.conf
Code: 
PF {
    device : "cxl0";
    num_vfs : 5;
}
#DEFAULT {
#    passthrough : true;
#}
#VF-0 {passthrough : true;}
#VF-1 {passthrough : true;}

/boot/loader.conf
Code: 
#####  Chelsio  #####
t5fw_cfg_load="YES"
if_cxgbe_load="YES"
if_cxgbev_load="YES"
Click to expand...
I also found documentation about this file, but as far as I understand this is used to enable the virtual interfaces when FreeBSD is running natively on the host. But I'm running FreeBSD as a VM on a Proxmox host.

I've enabled the virtual interfaces on the proxmox host and I'm just passing through one of the virtual interfaces to the FreeBSD VM. For some reason a ping to the ip assigned to this interface in the FreeBSD VM works, but I cannot establish tcp connections. (e.g. an SSH connection)

On OPNsense this works for some reason. I only loaded the kernel drivers (kldload if_cxgbe and kldload if_cxgbev) and assigned an ip to the interface and it just worked.

I'm currently trying to find out what's the difference in OPNsense. I found an single additional commit in the FreeBSD kernel of OPNsense, but I don't know it this is making a difference or if it's a different configuration. Here is a link to the commit: https://github.com/opnsense/src/commit/c4da7b7e66e4d90c45ddd781850db887ac2f0f06
 
Phishfry said:
Yea no need for SR-IOV settings file inside VM than for guest.
Click to expand...
I finally found out what is causing this problem. The hardware checksum offloading does not work on the virtual interface for some reason. If I disable the hardware checksum offloading (ifconfig cxlv0 -rxcsum -txcsum) I can connect to ssh and other tcp ports on the VM.

Is there a reason why the hardware checksum offloading does not work on the virtual interface?

There is also another problem. When the network card is configured with ipv6 I get a lof of those messages in dmesg:
cxlv0: a looped back NS message is detected during DAD for fe80:1::444:2cff:fec6:8000. Another DAD probes are being sent.

I found a bug report about this message here, but sadly there is not solution but to disable the DAD check completely.
But the bug is only related to the mce driver. (NVIDIA Mellanox ConnectX-4/5/6) Is there maybe a way to fix this for Chelsio cards?
 
TheNetStriker said:
Is there a reason why the hardware checksum offloading does not work on the virtual interface?
Click to expand...
I turn a bunch off at host level. I found these affect my network. Every network is different.
Code: 
### LAGG ###
ifconfig_cxl0="up mtu 9000 -tso4 -tso6 -lro -vlanhwtso"
ifconfig_cxl1="up mtu 9000 -tso4 -tso6 -lro -vlanhwtso"
ifconfig_cxl2="up mtu 9000 -tso4 -tso6 -lro -vlanhwtso"
ifconfig_cxl3="up mtu 9000 -tso4 -tso6 -lro -vlanhwtso"
cloned_interfaces="lagg0"


Disabling these features give more ressource to the chip:
Click to expand...

The benefit can be huge:
Click to expand...
 
You must log in or register to reply here.
Back
Top