Hello everyone.
I have a Mellanox MCX354A-FCBT CX354A ConnectX-3 VPI card in my NAS running 14.2 that I am trying to use to connect to my main server using IPOIB.
The main server is using a Mellanox ConnectX-IB (MT27600) card running Ubuntu as that card, to the best of my knowledge, doesn't work with FreeBSD. The Ubuntu server is also running opensm.
The card is working somewhat, but is very slow compared to having used it on Linux.
The card shows up under
and if I load the modules mlx4ib(4) and ipoib the card's two ports show up under ifconfig as ib0 and ib1:
However, the issue I am having is that I cannot seem to enable jumbo frames. The MTU defaults to 2044, which I believe is the max for datagram mode. On Linux, I was able to set the card to connected mode and enable jumbo frames with the MTU set to 65520, but on FreeBSD it just defaults to the smaller frame size as it seems to be stuck in datagram mode.
When I try to set the MTU higher than 2044 I get the following notice:
I believe 2044 is the max MTU for datagram mode. Does anyone know how to change it to connected mode?
On Linux I can just enter the following to change modes:
I believe this is limiting my speed. When the machine was running Linux, iperf3 would give me an average of around 18gbps, where as on FreeBSD I'm lucky to get around 3gbps.
I did see the InfiniBand Wiki page which mentions creating a custom kernel with options IPOIB_CM, although I may have misunderstood things.
I gave that a try, creating a file /etc/src.conf which contained only one line:
Then created a new kernel config based off of GENERIC named MYKERNEL with the following lines added:
After building and installing MYKERNEL I was still unable to set the MTU higher than 2044, and now iperf3 shows even slower speeds averaging around 1.2gbps.
I had to make sure that /etc/rc.conf was not loading ipoib nor mlx4ib as that would cause the network interfaces to show up twice for ifconfig. With the custom kernel installed, I only have the following modules loaded:
and the following defaults for /boot/loader.conf:
With the new kernel installed, I do get something from dmesg about ipoib_cm saying that it failed:
The full dmseg is as follows:
Anyone have any idea why ipoib_cm is failing to load or how else I can set the card to connected mode? Is there any way to speed it up or make the frames larger for data transfer between the two machines?
Any help is appreciated!
I have a Mellanox MCX354A-FCBT CX354A ConnectX-3 VPI card in my NAS running 14.2 that I am trying to use to connect to my main server using IPOIB.
The main server is using a Mellanox ConnectX-IB (MT27600) card running Ubuntu as that card, to the best of my knowledge, doesn't work with FreeBSD. The Ubuntu server is also running opensm.
The card is working somewhat, but is very slow compared to having used it on Linux.
The card shows up under
pciconf -lv:
Code:
mlx4_core0@pci0:5:0:0: class=0x020700 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1003 subvendor=0x15b3 subdevice=0x0050
vendor = 'Mellanox Technologies'
device = 'MT27500 Family [ConnectX-3]'
class = network
subclass = InfiniBandand if I load the modules mlx4ib(4) and ipoib the card's two ports show up under ifconfig as ib0 and ib1:
Code:
$ ifconfig
em0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=4e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 2c:44:fd:15:01:28
inet 192.168.50.68 netmask 0xffffff00 broadcast 192.168.50.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
re0: flags=1008802<BROADCAST,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
ether 00:13:3b:e7:2f:8e
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ib0: flags=1008063<UP,BROADCAST,NEEDSEPOCH,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 2044
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 0.0.2.8.fe.80.0.0.0.0.0.0.e4.1d.2d.3.0.4c.5b.81
inet 192.168.200.1 netmask 0xffffff00 broadcast 192.168.200.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ib1: flags=1008063<UP,BROADCAST,NEEDSEPOCH,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 4092
options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
lladdr 0.0.2.9.fe.80.0.0.0.0.0.0.e4.1d.2d.3.0.4c.5b.82
inet 192.168.200.2 netmask 0xffffff00 broadcast 192.168.200.255
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>However, the issue I am having is that I cannot seem to enable jumbo frames. The MTU defaults to 2044, which I believe is the max for datagram mode. On Linux, I was able to set the card to connected mode and enable jumbo frames with the MTU set to 65520, but on FreeBSD it just defaults to the smaller frame size as it seems to be stuck in datagram mode.
When I try to set the MTU higher than 2044 I get the following notice:
Code:
$ sudo ifconfig ib0 inet 192.168.200.1 netmask 255.255.255.0 mtu 65520 up
ifconfig: ioctl SIOCSIFMTU (set mtu): Invalid argument
$ ifconfig ib0<br>ib0: flags=1008063<UP,BROADCAST,NEEDSEPOCH,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 2044
options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
lladdr 0.0.2.10.fe.80.0.0.0.0.0.0.e4.1d.2d.3.0.4c.5b.81nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>I believe 2044 is the max MTU for datagram mode. Does anyone know how to change it to connected mode?
On Linux I can just enter the following to change modes:
echo connected | sudo tee /sys/class/net/ibp0s7d1/mode and it then allows me to set jumbo frames. On FreeBSD this doesn't work as we don't use /sysI believe this is limiting my speed. When the machine was running Linux, iperf3 would give me an average of around 18gbps, where as on FreeBSD I'm lucky to get around 3gbps.
I did see the InfiniBand Wiki page which mentions creating a custom kernel with options IPOIB_CM, although I may have misunderstood things.
I gave that a try, creating a file /etc/src.conf which contained only one line:
Code:
WITH_OFED='yesThen created a new kernel config based off of GENERIC named MYKERNEL with the following lines added:
Code:
# For IPoIB connected mode and SDP
options OFED
options SDP
options IPOIB_CM
# For ConnectX-3 IB support
device mlx4
device mlx4ib
device ipoib
device mlx4en
device mthcaAfter building and installing MYKERNEL I was still unable to set the MTU higher than 2044, and now iperf3 shows even slower speeds averaging around 1.2gbps.
I had to make sure that /etc/rc.conf was not loading ipoib nor mlx4ib as that would cause the network interfaces to show up twice for ifconfig. With the custom kernel installed, I only have the following modules loaded:
Code:
$ kldstat
Id Refs Address Size Name
1 36 0xffffffff80200000 215d518 kernel
2 1 0xffffffff8235e000 5da658 zfs.ko
3 1 0xffffffff82939000 77d8 cryptodev.ko
4 1 0xffffffff833e5000 3390 acpi_wmi.ko
5 1 0xffffffff833e9000 4250 ichsmb.ko
6 1 0xffffffff833ee000 2178 smbus.ko
7 1 0xffffffff833f1000 8e00 ioat.ko
8 1 0xffffffff833fa000 3360 uhid.ko
9 1 0xffffffff833fe000 4364 ums.ko
10 1 0xffffffff83403000 2a68 mac_ntpd.ko
Code:
$ cat /boot/loader.conf
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
cryptodev_load="YES"
zfs_load="YES"With the new kernel installed, I do get something from dmesg about ipoib_cm saying that it failed:
Code:
$ dmesg | grep ipoib
ib0: ipoib_cm_post_receive_srq failed for buf 0
ib1: ipoib_cm_post_receive_srq failed for buf 0The full dmseg is as follows:
Anyone have any idea why ipoib_cm is failing to load or how else I can set the card to connected mode? Is there any way to speed it up or make the frames larger for data transfer between the two machines?
Any help is appreciated!