25G in a jail or bhyve

B

benoitc

Is there a way to get the full line speed of the NIC in a jail or bhyve ?

I have a server with a mlx5en(4) NIC, a Mellanox ConnectX-4 Lx 2x25G. I would like to setup it as a router and provide a wan for the internal vms /jails and others machines.

Unfortunately there is an issue using SRV-IO with this card. I get a DAD detection error until I disable it: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=270859

I'm not comfortable to disable DAD. Sois there another way to use the full capacity of the card? vnet appears s a 10G interface. .

Code: 
# ifconfig vnet0
vnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
	options=8<VLAN_MTU>
	ether 0e:20:98:4c:84:f0
	hwaddr 02:33:37:e1:d2:0b
	inet6 2a0e:XXXX:XXXX:XXX::4 prefixlen 64
	inet6 fe80::c20:98ff:fe4c:84f0%vnet0 prefixlen 64 scopeid 0x2
	groups: epair
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
 
benoitc said:
vnet appears s a 10G interface. .
Click to expand...
On top of that it is looking like a copper interface.

media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
Click to expand...

Some of the questions I have are SFP28 modules and how they show up.
Did you look at how the card shows up before passthru? Is it showing dual 25G interfaces in host ifconfig?
Does ifconfig mlenx5 -vvv show the transceiver?

What about the firmware? Are you loading it in host and client? Host only?
 
Phishfry said:
On top of that it is looking like a copper interface.



Some of the questions I have are SFP28 modules and how they show up.
Did you look at how the card shows up before passthru? Is it showing dual 25G interfaces in host ifconfig?
Does ifconfig mlenx5 -vvv show the transceiver?

What about the firmware? Are you loading it in host and client? Host only?
Click to expand...

The card without passthru appears to get full 25Gb:

Code: 
root@hv0:~ # ifconfig -vvv mce0
mce0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
	options=7eed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP,NOMAP,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO>
	ether 9c:dc:71:4c:84:f0
	inet6 fe80::9edc:71ff:fe4c:84f0%mce0 prefixlen 64 scopeid 0x4
	inet6 2a0e:XXXX:XXXX:XXXX::12 prefixlen 64
	media: Ethernet 25GBase-SR <full-duplex,rxpause,txpause>
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	plugged: SFP/SFP+/SFP28 100GBASE-SR4 or 25GBASE-SR (LC)
	vendor: FINISAR CORP. PN: FTLF8536P4BCL-HP SN: MY891715DE DATE: 2019-04-24
	module temperature: 44.73 C voltage: 3.29 Volts
	lane 1: RX power: 0.61 mW (-2.17 dBm) TX bias: 44.70 mA

	SFF8472 DUMP (0xA0 0..127 range):
	 03 04 07 00 00 00 00 00 00 00 00 06 ff 00 00 00
	 02 00 0a 07 46 49 4e 49 53 41 52 20 43 4f 52 50
	 2e 20 20 20 02 00 90 65 46 54 4c 46 38 35 33 36
	 50 34 42 43 4c 2d 48 50 41 20 20 20 03 52 00 1d
	 08 1a 67 00 4d 59 38 39 31 37 31 35 44 45 20 20
	 20 20 20 20 31 39 30 34 32 34 31 37 68 f0 08 b3
	 00 00 80 d0 80 81 45 01 38 34 35 33 39 38 2d 42
	 32 31 20 20 20 20 00 00 d4 7d ef 62 4e 52 90 6d

Abiut the firmware i'm not sure I understand the question can you elaborate?
 
The reason I brought up firmware is it relates to how you run the card.

You already mentioned you want high speed router. Jail or Bhyve? Valid questions.

So when you run network cards in Bhyve you have the option as you mentioned of SR-IOV.
This emulates many 1G or 10G connections virtually. Good for sharing pipes.
But you mentioned problems with it, so go pure PCI passthru mode.
Don't load firmware on host (if auto-loading you may need to block it). Nor interfaces.
Allow the Bhyve instance to become the host user by loading mlxen5 firmware there.
Worth a try. Maybe need to do> mlxen5_load="NO" in host loader.conf
 
Just in case you don't know:

The PR you are responding to:
do you have any hint to help to solve this issue. I can try them.
Click to expand...

The beloved HPS passed away recently. He was the maintainer for this driver.
 
Phishfry said:
Just in case you don't know:

The PR you are responding to:


The beloved HPS passed away recently. He was the maintainer for this driver.
Click to expand...
I learnt that :( I can run sr-iov by disabling DAD but I am not comfortable with it; That is after all one of the improvements of IPv6...

So I can run the router in a jail but then it seems i can only get 10G driver and not 25G as expected. But maybe that is not an issue...
 
benoitc said:
So I can run the router in a jail but then it seems i can only get 10G driver and not 25G as expected. But maybe that is not an issue...
Click to expand...
Just my opinion. I don't think the shared kernel of a jail is good for high speed router.

Storage provisioned to a jail on the otherhand is pretty close to line speed compared to Bhyve which hits hard.
 
You must log in or register to reply here.
Back
Top