Can't make bhyve with lagg failover work on wifi

[JonyPost]

Hello,

I'm using bhyve on 12.1-RELEASE-p6, with failover mode between ethernet and wireless interfaces enabled and configured as following:

ifconfig_em0="ether 10:0b:a9:9e:76:28"
wlans_iwn0="wlan0"
ifconfig_wlan0="WPA"
cloned_interfaces="lagg0"
ifconfig_lagg0="up laggproto failover laggport em0 laggport wlan0 DHCP"
ifconfig_bridge0="addm lagg0 addm tap0"

The problem with that configuration is that the bhyve VM's networking only works when I'm using ethernet, and doesn't work on wireless.

Here's the ifconfig output:

em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=800000<>
    ether 10:0b:a9:9e:76:28
    hwaddr f0:de:f1:d0:ce:60
    media: Ethernet autoselect (100baseTX <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
wlan0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 10:0b:a9:9e:76:28
    groups: wlan
    ssid Basis channel 6 (2437 MHz 11g ht/20) bssid 38:2c:4a:c3:1e:e0
    regdomain ETSI country RU authmode WPA2/802.11i privacy ON
    deftxkey UNDEF AES-CCM 2:128-bit txpower 30 bmiss 10 scanvalid 60
    protmode CTS ampdulimit 64k ampdudensity 8 -amsdutx amsdurx shortgi
    -stbc -ldpc wme roaming MANUAL
    media: IEEE 802.11 Wireless Ethernet MCS mode 11ng
    status: associated
    nd6 options=2b<PERFORMNUD,ACCEPT_RTADV,IFDISABLED,AUTO_LINKLOCAL>
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 10:0b:a9:9e:76:28
    inet 10.0.0.116 netmask 0xffffff00 broadcast 10.0.0.255
    laggproto failover lagghash l2,l3,l4
    laggport: em0 flags=5<MASTER,ACTIVE>
    laggport: wlan0 flags=0<>
    groups: lagg
    media: Ethernet autoselect
    status: active
    nd6 options=2b<PERFORMNUD,ACCEPT_RTADV,IFDISABLED,AUTO_LINKLOCAL>
tap0: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=80000<LINKSTATE>
    ether 00:bd:6a:63:f7:00
    groups: tap
    media: Ethernet autoselect
    status: no carrier
    nd6 options=2b<PERFORMNUD,ACCEPT_RTADV,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:d5:be:df:87:00
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 2000000
    member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 4 priority 128 path cost 200000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>

Command I'm using to start bhyve VM:

bhyve -c 2 -m 2G -H -w \
        -s 0,hostbridge \
        -s 4,ahci-hd,/vm/w7/w7.img,sectorsize=512 \
        -s 5,virtio-net,tap0 \
        -s 30,xhci,tablet \
        -s 31,lpc \
        -l com1,stdio \
        -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
        w7

What is the best way to configure interfaces with lagg failover, so that my VM's can use the network on wireless?

 

[SirDice]

The problem with that configuration is that the bhyve VM's networking only works when I'm using ethernet, and doesn't work on wireless.

This is the same problem as trying to bridge(4) your wired and wireless networks. Basically your wireless interface has to spoof MAC addresses for this to work and most (all?) wireless drivers don't like that.

 

[hsw]

The same thing happens with vnet jails, once the lagg fails over to wlan the jail is no longer able to send anything. I looked at the iwm and iwn drivers an the only support one virtual device, so no possibility of creating a second wlan with the jail;s MAC address. I saw the the rtwn does support two virtual interfaces, but my card is slightly newer RTL8822BE (not supported) so I cannot test this at present.

 

[Phishfry]

I can't help with the LAGG problem but I do use a bridge with both wired and wireless interfaces.
I am using a bridge on my PCEngines APU2 that I use as a wireless access point using Atheros AR9580.
Previously I used dnsmasq for dhcp to each interface both wired and wireless.
Then I figured out how to bridge the wireless interface.
The key was specifying the MTU for the wireless interface. All bridge members must use the same MTU.
I notice your wireless interface is showing MTU 1500 so this may not be any help.
Here is my setup:

cloned_interfaces="bridge0"
ifconfig_bridge0="addm igb0 addm igb1 addm igb2 addm wlan0 SYNCDHCP"
wlans_ath0="wlan0"
ifconfig_wlan0="up mtu 1500"
create_args_wlan0="wlanmode hostap country US ssid apu2ap channel 158"
ifconfig_igb0="up"
ifconfig_igb1="up"
ifconfig_igb2="up"
hostapd_enable="YES"
gateway_enable="YES"
defaultrouter="192.168.1.1"

Here is part of my relevant ifconfig:

wlan0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 4c:5e:0c:11:65:38
    groups: wlan
    ssid apu2ap channel 36 (5180 MHz 11a ht/40+) bssid 4c:5e:0c:11:65:38
    regdomain FCC country US ecm authmode WPA2/802.11i privacy MIXED
    deftxkey 3 TKIP 2:128-bit TKIP 3:128-bit txpower 17 mcastrate 6
    mgmtrate 6 scanvalid 60 ampdulimit 64k ampdudensity 8 shortgi wme
    burst dtimperiod 1 -dfs
    media: IEEE 802.11 Wireless Ethernet autoselect mode 11na <hostap>
    status: running
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 00:0d:b9:47:05:8c
    inet 192.168.1.104 netmask 0xffffff00 broadcast 192.168.1.255
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: wlan0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 33333
    member: igb2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 3 priority 128 path cost 2000000
    member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 2 priority 128 path cost 2000000
    member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 2000000
    groups: bridge