iohyve networking issue

fnoyanisi

fnoyanisi

Hi,

My host OS is FreeBSD 12.1-RELEASE and I am running two bhyve guests on it. One of them is a Debian GNU/Linux and the other on is FreeBSD-CURRENT.

I use iohyve to manage guest operating systems. Debian uses tap1 whereas FreeBSD-CURRENT uses tap0 as the network interface.

The Debian guest can ping the default gateway, which is 192.168.1.254, where FreeBSD-CURRENT guest cannot. I noticed that iohyve creates three interfaces inside the bridge0, which seems a bit weird to me. The configurations of the host and the guest are below. How can I get the FreeBSD-CURRENT machine ping the default router?

Code: 
root@beastie:/home/fnoyanisi # iohyve getall debian
Getting debian iohyve properties...
bargs            -A_-H_-P
bhyve_path       /usr/sbin/bhyve
boot             1
con              nmdm1
cpu              1
description      Sat May 30 22:41:57 UTC 2020
install          no
loader           grub-bhyve
name             debian
os               default
persist          1
ram              2048M
size             30G
tap              tap1
template         NO
utc              YES
vnc              NO
vnc_h            600
vnc_ip           127.0.0.1
vnc_tablet       NO
vnc_w            800
vnc_wait         NO
root@beastie:/home/fnoyanisi # iohyve getall freebsd-current
Getting freebsd-current iohyve properties...
bargs            -A_-H_-P
bhyve_path       /usr/sbin/bhyve
boot             1
con              nmdm0
cpu              1
description      Thu Feb 13 13:29:26 UTC 2020
install          no
loader           bhyveload
name             freebsd-current
os               default
persist          1
ram              2048M
size             35G
tap              tap0
template         NO
utc              YES
vnc              NO
vnc_h            600
vnc_ip           127.0.0.1
vnc_tablet       NO
vnc_w            800
vnc_wait         NO

My iohyve configuration in rc.conf is just a few lines, I let iohyve to do the configuration
Code: 
iohyve_enable="YES"
iohyve_flags="kmod=1 net=re0"
kld_list="nmdm vmm"
which creates the following in the ifconfig configuration
Code: 
root@beastie:/home/fnoyanisi # ifconfig
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
    ether 7c:d3:0a:29:96:21
    inet 192.168.1.59 netmask 0xffffff00 broadcast 192.168.1.255
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 10.1.1.13 netmask 0xffffffff
    inet 10.1.1.14 netmask 0xffffffff
    inet 10.1.1.15 netmask 0xffffffff
    inet 10.1.1.16 netmask 0xffffffff
    inet 10.1.1.17 netmask 0xffffffff
    inet 10.1.1.18 netmask 0xffffffff
    inet 10.1.1.19 netmask 0xffffffff
    inet 10.1.1.20 netmask 0xffffffff
    inet 10.1.1.10 netmask 0xffffff00
    inet 10.1.1.12 netmask 0xffffffff
    inet 10.1.1.11 netmask 0xffffffff
    groups: lo
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 7c:5c:f8:fe:ab:aa
    inet 192.168.1.73 netmask 0xffffff00 broadcast 192.168.1.255
    groups: wlan
    ssid ISI-Internet channel 7 (2442 MHz 11g) bssid 8c:15:c7:ba:06:e0
    regdomain FCC country US authmode WPA2/802.11i privacy ON
    deftxkey UNDEF TKIP 2:128-bit txpower 30 bmiss 10 scanvalid 60
    protmode CTS wme roaming MANUAL
    media: IEEE 802.11 Wireless Ethernet OFDM/36Mbps mode 11g
    status: associated
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
    groups: pflog
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: iohyve-bridge-re0
    ether 02:86:c2:97:9d:00
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 8 priority 128 path cost 2000000
    member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 2000000
    member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
    groups: bridge
    nd6 options=1<PERFORMNUD>
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: iohyve-debian--
    options=80000<LINKSTATE>
    ether 00:bd:91:05:f7:01
    groups: tap
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    Opened by PID 1952
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: iohyve-freebsd-current--
    options=80000<LINKSTATE>
    ether 00:bd:cd:07:f7:00
    groups: tap
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    Opened by PID 2129
root@beastie:/home/fnoyanisi #

The network configuration in the FreeBSD-CURRENT host is given below (IP is statically configured in rc.conf)
Code: 
root@patch:~ # ifconfig
vtnet0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether 00:a0:98:4b:80:ec
        inet 192.168.1.50 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet 10Gbase-T <full-duplex>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
root@patch:~ # netstat -r
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.254      UGS      vtnet0
localhost          link#2             UH          lo0
192.168.1.0/24     link#1             U        vtnet0
192.168.1.50       link#2             UHS         lo0

Internet6:
Destination        Gateway            Flags     Netif Expire
::/96              localhost          UGRS        lo0
localhost          link#2             UH          lo0
::ffff:0.0.0.0/96  localhost          UGRS        lo0
fe80::/10          localhost          UGRS        lo0
fe80::%lo0/64      link#2             U           lo0
fe80::1%lo0        link#2             UHS         lo0
ff02::/16          localhost          UGRS        lo0
 
fnoyanisi said:
I noticed that iohyve creates three interfaces inside the bridge0, which seems a bit weird to me.
Click to expand...
Two interfaces are your bhyve guests (tap0,tap1) and the other interface (re0) is the bridge's uplink. What's weird about that?
 
This is a problem with routing table. You may want to set rc.local to add new routes via specific interfaces.

And given that all VMs, bridge(re0) and outgoing interface(vtnet0) share the same network, that makes the case easier. Have you tried adding the outgoing interface to the bridge0? Usually, you would do that in rc.conf:
Code: 
cloned_interfaces="bridge0 tun0 lo0...."
ifconfig_bridge0="addm vtnet0 addm re0 addm...."
 
I removed the the manual configuration from the rc.conf and let iohyve to deal with the networking for the VMs, which works for all the VMs except the freebsd-current.

I tried to install a new FreeBSD-CURRENT image and it works okay out of the box. I compared rc.conf and other settings and they seem to be the same for the new and the old (not working) VMs.

It must be something with tap0 on the host side I think, there is nothing else I can think of.

I will stop chasing the issue and use the new FreeBSD-CURRENT VM I just installed. I use these VMs for testing patches and I think the new VM will serve the purpose.

Thanks for the help!
 
You do know that -CURRENT is an unsupported development version? It's basically the developer's playground, any part of it can break at any given time.
 
You must log in or register to reply here.
Back
Top