Solved bhyve bridge no route to host, what am i missing ?

NapoleonWils0n

NapoleonWils0n

im following the handbook and trying to create a bhyve vm

freebsd 14.1 p6

docs.freebsd.org

Chapter 24. Virtualization

Virtualization software allows multiple operating systems to run simultaneously on the same computer
docs.freebsd.org docs.freebsd.org

but i cant get the bridge to work
and get no route to host with ping

im using a lagg failover device with ethernet and wifi
just wonder if that causing an issue

first time trying bhyve
i dont think i missed anything from the handbook

1 - install packages

Code: 
doas pkg install bhyve-firmware grub2-bhyve edk2-bhyve

2 - zfs create datasets

Code: 
doas zfs create -o mountpoint=/vm zroot/vm
doas zfs create -o mountpoint=/vm/iso zroot/vm/iso

3 - rc.conf vm settings

Code: 
doas sysrc kld_list+=vmm
doas sysrc vm_enable=YES
doas sysrc vm_dir="zfs:zroot/vm"

4 - ifconfig

im using a lagg0 failover device with ethernet and wifi

i have to add the lagg0 device to the bridge
if i try and add the ethernet device i get an error

Code: 
doas ifconfig tap0 create
doas sysctl net.link.tap.up_on_open=1
doas ifconfig bridge0 create
doas ifconfig bridge0 addm lagg0 addm tap0
doas ifconfig bridge0 up

Code: 
ifconfig

Code: 
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 127.0.0.1 netmask 0xff000000
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
wlan0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=0
    ether 78:2b:46:ee:27:a3
    groups: wlan
    ssid "" channel 11 (2462 MHz 11g)
    regdomain ETSI country GB authmode WPA1+WPA2/802.11i privacy ON
    deftxkey UNDEF txpower 30 bmiss 7 scanvalid 60 protmode CTS wme
    roaming MANUAL
    parent interface: iwlwifi0
    media: IEEE 802.11 Wireless Ethernet autoselect (autoselect)
    status: no carrier
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo1: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 10.10.0.1 netmask 0xffffff00
    inet 10.10.0.2 netmask 0xffffffff
    inet 10.10.0.3 netmask 0xffffffff
    inet 10.10.0.4 netmask 0xffffffff
    inet 10.10.0.6 netmask 0xffffffff
    inet 10.10.0.7 netmask 0xffffffff
    inet 10.10.0.8 netmask 0xffffffff
    inet 10.10.0.9 netmask 0xffffffff
    inet 10.10.0.10 netmask 0xffffffff
    inet 10.10.0.11 netmask 0xffffffff
    inet 10.10.0.12 netmask 0xffffffff
    inet 10.10.0.5 netmask 0xffffff00
    inet6 fe80::1%lo1 prefixlen 64 scopeid 0x4
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ue0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=80008<VLAN_MTU,LINKSTATE>
    ether 78:2b:46:ee:27:a3
    hwaddr 00:50:b6:10:e9:75
    media: Ethernet autoselect (100baseTX <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lagg0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=0
    ether 78:2b:46:ee:27:a3
    hwaddr 00:00:00:00:00:00
    inet 192.168.1.131 netmask 0xffffff00 broadcast 192.168.1.255
    laggproto failover lagghash l2,l3,l4
    laggport: wlan0 flags=0<>
    laggport: ue0 flags=5<MASTER,ACTIVE>
    groups: lagg
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
tap0: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=80000<LINKSTATE>
    ether 58:9c:fc:10:07:7c
    groups: tap
    media: Ethernet 1000baseT <full-duplex>
    status: no carrier
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=0
    ether 58:9c:fc:10:ff:e1
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 2000000
    member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 3 priority 128 path cost 200000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>

5 - fetch the iso

Code: 
cd /vm/iso
doas fetch 'https://download.freebsd.org/releases/ISO-IMAGES/14.2/FreeBSD-14.2-RELEASE-amd64-bootonly.iso'

6 - truncate

Code: 
doas truncate -s 16G freebsd14.img

7 - start the vm

from the /vm/iso directory

Code: 
doas sh /usr/share/examples/bhyve/vmrun.sh -c 1 -m 1024M -t tap0 -d freebsd14.img \
-i -I FreeBSD-14.2-RELEASE-amd64-bootonly.iso freebsd

8 - boot to shell

select shell from install, shell, live

running ping

Code: 
ping 8.8.8.8

shows no route to host

trying the install option also has no route to host

settings:

Code: 
/etc/rc.conf

relevant settings

Code: 
# laggo failover device
# ethernet mac address set to wifi mac address
ifconfig_ue0="ether 78:2b:46:ee:27:a3"
wlans_iwlwifi0="wlan0"
ifconfig_wlan0="WPA"
create_args_wlan0="country GB"
cloned_interfaces="lagg0 lo1"
ifconfig_lagg0="up laggproto failover laggport ue0 laggport wlan0 DHCP"

# bhyve
vm_enable="YES"
vm_dir="zfs:zroot/vm"

also tried adding the following settings

Code: 
# laggo failover device
# ethernet mac address set to wifi mac address
ifconfig_ue0="ether 78:2b:46:ee:27:a3"
wlans_iwlwifi0="wlan0"
ifconfig_wlan0="WPA"
create_args_wlan0="country GB"
cloned_interfaces="lagg0 lo1 bridge0 tap0"
ifconfig_bridge0="addm lagg0 addm tap0"
ifconfig_lagg0="up laggproto failover laggport ue0 laggport wlan0 DHCP"

# bhyve
vm_enable="YES"
vm_dir="zfs:zroot/vm"

pf.conf

Code: 
/etc/pf.conf

Code: 
#=========================================================================#
# variables, macro and tables                                             #
#=========================================================================#

int_if="lagg0" # lagg0 failover
#int_if="ue0" # usb to ethernet adaptor
#int_if="bge0" # thunderbolt to ethernet adaptor
#int_if="wlan0" # iwlwifi
#int_if="wlan1" # ralink usb wifi
vpn_if="tun0" # vpn interface
all_networks="0.0.0.0/0"
vpn_network="$vpn_if:network"
# 6881, 6882 = transmission. 22000, 21025 = syncthing
tcp_services = "{ ntp, 6881, 22000 }" # tcp services - torrent
udp_services = "{ ntp, 6882, 21025 }" # udp services - torrent
icmp_types = "{ echoreq, unreach }"
tcp_state="flags S/SA keep state"
udp_state="keep state"

#table <internet> { $all_networks, !self, !$int_if:network } # internet
#table <lan> { $int_if:network, !self }                      # lan network
table <myself> { self }                                     # self
table <martians> { 0.0.0.0/8 10.0.0.0/8 127.0.0.0/8 169.254.0.0/16     \
            172.16.0.0/12 192.0.0.0/24 192.0.2.0/24 224.0.0.0/3 \
            192.168.0.0/16 198.18.0.0/15 198.51.100.0/24        \
            203.0.113.0/24 }                         # broken networks

#=========================================================================#
# global policy                                                           #
#=========================================================================#

set block-policy drop
set loginterface $int_if
set fingerprints "/etc/pf.os"
set skip on lo0
scrub in all fragment reassemble no-df max-mss 1440
# nat jail
nat on $int_if from {lo1:network} to any -> ($int_if)
antispoof log quick for { lo $int_if } label "block_spoofing"

#=========================================================================#
# block                                                                   #
#=========================================================================#

block log all # block log all
block return out quick inet6 all tag IPV6 # block ipv6
block in quick inet6 all tag IPV6 # block ipv6

# block broken networks - turned off for synergy
# block in quick from { <martians> no-route urpf-failed } to any tag BAD_PACKET

#=========================================================================#
# anchors                                                                 #
#=========================================================================#

# emerging threats - anchor
#anchor "emerging-threats"
#load anchor "emerging-threats" from "/etc/pf.anchors/emerging-threats"

# openvpn - anchor
anchor "openvpn"

#=========================================================================#
# traffic tag                                                             #
#=========================================================================#

# icmp
pass inet proto icmp all icmp-type $icmp_types keep state tag ICMP

# Allow the tcp and udp services defined in the macros at the top of the file
pass in on $int_if inet proto tcp from any to ($int_if) port $tcp_services $tcp_state tag TCP_IN
pass in on $int_if inet proto udp from any to ($int_if) port $udp_services $udp_state tag UDP_IN

# outbound traffic
block out on $int_if all
pass out quick on $int_if all modulate state
#pass out quick on $int_if from <myself> to <lan> modulate state tag LAN_OUT
#pass out quick on $int_if from <myself> to <internet> modulate state tag INTERNET_OUT
 
I'm also starting my bhyve journey and found the basic bridge stuff works as documented.

So maybe (if you can) go for a simpler start - without the lagg and without pf etc.

Then fewer moving parts to work through - get that simpler set-up working and then build on it - lagg and then pf etc.
 
Are you sure tap0 stays up after it's added to the bridge? It's kind of the wrong order...
 
i just followed the handbook
first time playing around with byhve

have i got something arse about elbow

with the vm started

Code: 
tap0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:07:7c
        groups: tap
        media: Ethernet 1000baseT <full-duplex>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 91993
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=0
        ether 58:9c:fc:10:ff:e1
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000000
        member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 200000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
 
ifconfig in the byhve vm with the shell option

Code: 
vtnet0: flags=1008802<BROADCAST,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether 00:a0:98:a1:9d:24
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
 
in the bhyve vm using the shell option

run dhclient with vtnet0

Code: 
dhclient vtnet0

Code: 
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether 00:a0:98:a1:9d:24
        inet 192.168.1.141 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

ping now works

have to see if install works
 
install now working

not sure what the issue was or is
but im getting further than i did before


20241204_21h54m04s_grim.png
 
running ifconfig after booting into freebsd
to set the bhyve jail works

Code: 
doas ifconfig tap0 create
doas sysctl net.link.tap.up_on_open=1
doas ifconfig bridge0 create
doas ifconfig bridge0 addm lagg0 addm tap0
doas ifconfig bridge0 up

with these settings in my rc.conf

Code: 
# laggo failover device
# ethernet mac address set to wifi mac address
ifconfig_ue0="ether 78:2b:46:ee:27:a3"
wlans_iwlwifi0="wlan0"
ifconfig_wlan0="WPA"
create_args_wlan0="country GB"
cloned_interfaces="lagg0 lo1"
ifconfig_lagg0="up laggproto failover laggport ue0 laggport wlan0 DHCP"
 
after about a gallon of tea
i had a crazy idea

with jails you can mount the /tmp directory from the host to the jail

so applications launched in the jail use the Wayland or X11 socket
and are displayed on the Freebsd host

you can share zfs datasets with nfs


which got me thinking
what about sharing the zroot/tmp zfs dataset from the Freebsd host with nfs

and then mounting it in the bhyve vm
to allow access to the Wayland or X11 socket on the freebsd host

so instead of using a vnc client
you could use the Wayland or X11 socket from the host

and applications or a whole desktop run in the vm
would be displayed natively on Freebsd

i can ping from the bhyve to the Freebsd host and vice versa
so the connection works both ways

you would also need to mount the XDG_RUNTIME_DIR from the freebsd host to the jail
and set the DBUS_SESSION_BUS_ADDRESS in the vm so it matches the address on the freebsd host

in theory it might just work
 
You must log in or register to reply here.
Back
Top