Having difficulty with getting network connectivity to a bhyve instance

B

BeeEsDee

I'm currently playing around with bhyve for the first time and am trying to setup networking. My objective is to have this bhyve instance live in its own network concealed within my machine and use the host for reaching the outside world.

This is what I have so far:

Code: 
# Create the bridge.

ifconfig bridge create
ifconfig bridge0 name vswitch
iconfig vswitch up
ifconfig vswitch addm em0
ifconfig vswitch inet 172.77.15.1/24

# Create the tap interface.

ifconfig tap0 create
ifconfig vswitch addm tap0
I have configured my pf.conf as follows:

Code: 
vms_network="172.77.15.1/24"
...
nat on em0 from $vms_network to any -> (em0)
...

And then I start my vm with as follows:

Code: 
sudo bhyve -c 2 -m 4G -w -H \
    -s 0,hostbridge \
    -s 3,ahci-cd,/somewhere/on/disk/archlinux-2021.07.01-x86_64.iso \
    -s 4,virtio-blk,/dev/zvol/pool-1/bhyve/vms/arch \
    -s 5,virtio-net,tap0 \
    -s 29,fbuf,tcp=0.0.0.0:5900,w=1920,h=1080,wait \
    -s 30,xhci,tablet \
    -s 31,lpc -l com1,stdio \
    -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
    arch

And then this is about where I get lost. I can connect to the VM through VNC and it boots successfully, but then it has no network connection. My intention was to set everything statically... which I think I'm supposed to do within the VM's OS? Trying to do so doesn't fix anything (by which I mean setting a static IP of 172.77.15.2 with 172.77.15.1 as the default gateway). I feel like I'm probably missing a lot.

Could someone help me out by walking me through the configuration I should be using to set up what I want? Thanks!
 
Have you tried pinging your guest from your host? Does that work? Can your guest ping your host? Lets first try to get guest/host communications working, before moving to NAT.
You might still want to check your rc.conf or sysctl.conf whether you enabled packet forwarding. If that isn't turned on, NAT won't work.
 
The most simple way I managed to do it is to leave away the bridge and just create a tap device. Then put an IP adress on that tap device (on the host), put the neighbour IP address on the vtnet0 interface (in the guest), both a netmask of 0xfffffffc, and then that works.
The disadvantage is that now all guest traffic must be routed through the host (layer3, IP-stack), which is a performance penalty. The more effective way is to use a bridge, so the guest can directly access the outbound interface and outside world.

BeeEsDee said:
to have this bhyve instance live in its own network concealed within my machine and use the host for reaching the outside world.
Click to expand...
If that is the case and the guest uses a separate network that is not visible (or route-able) on the outside, then this must be someway nat'ed on the host. Then the traffic must be routed through the host anyway, and it appears to me somehow pointless to bridge it to the outbound interface. (At least I can't figure how that might work.)

I am doing just that, and I am using a bridge only to connect multiple guests together (and to the host). The outside physical netif is then a different matter, and that is where the nat works (and rewrites all packets to the host's own IP). But then I am using netgraph bridges, and ipfw - because, honestly, I don't understand the other stuff. YMMV.
 
cmoerz said:
Have you tried pinging your guest from your host? Does that work? Can your guest ping your host? Lets first try to get guest/host communications working, before moving to NAT.
You might still want to check your rc.conf or sysctl.conf whether you enabled packet forwarding. If that isn't turned on, NAT won't work.
Click to expand...
Ah, I hadn't enabled packet forwarding! I thought I had as I have jails using shared IPs on lo1 currently NATed through em0. Not sure how that's been working all this time o_O. Anyway, now everything works! Thanks!

PMc said:
Then the traffic must be routed through the host anyway, and it appears to me somehow pointless to bridge it to the outbound interface.
Click to expand...

I was wondering about this. I started without adding em0 to the bridge, but added it later when nothing was working. Do I not need to do that at all? Is the NAT enough to somehow carry the data between the bridge and the outside world without "physically connecting" the bridge to em0?
 
BeeEsDee said:
I was wondering about this. I started without adding em0 to the bridge, but added it later when nothing was working. Do I not need to do that at all? Is the NAT enough to somehow carry the data between the bridge and the outside world without "physically connecting" the bridge to em0?
Click to expand...
If you connect em0 to the bridge you have a direct connection to the 'outside' world. If you don't add em0 then NAT translates all traffic to the address of the host. It's just a different network setup. If you have a 'direct' connection you're going to need to fix your routing, so other hosts on your network can find the network that's used for the VMs. This is just basic TCP/IP networking.
 
Alain De Vos said:
Or use vm-bhyve it does networking for you.
Click to expand...
This is no longer the case for NAT-based configurations.

Bash: 
$ sudo vm switch nat public on
/usr/local/sbin/vm: WARNING: internal nat support is currently disabled
/usr/local/sbin/vm: WARNING: please add an address to the virtual switch and configure your firewall for NAT manually
 
Appreciating the advice, here.

Trying this out with an Arch Linux installation under bhyve, I believe that the following might work out as an alternative to adding an address and subnet for the tap interface added by bhyve - it was `tap3` here. I'd named the vm switch `vmsw0`. On the FreeBSD host, after the tap device was created with vm-bhyve, I simply added the tap device manually to the switch created with vm-bhyve

(Update: This has to be repeated after ever VM guest reboot, to add the VM's tap device to the VM's switch)

Bash: 
vm switch add vmsw0 tap3

At this point, albeit, I'm not certain as to which command resulted in that tap3 device being created.

Bash: 
[root(u1000)@riparian /opt/bhyve/vm ]# ifconfig tap3

tap3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet-trundle-0-public
        options=80000<LINKSTATE>
        ether 00:bd:97:41:d4:03
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 11368

I'm certain it was created with one of the vm switch commands.

Beginning from the Bhyve arch template at `/usr/local/share/examples/vm-bhyve/arch.conf` the enp0s5 network interface under the Arch linux guest has been configured to be on the same subnet as the single physical ethernet address (re0) on the bhyve host (FreeBSD). This was in using the `net-tools` package from Arch, which provides a normal `ifconfig` cmd.

The vm-vmsw0 interface on the bhyve host has an address for a different subnet than the bhyve guest, and re0/ethernet interfaces, but this still works out. (it might be able to reach any subnet for an interface added to the vm switch?)

Bash: 
root(u1000)@riparian /opt/bhyve/vm ]# ifconfig re0

re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether c4:34:6b:48:45:9d
        inet 192.168.50.5 netmask 0xfffffc00 broadcast 192.168.51.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

root(u1000)@riparian /opt/bhyve/vm ]# ifconfig vm-vmsw0

vm-vmsw0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether a2:14:80:ed:19:8b
        inet 192.168.100.100 netmask 0xffffff00 broadcast 192.168.100.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 10 priority 128 path cost 2000000
        member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 200000
        groups: bridge vm-switch viid-3edf0@
        nd6 options=1<PERFORMNUD>

The tap3 and re0 interfaces have both been added to the switch, using the 'vm switch add' cmd from the vm-bhyve port. The configuration for the switch, under vm-bhyve:

Bash: 
root(u1000)@riparian /opt/bhyve/vm ]# vm switch info vmsw0

------------------------
Virtual Switch: vmsw0
------------------------
  type: standard
  ident: vm-vmsw0
  vlan: -
  physical-ports: re0 tap3
  bytes-in: 123126 (120.240K)
  bytes-out: 55495583 (52.924M)

  virtual-port
    device: tap3
    vm: trundle


After configuring an address and subnet for the vm switch interface (vm-vmsw0 here) using ifconfig on the host, then adding the host's ethernet interface and the VM's tap interface to the VM switch manually, it might not be absolutely needed to define a subnet for the VM, with this configuration? With that tap interface and the host's physical interface added to the vm switch, the VM guest can then access the same subnet as the physical interface on the host. It may go into NAT from there?

The VM can access the SSH service on the VM host. The host can access sshd under the VM. Using ipfw for NAT on the host, the VM can also access the Internet.

Update: It seems that I need to run vm switch add vmsw0 tap3 again after every reboot of the virtual guest. No additional subnets here though ..

I'm glad someone mentioned that TAP device, I wasn't sure what it was there for, lol.
 
I am trying to setup Whonix gateway and Whonix workstation in bhyve. After long hours of playing with vm-bhyve switches i just decided to use native ifconfig bridge. Currently i am having three VMs. Archlinux, WGateway and WWorkstation. My ifconfig has been build like:

ifconfig bridge0 create
ifconfig bridge0 addm re0 up


Create tap interfaces:
ifconfig tap0 create
ifconfig bridge0 addm tap0 up

To make it persistent:
My /etc/rc.conf is
Code: 
cloned_interfaces="bridge0 tap0"
ifconfig_bridge0="addm tap0 addm re0 up"
ipfw_enable="NO"
firewall_enable="NO"
gateway_enable="YES"

My /boot/loader.conf has
Code: 
if_bridge_load="YES"
bridgestp_load="YES"

My Gateway VM config has:
Code: 
             loader="uefi"
             cpu=2
             memory=2G
             network0_type="virtio-net"
             network0_device="tap0"
             network1_type="virtio-net"
             network1_switch="internal"  # for Workstation
             disk0_type="virtio-blk"
             disk0_name="disk0.img"
I was testing tap0 by temporarly adding to my archlinux .conf file and the arch VM is getting network, therefore tap0 is working fine. However gateway not.
I was also able to getting network with
Code: 
vm switch create -t manual -b bridge0 public
vm switch create -t manual -b bridge0 external
vm switch create internal
but the result had the same. Working network with arch VM but not working with WhonixGateway. So my question is what else need to configure in the host or in gateway to have network??
 
tyler19820201 you have the same question in another post!
S

bhyve Thread 'Help with bhyve networking'

Hi friends,
I decided to use FreeBSD as a daily and it has been great so far (figured out lots on my own with some help from this great community of course!).
But I have something that has me tearing out my hair - I have VM's on bhyve and they run beautifully but for one thing: networking.
I cannot seem to figure out how to get them online!!!
This despite having almost the same system but on a server in the house that uses a cabled connection and works beautifully!

This problem is on my laptop.
Now, what do I do?
Can someone please help me and tell me what goes where or point out what I...

I just answered there.
 
You must log in or register to reply here.
Back
Top