Solved virtio settings in host rc.conf for two virtual interfaces on one physical i/f

byrnejb · Aug 8, 2022

I need to configure /etc/rc.conf on the FreeBSD-12.3p5 host system with one physical NIC so that two virtio network devices can be configured in the guest. I already have a jail running on this host with one virtio i/f which works.

My current rc.conf relating to networking contains these lines:

Code:

### Networking
## Setup a bridge to enable vnet
## VNET Jails (iocage) - also see loader.conf and sysctl.conf settings

cloned_interfaces="bridge0"
ifconfig_bridge0="addm igb0 up"
ifconfig_igb0="up"

ifconfig_igb0="inet A.B.216.41/25"

defaultrouter="A.B.216.1"

What I think might be the way to do this is to add these lines to rc.conf:

Code:

cloned_interfaces="bridge_vm0"
cloned_interfaces="bridge_vm1"
Ifconfig_bridge_vm0="addm igb0 addm tap0 up"
ifconfig_bridge_vm1="addm igb0 addm tap1 up"

Is this correct?

SirDice · Aug 8, 2022

byrnejb said:
Code:

cloned_interfaces="bridge_vm0" cloned_interfaces="bridge_vm1"

rc.conf is essentially a shell script with variables. You are redefining the same variable twice.

Code:

#!/bin/sh

foo="bar"
foo="not bar"

echo $foo

byrnejb · Aug 8, 2022

So:

Code:

cloned_interfaces="bridge0 bridge_vm0 bridge_vm1"
ifconfig_bridge0="addm igb0 up"
ifconfig_bridge_vm0="addm igb0 addm tap0 up"
ifconfig_bridge_vm1="addm igb0 addm tap1 up"

Is this the correct way to set up networking on the host to support two virtio-net i/fs on bhyve vm's?

SirDice · Aug 8, 2022

Why are you creating three bridges? And why don't you simply add the second VM to the first bridge? You don't need to create a bridge(4) interface for each VM. You can attach multiple VMs (or jails) to the same bridge (as long as they need to be on the same network this should be fine).

Think of that bridge as a switch. Do you use a separate switch for each machine? Or do you hook up multiple machines to the same switch?

byrnejb · Aug 8, 2022

This vm requires two separate networks; one on the public and one on the private address space. I thought that I would keep the existing bridge separate from those I plan to use for the vm. I inferred from what I had read elsewhere that to have two separate virtual i/fs with separate mac addresses on one vm would required two bridges.

To discover if this is true I why I am asking this question.

SirDice · Aug 8, 2022

byrnejb said:
This vm requires two separate networks; one on the public and one on the private address space.

That's not how you've set up the bridges, you tied everything together on igb0. Thus, they're all on the same network as igb0.

byrnejb · Aug 8, 2022

This is why I am asking. The host only has one physical interface. Is it impossible to have that interface bridged with two separate virtual interfaces on one vm each with its own mac?

SirDice · Aug 8, 2022

byrnejb said:
This vm requires two separate networks; one on the public and one on the private address space. I thought that I would keep the existing bridge separate from those I plan to use for the vm. I inferred from what I had read elsewhere that to have two separate virtual i/fs with separate mac addresses on one vm would required two bridges.

Right, I think I understand what you're trying to do.

Code:

cloned_interfaces="bridge0 bridge1"
ifconfig_bridge0="addm igb0 addm tap0 up"
ifconfig_bridge1="addm tap1 up"
ifconfig_igb0="inet A.B.216.41/25"

That will give you two virtual interfaces; tap0 and tap1. Both of them connected to the same VM, tap0 is the first virtio-net interface, tap1 the second. The traffic on bridge1 is your 'local' private network. You can connect other VMs (or jails) to this 'local' network for example.

byrnejb · Aug 9, 2022

Here is the current status:

1. rc.conf contains this:

Code:

cloned_interfaces="bridge0 bridge1 bridge2"
ifconfig_bridge0="addm igb0 up"
ifconfig_bridge1="addm tap0 up"
ifconfig_bridge2="addm tap1 up"
ifconfig_igb0="inet A.B.71.41/25 up"

2. vm switch list shows this:

Code:

NAME            TYPE    IFACE    ADDRESS  PRIVATE  MTU  VLAN  PORTS
vm_public_net   manual  bridge1  n/a      no       n/a  n/a   n/a
vm_private_net  manual  bridge2  n/a      no       n/a  n/a   n/a

3. ifconfig shows this:

Code:

vnet0.1: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: sshpipe-3 as nic: epair0b
    options=8<VLAN_MTU>
    ether 70:85:c2:14:e6:67
    hwaddr 02:b3:c7:29:05:0a
    inet6 fe80::7285:c2ff:fe14:e667%vnet0.1 prefixlen 64 tentative scopeid 0x5
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:1a:39:ed:22:00
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vm-vm_public_net
    ether 02:1a:39:ed:22:01
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    groups: bridge vm-switch viid-8b9e1@
    nd6 options=9<PERFORMNUD,IFDISABLED>
bridge2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vm-vm_private_net
    ether 02:1a:39:ed:22:02
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    groups: bridge vm-switch viid-18470@
    nd6 options=9<PERFORMNUD,IFDISABLED>

4. /zroot/vm/vbsd-01/ contains this:

Code:

loader="bhyveload"
cpu=4
memory=8G
utctime=yes
network0_type="virtio-net"
network0_switch="vm_public_net"
network1_type="virtio-net"
network1_switch="vm_private_net"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"
uuid="c2a4cca4-17f5-11ed-b580-7085c2da884f"
network0_mac="58:9c:fc:01:0a:5c"
network1_mac="58:9c:fc:0e:bc:40"

vbsd-01 was created with this command:

 NGUEST=vbsd-01 ;  vm create  -s 16G -t freebsd-zvol-4x8 $NGUEST &&  vm install $NGUEST FreeBSD-12.3-RELEASE-i386-dvd1.iso

5. after starting vbsd-01 ifconfig shows this:

Code:

vnet0.1: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: sshpipe-3 as nic: epair0b
    options=8<VLAN_MTU>
    ether 70:85:c2:14:e6:67
    hwaddr 02:b3:c7:29:05:0a
    inet6 fe80::7285:c2ff:fe14:e667%vnet0.1 prefixlen 64 tentative scopeid 0x5
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:1a:39:ed:22:00
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vm-vm_public_net
    ether 02:1a:39:ed:22:01
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 8 priority 128 path cost 2000000
    groups: bridge vm-switch viid-8b9e1@
    nd6 options=9<PERFORMNUD,IFDISABLED>
bridge2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vm-vm_private_net
    ether 02:1a:39:ed:22:02
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 9 priority 128 path cost 2000000
    groups: bridge vm-switch viid-18470@
    nd6 options=9<PERFORMNUD,IFDISABLED>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vmnet-vbsd-01-0-vm_public_net
    options=80000<LINKSTATE>
    ether 58:9c:fc:10:60:30
    inet6 fe80::5a9c:fcff:fe10:6030%tap0 prefixlen 64 tentative scopeid 0x8
    groups: tap vm-port
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    Opened by PID 6797
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vmnet-vbsd-01-1-vm_private_net
    options=80000<LINKSTATE>
    ether 58:9c:fc:10:ff:d5
    inet6 fe80::5a9c:fcff:fe10:ffd5%tap1 prefixlen 64 scopeid 0x9
    groups: tap vm-port
    media: Ethernet autoselect
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    Opened by PID 6797

6. vm list shows this:

Code:

NAME      DATASTORE  LOADER     CPU  MEMORY  VNC  AUTOSTART  STATE
vbsd-01   default    bhyveload  4    8G      -    No         Running (6797)

7. ifconfig on vbsd-01 shows this:

Code:

vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether 58:9c:fc:01:0a:5c
        inet A.B.71.99 netmask 0xffffff80 broadcast 216.185.71.127
        media: Ethernet 10Gbase-T <full-duplex>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vtnet1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether 58:9c:fc:0e:bc:40
        inet 192.168.216.99 netmask 0xffff0000 broadcast 192.168.255.255
        media: Ethernet 10Gbase-T <full-duplex>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

and netstat -rn shows:

Code:

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default           A.B.71.1       UGS      vtnet0
127.0.0.1          link#3             UH          lo0
192.168.0.0/16     link#2             U        vtnet1
192.168.216.99     link#2             UHS         lo0
A.B.71.0/25    link#1             U        vtnet0
A.B.71.99      link#1             UHS         lo0

However, I cannot reach the network:

Code:

ping 2A.B.71.1
PING A.B.71.1 (A.B.71.1): 56 data bytes
^C
--- 216.185.71.1 ping statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss
root@vbsd-01:~ # ping 192.168.0.1
PING 192.168.0.1 (192.168.0.1): 56 data bytes
ping: sendto: Host is down
^C
--- 192.168.0.1 ping statistics ---
6 packets transmitted, 0 packets received, 100.0% packet loss

bridge0 was already intended for a jail vnet so I used bridge1 and bridge2 instead.

What else is required to get the network connection in the vm?

VladiBG · Aug 9, 2022

You are missing uplink ports for your vm_public_net switch
When you are going to manage your bhyve using vm-bhyve then stick only to it to creating/managing the vm-switches instead of creating them manually.
You can use the same brige0 as your vm_public_net switch as your uplink interface igb0 is already member (connected) to it.

byrnejb · Aug 9, 2022

VladiBG said:
You are missing uplink ports for your vm_public_net switch

Sorry if I appear thick but are you saying that rc.conf should look like this:

Code:

cloned_interfaces="bridge0 bridge1"
ifconfig_bridge0="addm igb0 addm tap0 up"
ifconfig_bridge1="addm bridge0 addm tap1 up"
ifconfig_igb0="inet A.B.71.41/25 up"

and that the vm-switch commands should be:

Code:

vm switch create vm_public_net bridge0
vm switch create vm_private_net bridge1

VladiBG said:
When you are going to manage your bhyve using vm-bhyve then stick only to it to creating/managing the vm-switches instead of creating them manually.
You can use the same brige0 as your vm_public_net switch as your uplink interface igb0 is already member (connected) to it.

https://forums.freebsd.org/threads/...rfaces-on-one-physical-i-f.86097/#post-577781

Code:

Using a custom bridge

Sometimes you may want to configure a bridge interface manually, if you want to use functionality
 not supported directly by vm-bhyve. In this case you can create a bridge interface manually in /etc/rc.conf,
 then import this into vm-bhyve.

# vm switch create -t manual -b bridge0 customswitch

This command assumes you have already created bridge0 manually. When run vm-bhyve will assign a description
 to the bridge interface. This means that if a guest has networkX_switch="customswitch" specified in the
configuration, it will have that interface connected to your custom bridge.

SirDice · Aug 9, 2022

You probably want to keep ifconfig_bridge0 in rc.conf. I understood you already had a jail attached to that one? So it's fine to import just the bridge0 manually in vm(8) as you did. Then as VladiBG mentioned, use vm switch create and let vm(8) create and manage the second bridge.

Code:

cloned_interfaces="bridge0"
ifconfig_bridge0="addm igb0 up"
ifconfig_igb0="inet A.B.71.41/25"

Creating the second bridge:

Code:

# vm create switch vm_private_net

You should end up with bridge0->vm_public_net and a vm_private_net bridge(4) interface.

This is good:

Code:

network0_type="virtio-net"
network0_switch="vm_public_net"
network1_type="virtio-net"
network1_switch="vm_private_net"

That's exactly how you should connect two networks to a VM.

VladiBG · Aug 9, 2022

Yes it's look good, just add the manual vm_public_net switch to brige0 from your /etc/rc.conf

byrnejb · Aug 9, 2022

Here is what I understand to be required:

In rc.conf:

Code:

cloned_interfaces="bridge0"
ifconfig_bridge0="addm igb0 up"
ifconfig_igb0="inet A.B.71.41/25 up"
defaultrouter="A.B.71.1"

In vbsd-01.conf:

Code:

network0_type="virtio-net"
network0_switch="vm_public_net"
network1_type="virtio-net"
network1_switch="vm_private_net"

From the command line:

Code:

vm switch create -t manual -b bridge0 vm_public_net
vm create switch vm_private_net

Now when I start the vm I should have network connectivity, correct?

And it works.

Thank you very much.

Solved virtio settings in host rc.conf for two virtual interfaces on one physical i/f

byrnejb

SirDice

Administrator

byrnejb

SirDice

Administrator

byrnejb

SirDice

Administrator

byrnejb

SirDice

Administrator

byrnejb

VladiBG

byrnejb

SirDice

Administrator

VladiBG

byrnejb