jails Set up iocage jail with two vnet interfaces

byrnejb

byrnejb

On a FreeBSD-12.3p5 host I have an iocage administered jail for which I would like to configure two separate i/f. I have this setup working for a bhyve administered vm on that host but I also need a jail. Here is what I have:

rc.conf:
Code: 
cloned_interfaces="bridge0"
ifconfig_bridge0="addm igb0 up"
ifconfig_igb0="inet A.B.71.41/25 up"
defaultrouter="A.B.71.1"

iocage:
Code: 
ip4_addr:vnet0|A.B.71.124/25,vnet1|192.168.216.124/16
vnet:1
vnet0_mac:7085c214e667 7085c214e668
vnet1_mac:7085c214e331 7085c214e332
vnet2_mac:none
vnet3_mac:none
vnet_default_interface:auto
vnet_interfaces:none

On the Jail I see this:
Code: 
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 70:85:c2:14:e6:68
    hwaddr 02:d3:4b:06:64:0b
    inet 216.185.71.124 netmask 0xffffff80 broadcast 216.185.71.127
    inet6 fe80::7285:c2ff:fe14:e668%epair0b prefixlen 64 tentative scopeid 0x3
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

On the host I see this:
Code: 
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6>
    ether 70:85:c2:da:88:4f
    inet A.B.71.41 netmask 0xffffff80 broadcast A.B.71.127
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
. . .
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vm-vm_public_net
    ether 02:1a:39:ed:22:00
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 2000
    member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 2000000
    groups: bridge vm-switch viid-8b9e1@
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: sshpipe-3 as nic: epair0b
    options=8<VLAN_MTU>
    ether 70:85:c2:14:e6:67
    hwaddr 02:d3:4b:06:64:0a
    inet6 fe80::7285:c2ff:fe14:e667%vnet0.5 prefixlen 64 scopeid 0x5
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Without additional configuration iocage evidently creates and adds epair0b to the jail and vnet0.5 to the host with both belonging to the epair group and using the i gb0 interface. How does one go about adding the second epair1b and vnet1.X to use the bridge0 interface? I have tried discovering what, if any, importance the vnet_interfaces option has on this but nothing that I have found makes things any clearer.

If I try this:
Code: 
 iocage set vnet_interfaces="igb0,bridge0" sshpipe-3
vnet_interfaces: none -> igb0,bridge0

Then the jail fails to start:
Code: 
iocage start sshpipe-3
* Starting sshpipe-3
  + Start FAILED
ifconfig: interface igb0,bridge0 does not exist
jail: ioc-sshpipe-3: /sbin/ifconfig igb0,bridge0 vnet ioc-sshpipe-3: failed

If I try this:
Code: 
iocage set vnet_interfaces="igb0" sshpipe-3
vnet_interfaces: none -> igb0

When I start the jail then the host loses connectivity on igb0.

What is required to specify two independent vnet i/fs in iocage? Do I need to create the epairs in rc.conf and put these in vnet_interfaces?

When I tried this:
Code: 
cloned_interfaces="bridge0 epair0 epair1"
ifconfig_bridge0="addm igb0 epair0a epair1a up"
ifconfig_igb0="inet A.B.71.41/25 up"
ifconfig_epair0a="up"
ifconfig_epair1a="up"
defaultrouter="A.B.71.1"

Together with this:
Code: 
iocage set vnet_interfaces="epair0b epair1b" sshpipe-3
vnet_interfaces: none -> epair0b epair1b

Starting the jail gives this:
Code: 
iocage start sshpipe-3
* Starting sshpipe-3
  + Started OK
  + Using devfs_ruleset: 1005 (iocage generated default)
  + Configuring VNET OK
  + Using IP options: vnet vnet.interface=epair0b vnet.interface=epair1b
  + Starting services OK
  + Executing poststart OK

Which results in this:
Code: 
. . .
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6>
    ether 70:85:c2:da:88:4f
    inet 216.185.71.41 netmask 0xffffff80 broadcast 216.185.71.127
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:1a:39:ed:22:00
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.12 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 8 priority 128 path cost 2000
    member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
epair0a: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:80:11:a4:22:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair0b: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:80:11:a4:22:0b
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair1a: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:e6:d0:56:cb:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vnet0.12: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: sshpipe-3 as nic: epair0b
    options=8<VLAN_MTU>
    ether 70:85:c2:14:e6:67
    hwaddr 02:7a:32:44:90:0a
    inet6 fe80::7285:c2ff:fe14:e667%vnet0.12 prefixlen 64 scopeid 0x8
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Which provides only one vnet i/f.

It seems to me that iocage is overwriting vnet.interface with the last value given to vnet_interfaces. The examples that I have found use net.interfaces+=<value>. I thought that vnet.interface might take a comma separated list, but using this:
Code: 
iocage set vnet_interfaces="epair0b,epair1b" sshpipe-3
vnet_interfaces: epair0b epair1b -> epair0b,epair1b
Results in this:
Code: 
iocage start sshpipe-3
* Starting sshpipe-3
  + Start FAILED
ifconfig: interface epair0b,epair1b does not exist
jail: ioc-sshpipe-3: /sbin/ifconfig epair0b,epair1b vnet ioc-sshpipe-3: failed
 
The iocage man page says this:

Multiple interface format:

interface|ip-address/netmask,interface|ip-address/netmask

On shared IP jails, an interface name given before the IP
address adds an alias to that interface.

A netmask in either dotted-quad or CIDR form given after
the IP address is used when adding the IP alias.

In VNET jails, the interface is configured with the IP
addresses listed.

Example:

"vnet0|192.168.0.10/24,vnet1|10.1.1.10/24"

Interfaces vnet0 and vnet1 are configured in a VNET jail.
In this case, no network configuration is necessary in the
jail's rc.conf file.

Default: none
. . .
interfaces=[vnet0:bridge0,vnet1:bridge1 | vnet0:bridge0]
By default, there are two interfaces specified with their
bridge association. Up to four interfaces are supported.
Interface configurations are separated by commas. The
format is interface:bridge, where the left value is the
virtual VNET interface name and the right value is the
bridge name where the virtual interface should be attached.

Default: vnet0:bridge0,vnet1:bridge1

Source: local
Click to expand...
Which leaves me rather confused as to how the configuration should be done. If I use
Code: 
cloned_interfaces="bridge0 bridge1"
ifconfig_bridge0="addm igb0 up"
ifconfig_bridge1="addm igb0 up"

And set:
Code: 
iocage set vnet_interfaces="none" sshpipe-3

Then I still only get one vnet:
Code: 
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:1a:39:ed:22:00
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.14 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 10 priority 128 path cost 2000
    member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:1a:39:ed:22:01
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.14: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: sshpipe-3 as nic: epair0b
    options=8<VLAN_MTU>
    ether 70:85:c2:14:e6:67
    hwaddr 02:f9:ea:a2:20:0a
    inet6 fe80::7285:c2ff:fe14:e667%vnet0.14 prefixlen 64 scopeid 0xa
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
 
Did you ever get this sorted? I've been doing this same dance all day.

edit: also noting that adding members to a bridge, either by hand while testing or by letting iocage do it results in the interface shutting down for a few seconds, which seems kind of not good?
 
This may help: Thread use-a-virtual-network-interface-on-a-host.76420/post-471919

First, note that my understanding of routing is very limited and what I think I know is undoubtedly flawed.

I do not think that I ever got this to work over a single physical NIC using vnet for jails. What I ended up doing was simply using two physical NICs (em0 and em1) and assigning aliases to each interface for jails that required two networks.

Code: 
Network settings for jail: sshpipe-2
allow_chflags:0
allow_mlock:0
allow_mount:0
allow_mount_devfs:0
allow_mount_fusefs:0
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0[THREAD][THREAD][/THREAD][/THREAD]
allow_mount_zfs:0
allow_quotas:0
allow_raw_sockets:1
allow_set_hostname:1
allow_socket_af:0
allow_sysvipc:0
allow_tun:0
allow_vmm:0
assign_localhost:1
boot:1
host_domainname:none
host_hostname:sshpipe-2.hamilton.harte-lyne.ca
host_hostuuid:sshpipe-2
host_time:1
hostid:53d19f64-d663-a017-8922-003048c5f236
hostid_strict_check:0
ip4:new
ip4_addr:em0|216.185.71.114/25,em1|192.168.216.114/16
ip4_saddrsel:1
ip6:new
ip6_addr:none
ip6_saddrsel:1
ip_hostname:0
jail_zfs_dataset:iocage/jails/sshpipe-2/data
localhost_ip:127.0.114.1
nat_backend:ipfw
notes:Alternate sshd portal and hp3000 pdf virtual printer
resolver:nameserver 216.185.71.33;nameserver 216.185.71.34;search harte-lyne.ca hamilton.harte-lyne.ca;options edns0 timeout:5 attempts:3
state:up

ZFS origin for jail: sshpipe-2
NAME                          PROPERTY  VALUE   SOURCE
zroot/iocage/jails/sshpipe-2  origin    -       -

However, when dealing with vnets for BHyve I have this in my rc.conf:
Code: 
## Setup a bridge to enable vnet
## VNET Jails (iocage) - also see loader.conf and sysctl.conf settings

cloned_interfaces="bridge0"
ifconfig_bridge0="addm igb0 igb1 up"
ifconfig_igb0="inet 216.185.71.41/25 up"
ifconfig_igb1="inet 192.168.216.41/16 up"
defaultrouter="216.185.71.1"

And I leave vm-bhyve to take care of the epairs.
 
byrnejb said:
Code: 
cloned_interfaces="bridge0" 
ifconfig_bridge0="addm igb0 igb1 up" 
ifconfig_igb0="inet 216.185.71.41/25 up" 
ifconfig_igb1="inet 192.168.216.41/16 up"
Click to expand...
You are creating a bridge (a layer 2 connection) with igb0 (external WAN?) and igb1 (internal LAN?). That's a really bad idea.
 
I'm not sure which bit of fiddling got me to the right place, but I do have it working currently.
This is the iocage config for the jail:

JSON: 
{
    "allow_raw_sockets": 1,
    "basejail": 1,
    "boot": 1,
    "defaultrouter": "xxx.xxx.xxx.225",
    "host_hostname": "haproxy01",
    "host_hostuuid": "haproxy01",
    "interfaces": "vnet0:bridge0,vnet1:bridge1",
    "ip4_addr": "vnet0|xxx.xxx.xxx.246/27,vnet1|10.99.88.240/24",
    "jail_zfs_dataset": "iocage/jails/haproxy01/data",
    "last_started": "2023-08-31 03:56:44",
    "release": "13.2-RELEASE-p2",
    "vnet": 1,
    "vnet0_mac": "3cecefc98f34 3cecefc98f35",
    "vnet1_mac": "3cecef659a71 3cecef659a72",
    "vnet_default_interface": "ext0"
}

And the relevant HOST rc.conf stuff:

Code: 
# bridges for vnet jail(s)
cloned_interfaces="bridge0 bridge1"
ifconfig_bridge0="addm ext0 up"
ifconfig_bridge1="addm int0 up"

I *believe* the "vnet_interfaces" iocage parameter specifying the bridges (and letting iocage create them and add the pairs to them) was what was throwing me off.

I also have CARP running in this jail (and another just like it). With CARP, it was important to set all this stuff in the HOST's sysctl.conf or the CARP advertisements just never made it out of the epair(4) (and even watching pf drops, there was no logging of the advertisements being blocked):

Code: 
# VNET jails (required for CARP in a jail)
net.link.bridge.pfil_onlyip=0  # Only pass IP packets when pfil is enabled
net.link.bridge.pfil_bridge=0  # Packet filter on the bridge interface
net.link.bridge.pfil_member=0  # Packet filter on the member interface

And for anyone finding this via google and trying to do CARP in a jail, I'll note this is how I set the CARP IP for the jail - it can't be set using iocage, but it can be set in the JAIL's rc.conf like so (I picked a random VHID to avoid collisions with our upstream who runs HSRP on the interface facing us):

Code: 
ifconfig_epair0b_alias0="vhid 248 pass foofoof00f00 advskew 1 xxx.xxx.xxx.248/27"

And lastly, if you find your net connection freezes up for a few seconds while starting/stopping a jail, this bug report is some good reading:

PR 221122

Basically, if your NIC supports fancy offloading capabilities, well... epair(4) does not, so when it gets added to the bridge, the NIC gets removed and re-added with the offending offloading capabilities disabled. In my case I was seeing about 5-10 second drops on VNET jail stop/start/restart. I was about to try out a few workarounds before I saw updates to this thread.. :)

Oh, byrnejb lastly, as to why bridging internal and external together is bad, well, the reason you have those two networks is usually because you want them to be two different networks. If you put a bridge between them, then they are no longer separate. It would be like going to the datacenter, plugging in a little switch and putting a cable from your external and internal networks into that switch.
 
You must log in or register to reply here.
Back
Top