Solved Jails vnet - FreeBSD Mastery examples

tommiie

tommiie

Hey all,

I follow along in Lucas' new book FreeBSD mastery: Jails. Starting on p. 159, I copied over the /usr/share/examples/jail/jib file and make it executable (something Lucas fails to mention). However, I use a different name for my physical interface: lab instead of jailether.
Code: 
ifconfig_em0_name="lab"
ifconfig_lab="up"

Next I follow along and create two test jails:
Code: 
test02 {
        vnet;
        vnet.interface = "e0a_test02";
        exec.prestart += "/usr/local/scripts/jib addm test02 lab";
        exec.poststop += "/usr/local/scripts/jib destroy test02";
}

test03 {
        vnet;
        vnet.interface = "e0a_$name";
        exec.prestart += "/usr/local/scripts/jib addm $name lab";
        exec.poststop += "/usr/local/scripts/jib destroy $name";
}
However, these interfaces do not get added to the labbridge interface that got created. I thus added them manually:

ifconfig labbridge addm e0b_test02 addm e0b_test03 up

Now, although both jails are connected to the bridge (and I've configured IP addresses on them) I still cannot ping one jail from the other.

Code: 
tom:~/ $ sudo jexec test02 ifconfig e0a_test02
e0a_test02: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:50:7f:0c:1a:6a
        hwaddr 02:18:17:5a:62:0a
        inet 203.0.113.2 netmask 0xffffff00 broadcast 203.0.113.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair
tom:~/ $ sudo jexec test03 ifconfig e0a_test03
e0a_test03: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:50:80:0c:1a:6a
        hwaddr 02:54:5d:26:1e:0a
        inet 203.0.113.3 netmask 0xffffff00 broadcast 203.0.113.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair

Anyone got a clue as to what I'm doing wrong?
 
tommiie said:
However, these interfaces do not get added to the labbridge interface that got created. I thus added them manually:
Click to expand...
The jib script requires that the B-side of the epair interfaces be assigned to the jail and the A-side will automatically be assigned to the bridge. I preferred it the other way around (B-side for bridge) but the script doesn't like that. Swapping the two sides, the jib script automatically adds the interface to the bridge. At least that problem got solved.
 
The command arp -an shows the correct MAC addresses:

Host:
root@phoenix:/home/tom # arp -an
? (203.0.113.50) at 02:2b:cd:46:61:00 on labbridge permanent [bridge]
? (203.0.113.2) at 0e:50:7f:0c:1a:6a on labbridge expires in 1156 seconds [bridge]
? (203.0.113.3) at 0e:50:80:0c:1a:6a on labbridge expires in 1165 seconds [bridge]


Jail:
root@test02:~ # arp -an
? (203.0.113.50) at 02:2b:cd:46:61:00 on e0b_test02 expires in 1138 seconds [ethernet]
? (203.0.113.2) at 0e:50:7f:0c:1a:6a on e0b_test02 permanent [ethernet]
? (203.0.113.3) at 0e:50:80:0c:1a:6a on e0b_test02 expires in 93 seconds [ethernet]
 
I figured it out. Disabling PF made everything work. I did not expect PF to block inter-jail traffic as those are interfaces owned by the jails. Anyway... Now I have to ponder how to configure PF to make this work. It's not mentioned in Lucas' book so I'm now reading through The book of PF in the hopes of finding answers, or at least some clues, there.
 
abishai said:
I disabled filtering on bridge and use pf in each jail. ansible generates everything for me, even bootstrap jails :p
Click to expand...
Could you share the relevant PF configuration? How do you disable filtering on the bridge? Edit: Reading if_bridge(4) I found out you can disable it with:
sysctl net.link.bridge.pfil_member=0
sysctl net.link.bridge.pfil_bridge=0

But a tip on how to set this in /etc/rc.conf would be nice.

It would be nice to see our Ansible setup as well, but that would be quite off-topic.
 
Is my setup I have bridge for vnet jails with uplink.
1. Filtering done on uplink interface to secure access to jails.
2. Filtering on bridge is disabled with sysctl net.link.bridge.pfil_member=0 This allows unrestricted jail-to-jail communication.
3. Jails have some autogenerated own pf.conf and pf enabled.
For example, this is for SSL terminator (reverse proxy)
Code: 
ext="e0b_proxy"
int="lo0" #loopback

reserved= "{ 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 255.255.255.255/32 }"
self="10.0.4.11"

set loginterface $ext
set skip on $int

scrub on $ext all
block all

pass out quick on $ext keep state

pass in on $ext inet proto tcp from any to $self port ssh keep state
pass in on $ext inet proto tcp from any to $self port https keep state

In my ansible host_vars I have something like this
Code: 
firewall:
  enabled: true
  ssh: true
  rules:
    - { pass: in, proto: tcp, from: any, to: https}

and some basic template
Code: 
ext="{{ansible_default_ipv4.device}}"
int="lo0" #loopback

reserved= "{ 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 255.255.255.255/32 }"
self="{{ansible_default_ipv4.address}}"

set loginterface $ext
set skip on $int

scrub on $ext all
block all

pass out quick on $ext keep state

{% if firewall.ssh %}
pass in on $ext inet proto tcp from any to $self port ssh keep state
{% endif %}

{% if firewall.custom is defined %}
{% for r in firewall.custom %}
{{ r }}
{% endfor %}
{% endif %}

{% if firewall.rules is defined %}
{% for r in firewall.rules %}
pass {{ r.pass }} on $ext inet proto {{ r.proto }} from {{ r.from }} to $self port {{ r.to }} keep state
{% endfor %}
{% endif %}
 
I fixed it (for now) with:
Code: 
block in on $ext_intf
pass out
So that PF does not filter the internal interfaces. But I will surely analyze your PF and Ansible configurations and implement parts of it. I very much like a rulebase starting with block all. I'll also look into the exact differences between net.link.bridge.pfil_member and net.link.bridge.pfil_bridge and whether setting those values might be better than my current block in on... statement.
 
Last edited:
tommiie said:
Now, although both jails are connected to the bridge (and I've configured IP addresses on them) I still cannot ping one jail from the other.

Anyone got a clue as to what I'm doing wrong?
Click to expand...
  1. If you take a look into the jib script itself, it lists the following usage:
    jib_addm_usage="addm [-b BRIDGE_NAME] NAME [!]iface0 [[!]iface1 ...]"
    So you could use the switch -b and your bridge's name labbridge to get the interfaces added to the bridge. I have not tested this.
  2. Why can't you ping your interfaces?
    Your interfaces are epairs (created by the jib script), correct me if you are doing something else.
    You could use my personal network troubleshooter checklist:
Code: 
Network problems?
(1) Does the interface have an IP address?
(2) Is IP forwarding enabled on the router?
(3) Check your firewall/NAT settings: router's and jail's.
    All deny rules should log!
    Firewall logs land in /var/log/security.
(4) Are routes configured properly?
    List routes on jail and router.
    Test routes: netstat -rn; traceroute; etc.
(5) Are DNS settings correct?
    What is in /etc/resolv.conf?
    Is the DNS server reachable (routes, etc.)
    Does the firewall allow packets to DNS on port 53?

Tools
* tcpdump
* ipfw: list, table all list, nat show 1 config
* nc / telnet
* netstat -rn
* traceroute
* dig, drill, nslookup
 
tommiie said:
Hey all,

I follow along in Lucas' new book FreeBSD mastery: Jails. Starting on p. 159, I copied over the /usr/share/examples/jail/jib file and make it executable (something Lucas fails to mention). However, I use a different name for my physical interface: lab instead of jailether.
Code: 
ifconfig_em0_name="lab"
ifconfig_lab="up"

Next I follow along and create two test jails:
Code: 
test02 {
        vnet;
        vnet.interface = "e0a_test02";
        exec.prestart += "/usr/local/scripts/jib addm test02 lab";
        exec.poststop += "/usr/local/scripts/jib destroy test02";
}

test03 {
        vnet;
        vnet.interface = "e0a_$name";
        exec.prestart += "/usr/local/scripts/jib addm $name lab";
        exec.poststop += "/usr/local/scripts/jib destroy $name";
}
However, these interfaces do not get added to the labbridge interface that got created. I thus added them manually:

ifconfig labbridge addm e0b_test02 addm e0b_test03 up

Now, although both jails are connected to the bridge (and I've configured IP addresses on them) I still cannot ping one jail from the other.

Code: 
tom:~/ $ sudo jexec test02 ifconfig e0a_test02
e0a_test02: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:50:7f:0c:1a:6a
        hwaddr 02:18:17:5a:62:0a
        inet 203.0.113.2 netmask 0xffffff00 broadcast 203.0.113.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair
tom:~/ $ sudo jexec test03 ifconfig e0a_test03
e0a_test03: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:50:80:0c:1a:6a
        hwaddr 02:54:5d:26:1e:0a
        inet 203.0.113.3 netmask 0xffffff00 broadcast 203.0.113.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair

Anyone got a clue as to what I'm doing wrong?
Click to expand...
I purchased this text book also. I really liked his lecture he had.
 
You must log in or register to reply here.
Back
Top