jails Isolating host-machine & sending traffic through a jail-proxy

madwebness

madwebness

While I'm relatively new to FreeBSD, I took my time to first study it (read quite a few books, including the ones on networking and jails) and then I took about 2 months trying to create a perfect workstation for myself, but so far I was unsuccessful in my attempts to do it according to my "design". For some reason I expected this to be easier than setting up the same thing on Linux, but it turned out to be more confusing -- yet, I haven not lost my enthusiasm. I'll describe the intended result below and would welcome any replies, however short, that can help me get close to it.

The major goal is to keep my bare metal desktop machine completely isolated, while allowing internet connections from jails & running software that requires internet connections from jail (in case of GUI apps, I'd like to make use of X-forwarding via ssh). Isolating the desktop is about the only goal I've yet achieved -- it was easy, of course, to add a couple of lines to /etc/pf.conf, however setting up functional networking with jails proved to be an issue.

Here, I must mention, I AM NOT using ezjails or any other jail management software -- not even jail.conf files is being used. I'm rather comfortable with Bash, so I can script whatever I prefer while specifying all jail options when calling jail. I also would like to understand my network setup completely and not rely on someone else's pre-defined setup.

Now, let's get straight to the network design. I'll try to "type" a diagram here, so it's immediately obvious what I'm trying to accomplish:

Code: 
----------------------      -----------------------------------------------
|              |RUNS:      |              |                          RUNS:
|   Desktop    |    *pf    |   Proxy VM   |                    * Wireguard
|              |           |              |                    * Privoxy
|--------------|           |--------------|
|VMN: 192.0.3.1|<~~~~~~~~~~|VMN: 192.0.3.2|<‒‒.
|LAN: 192.0.1.9|  fwd to   '--------------'   |
'**************'   EXT                        |
           |  ⠇                               |
    EXT    |  ⠇`jexec -l` or ssh only         |
 fwd from  |  ⠇......      |--------------|   |
Proxy only |         ⠇.....|  Browser VM  |   |   {
           |         ⠇     |--------------|   |   {   VMs only have access to
|--------------|     ⠇     |VMN: 192.0.3.3|‒‒>|   { outside world through Proxy VM
|   ROUTER     |     ⠇     '--------------'   |...{       and also ssh for
| connected to |     ⠇                        |   {  X11 Forwarding -- the rest
|   my ISP     |     ⠇     |--------------|   |   { is disallowed by `pf` on the
|--------------|     ⠇.....| Messenger VM |   |   {         bare metal.
|LAN: 192.0.1.1|     ⠇     |--------------|   |   {
'**************'     ⠇     |VMN: 192.0.3.4|‒‒>|
                     ⠇     '--------------'   |
                     ⠇                        |
                     ⠇     |--------------|   |
                     ⠇.....|    vm #N     |   |
                     ⠇     |--------------|   |
                     ⠇     |VMN: 192.0.3.n|‒‒>|
                     ⠇     '--------------'   |
                     ⠇...                  ...|
                              >> etc <<
​
I call jails "VMs" because some of them later might be bhyve instances instead. "VMN" means "VM Network".​

Tried using bridge + epairs for each VM, but got stuck at pf NAT rules (was able to connect to/from bare metal host, but unable to get internet access). I also tried aliases (such that each VM would get a separate IP-address on LAN) which largely worked, but I couldn't create a Wireguard connection inside the VM -- it's not allowed to create taps from within a jail that does not own its network. Throughout all of this, blocking rules in pf were DISABLED (commented) to ensure I got basic network connectivity.

I was under the impression it'd be easier to set it up under FreeBSD than it was on Linux -- this very network scheme worked "out of the box" on Linux with Docker. That is, it would create a bridge (or let you create one with subnet ip-range you want) and then direct traffic as pictured above. I didn't see any epair interfaces on Linux and would like, if possible, to avoid them in FreeBSD as well.

Both bare metal and jails are running FreeBSD 13.2.

Would appreciate your thoughts and advice or, perhaps, some specific configuration lines for ifconfig and pf.conf, if you have the time for it.
 
I must apologize, this indeed was a lack of consideration on my part to not post along the config files. So far, considering that each its own network with one end of the bridge connected to the host (end A) and the other end to the jail instance. As I mentioned, this worked somewhat in that it allowed me to connect to/from jails to the host, but not on through the default route of the host and on... that is the biggest issue.
Having said that, I'll first post the scripts (since all configuration is done through them at this point). There, through relevant cli arguments for the invoked commands you shall be able to deduct the configuration. Below are the two Bash script that start a jail instance:
 
Disclaimer: please forgive me if some variable names might be wrong or IP-addresses may be off by one or two (which, may be more critical). These scripts I posted I stripped them down off error messages, multiple ifs and all things unimportant to the actual problem.

The Problems
  1. Cannot get a connection from within the jail through the host (or another jail which then connects to the host and goes through its main route), but
    I can make connections between the host and the jail.

  2. Cannot make connections between various jails, it seems, although they seem to be on the subnet (192.0.3.0/24), but are all connected via different bridges.

  3. I would like to have 1 bridge, not a bridge per VM. That's kind of ridiculous, don't you think?

fbsd-jails/start is the one that starts the jail with all the configuration and invokes or changes whatever else is necessary for successful jail VM start and networking (well, hopefully).
Bash: 
#!/usr/bin/env bash

# This is used to assigned the last digit of the ip address. Because I want it so.
vm_index="$1"
if [[ -z $vm_index ]]; then
  echo "ERROR: you must provide numeric index for vm interface"
  exit 1
fi

mount_dir="$2" # Whatever user wants to mount in "rw" mode as nullfs.
jpath="/jail"
vm_name="vm${vm_index}"
vm_dir="$vm_name"
vm_path="$jpath/$vm_dir"
vm_hostname="${vm_dir}.vmlocal"

  # Not sure this part was important to include, but it works fine.
  # -------------------------------------------------------------------------------
  # MOUNTING NULLFS
  # -r means readonly, -t is filesystem type
  #
  # First one is the mount shared between many jails - scripts and dotfiles, thus should be readonly.
  mount -r -t nullfs $jpath/SHARE $vm_path/usr/local/share/jailshares
 
  # Second type of mounts is a directory user wants to mount. An account "me" already exists is the jail --
  # the called of this script only has to prodide a directory to mount in "rw" as $2.
  if [[ -n "$mount_dir" ]] && [[ -d "$mount_dir" ]]; then
    mount -t nullfs $mount_dir $vm_path/usr/home/me/main
  fi
  # ---------------------------------------------------------------------------------------------------

# Copy rc.conf.template to rc.conf, replace all %PLACEHOLDERS% in it.
# This file will be used to boot the VM, while rc.conf will remain a template.
#     NOTE: The contents of that template rc.conf file will be posted below.
cat $vm_path/etc/rc.conf.template | sed "s/%VM_INDEX%/$vm_index/g" > $vm_path/etc/rc.conf

### ATTENTION!
### WE'LL BE SWITCHING OUR FOCUS HERE, SO PAY ATTENTION.
### ANOTHER BASH SCRIPT RESPONSIBLE FOR CREATING VNETS
### THAT SCRIPT'S SOURCE WILL BE POSTED TOO -- AFTER THIS ONE.
#
../network/create-vlan-ext $vm_index # This is an +x file, so no problem
# ---^---
# AGAIN, YOU WILL FIND THE CODE FOR [FILE]network/create-vlan-ext[/FILE] SCRIPT BELOW.

# STARTING THE JAIL, passing most configuration options via the command line arguments. Just in case, I'll post my default
# jails config file at the very end of this post.
jail -vc -n $vm_name path=$vm_path                            \
     exec.clean exec.timeout=30                               \
     allow.mount.fdescfs allow.noset_hostname                 \
     allow.raw_sockets allow.sysvipc                          \
     mount.devfs devfs_ruleset=1005                           \
     exec.start="/bin/sh /etc/rc"                             \
     exec.stop="/etc/rc.shutdown"                             \
     exec.poststart="cpuset -c -j $vm_name -l 2-5"            \
     ip4.addr="192.0.3.$vm_index/32"                          \
     host.hostname="$vm_hostname"

# We now move the B-end of the epair into the created jail VM:
ifconfig $vnet_name vnet $vm_name

network/create-vlan executable bash script (called by the script above)
Bash: 
#!/usr/bin/env bash

# First I'll place the two functions. There will be more of them later, so forgive me for over-structuring the code at this point.

create_bridge() {
  ifconfig vmbridge1 inet 192.0.3.1/24 up
  #
  # NOTE: the outside network's default route from host is 192.0.1.1 (and, obviously, the assigned host's main network on igb0
  # has the mask of 192.0.1.0/24. Therefore, take note how the that "third"part of the ip address was changed from "1" to "3" for
  # all VM networks. I decided it'll be more obvious than changing that number to "2".
}

create_and_attach_vm_interface() {
 
  # mtu is chosen manually to be the same as ISP or VPN provider. Let me know if that makes sense and makes external or internal
  # connections faster or slower and whether settings the same mtu rate affected the CPU, requiring it to work less. I don't know, just
  # a fair assumption...
  #
  ifconfig epair     create                      && \
  ifconfig epair0a   inet 192.0.3.1/24 mtu 1412  && \
  ifconfig epair0b   inet $IPV4_ADDR/24 mtu 1412 && \
  ifconfig epair0a   name "${IFACE_NAME}a"       && \
  ifconfig epair0b   name "${IFACE_NAME}b"       && \
  ifconfig vmbridge1 addm "${IFACE_NAME}a"

  #
  # !!!ATTENTION:
  #
  #   Nothing will work until interface "${IFACE_NAME}b" is moved to an
  #   existing VM by the $PROJECT_ROOT/network/move script. If you're using this script
  #   from CLI, be mindful of that fact.
}

# MAIN PART
IFACE_NUM_SUFFIX=$1
IFACE_NAME="vm${IFACE_NUM_SUFFIX}"
IPV4_ADDR="${2:-"192.0.3.${IFACE_NUM_SUFFIX}"}"

# If brdge is there -- good, if not, then create it by calling the function above.
[[ -z "$(ifconfig | grep vmbridge1)" ]] && create_bridge

# Create and prepare all networks for the yet-to-be launched jail VM -- only thing skipped is moving that B-end of the epair.
# This can only be done once the jail instance is launched, and at this point it isn't just yet, because this script is being invoked
# by the other one, that's about to do it. But if the bridge's still no there, we'll exit with an error status.
if [[ -n "$(ifconfig | grep vmbridge1)" ]]; then
  create_and_attach_vm_interface
else
  exit 1
fi

Finally, the /etc/jails.conf, which has almost nothing in it, intentionally. What it has, is alos probably replaces by the cli args of the actualy command launching a jail instance.
Rich (BB code): 
# THIS IS A BARE MINIMUM CONFIG FOR JAILS, because most config options are
# managed by the scripts I posted above and I have no interest in keeping it
# configuration in the standard jails format.

exec.start="/bin/sh /etc/rc";
exec.stop="/etc/rc.shutdown";
exec.timeout=30;
exec.clean; # Prevents host from passing on ENV variables
 
You're still missing:

- host /etc/rc.conf
- /etc/pf.conf in the host and proxy
- rc.conf.template

Anyway, one thing to look at is to make sure that the host and proxy are both configured as gateways.

Your jail(8) command looks off to me. I think at the very least it's missing a vnet option. You're also assigning it an IP address, which you don't do when you use vnet. So I think you should remove the ip4.addr=... as well as ifconfig ... vnet, and instead use vnet.interface to assign the interface to the jail. I configure the jail's IP address in the jail's /etc/rc.conf.

So those are some suggestions to get you moving. I think you're going to have to do some good old-fashioned troubleshooting.

1. exec into the jail
2. examine ifconfig(8)
3. examine gateway config and routing table
 
Thank you for your response. I'm posting what you requested, however I'd like to point out that assigning ip address when creating a jail I think is actually possible and you'd want that. Either way, it's assigned both via the the command and the jail's /etc/rc.conf. What I'd like to poin't out is that the host <-> jail connectivity is fine, thus I got that part right. I think the part I'm missing is how to route the external requests from jails to the internet and get responses.

1. /etc/pf.conf -- for the purposes of testing the networking setup, the pf firewalls on both the host and the guests are completely disabled and are, therefore, irrelevant.

2. The rc.conf.template file used for all jails

INI: 
clear_tmp_enable="YES"
syslogd_flags="-ss"
sendmail_enable="NONE"
sshd_enable="YES"
gateway_enable="NO" # it is set to "YES" on the the jail-host

# The %VM_INDEX% placeholder will be replace by whichever ip is
# to be assigned by the launching script
ifconfig_vmi2b="inet 192.0.3.%VM_INDEX%/24 broadcast 10.0.2.255 mtu 1412"

# This is the jail-host machine. I can connect TO and FROM it,
# but NOT further on to the LAN network or the internet.
defaultrouter="192.0.3.1"

# Router's LAN IP assigned to the my host machine. Not sure if this line is useful here --  it seems
# to make no difference. To be clear once again, the "42" is my local static IP assigned to me by the physical
# Router device, which is connected to the internet.
if_igb0_alias="192.0.1.42/24"

3. The bare-metal host's machine /etc/rc.conf
Redacted to conceal some irrelevant information about hardware and installed software.[/SIZE][/SIZE]
INI: 
hostname="madewbness"
kld_list="fusefs"
clear_tmp_enable="YES"
syslogd_flags="-ss"
sendmail_enable="NONE"
devfs_system_ruleset="localrules"
sshd_enable="YES"
pf_enable="NO"
#pflog_enable="YES"
ifconfig_igb0="inet 192.0.1.42 netmask 255.255.255.0 broadcast 192.0.1.255"
defaultrouter="192.0.1.1"
gateway_enable="YES"
dumpdev="NO"
hald_enable="YES"
dbus_enable="YES"
jail_sysvipc_allow="YES"

UPDATE: you might be right that gateway_enable should be set to "YES" in /etc/rc.conf both the host and the VM (right now it's set to "YES" only on the host). However, I'm fairly certain I did try it both ways to no avail.
 
A point #1 I'd like to to reiterate, which is extremely important, but, perhaps, somehow, evaded the attention of the readers (and mine, even). I don't want to have to create a bridge + epair-a + epair-b interfaces for every jail VM. That's not only too many networks to parse through later on, but will probably be a nightmare to deal with in pf.conf (to re-iterate: for the purposes of testing this setup is pf completely disabled).

A point #2 is a question: can proper routing be done without pf & pf.conf or any other "firewall" altogether?

For the time being, however, if I can make it work - I'll go with the 3 interfaces per VM. Then figure out how to make it work some other way.
 
You shouldn’t need a bridge per VM. You should be able to connect all the epair a ends into a single bridge.

You will need pf (or some firewall) on the host because you’re doing NAT.

You might be able to do this without a firewall, if your jails are on the same subnet as the host and router. In which case, you probably just want IP aliasing on the host interface. Otherwise, the router has no way of knowing to route those packets back to the host. If you want no firewall, and vnet interfaces, then you’ll also need to configure the router to route those specific IPs back to the host.
 
Yes, I tried using aliases on 127.0.0.1/24 interface just to test it -- it kind of works for simple cases, but that didn't work for networking. As for router aliases and additional ip addresses - this isn't an option, I need connectivity without another device being enabled.
 
patmaddox said:
In that case, at a minimum you will need pf on the host to NAT your jail network to the external network.
Click to expand...
Great, I'll be handling the pf separately then, will maybe create another thread with a more specific question about NAT. So let's leave out the external internet connectivity out of it for now.

However, could you please correct my code or settings or tell me in words: how to have a single BRIDGE where each VM gets its own ip address on its 192.0.3.0/24 subnet. And the number of other created interfaces (epair) is minimized -- perhaps maybe there's a way NOT to use them or use some other kind of interfaces? Or get jails to connect to the bridge directly somehow? When running Docker on Linux, if I remember the ifconfig output correctly, it had its own bridge, each Docker instance would get an ip address on its subnet, but I have not seen any additional interfaces for individual Docker instances created. Or am I mistaken? If I'm not wrong about Docker, that was a perfect setup that I'd like to have.
 
How to create a single bridge, and add multiple members to it:

Code: 
# ifconfig bridge create
bridge3
# ifconfig epair create
epair0a
# ifconfig bridge3 addm epair0a
# ifconfig epair create
epair1a
# ifconfig bridge3 addm epair1a
# ifconfig bridge3
bridge3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 58:9c:fc:10:ae:41
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 0 ifcost 0 port 0
    member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 12 priority 128 path cost 2000
    member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 10 priority 128 path cost 2000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>

How to assign each VM to an IP:

Code: 
# jail -c name=myjail persist vnet vnet.interface=epair0b
# jexec myjail ifconfig epair0b inet 192.168.3.42/24
# jexec myjail ifconfig
lo0: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair0b: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:26:63:65:3d:0b
    inet 192.168.3.42 netmask 0xffffff00 broadcast 192.168.3.255
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

With vnet jails, you need to provide an interface. One of the mechanisms for doing that is epair(4) which produces a pair of interfaces - one gets plugged into the bridge, one gets assigned to the jail.

If you don't like the epair interfaces in the host, you can try netgraph(4). I've not used it, but this article describes how to use it.
 
I think I've read about netgraph and, on its surface, it looked worse than epairs and bridges. Will consider this thread closed and solved -- haven't tried the solution, but, I think, the conclusions are clear enough. Thank you for your responses.
 
You must log in or register to reply here.
Back
Top