Solved getting started with jails

[tOsYZYny]

I am interested in consolidating my router and workstation into a single physical box for ease of maintenance. I am thinking that I would have my workstation run the router inside a jail in which the router is assigned the physical network interfaces and it'd perform DNS, DHCP, and firewall duties. The little bit of reading seems to indicate that setting up graphical / X11 in a jail is a bit complex.

I need to figure out how to assign the interfaces to the host, but that seems doable:

VNET Jails with a dedicated ethernet interface

I have been reading up on VNET Jails and looking at the work in usr/src/share/examples/jails reading about jng (is this jails next generation?) and jib. Plus I have read the README and a good article on FreeBSD Journal plus some forum posts. My question is can I use a dedicated network...

forums.freebsd.org

My concern is, since the host no longer owns any network interfaces (since I assigned them to the guest), will that be a problem? Or should I not worry about that until I get to that point?

 

[Holger]

I am not a jails expert, but would NATing your jails solve the problem? When configuring that with pf(4)() you could also implement forwarding the relevant ports and such. But I am no network expert.

What I mean is the following:

• You create a network loopback device (e.g. lo1) and use that for your jails, not your physical interface.
• You then NAT your loopback device to the physical device.

This has the advantage that your physical network interface stays on the host and can use DHCP, etc., while the jails, running on the loopback device, can be assigned static IP addresses.

Here is how my pf.conf looks like for NATing my physical network interface (re0) to a loopback device (lo1), with which I configure all my jails (currently only one ):

nat on re0 from {lo1:network} to any -> (re0)

As for launching a graphical user interface inside your jail, I would strongly consider simply using a VNC solution like net/tigervnc-server. It's performant, gets the job done and -- maybe most importantly -- scales up to a fully virtualized solution (i.e. a physical server or VM somewhere in the network).

 

[tOsYZYny]

Hmm.

I am not a jails expert, but would NATing your jails solve the problem? When configuring that with pf(4)() you could also implement forwarding the relevant ports and such. But I am no network expert.

Not sure, I plan to use the workstation part daily and the router only whenever there is a problem or I need to monitor stuff. The router is going to own the WAN connection and get DHCP from the modem and the router should also own the LAN NIC so I can place a firewall there.

If the workstation is the host, I want to be able to get out to the Internet by magically forwarding traffic from lo0 through the LAN NIC and then the WAN.

My experience with VNC is that it is fine for "work", but I wouldn't want to use that to watch movies or play games. I'm not playing games, but movies or videos yes and I want to have 0 lag.

I plan to have a single jail at this time, for the router functionality, but I could increase that later. At that point in time, I would most likely want those instances to be on the LAN network with the router.

 

[Phishfry]

I say build the router totally separate.
Resist the urge to put anything on the box but dnsmasq and button it down with every setting you can find.
Then button that down with tripwire.

That was my approach. I like watching pfTop. It showed me that my Liebert UPS Monitoring modules were leaking onto the internet for ???.
So with no settings to turn that off I had to shield my own equipment with pf.
I like having dnsmasq filter list.

Install dnsmasq

Automatically updated, moderated and optimized lists for blocking ads, trackers, malware and other garbage - notracking/hosts-blocklists

github.com

root@x9srl:/home/firewall # pkg info
devcpu-data-1.39               Intel and AMD CPUs microcode updates
dnsmasq-2.86,1                 Lightweight DNS forwarder, DHCP, and TFTP server
gettext-runtime-0.21           GNU gettext runtime libraries and programs
gmp-6.2.1                      Free library for arbitrary precision arithmetic
got-0.60                       Game of Trees version control system
iftop-1.0.p4                   Display bandwidth usage on an interface by host
indexinfo-0.3.1                Utility to regenerate the GNU info page index
libidn2-2.3.2                  Implementation of IDNA2008 internationalized domain names
libunistring-0.9.10_1          Unicode string library
nettle-3.7.3                   Low-level cryptographic library
pftop-0.7_9                    Utility for real-time display of statistics for pf
pkg-1.17.5                     Package manager
readline-8.1.1                 Library for editing command lines as they are typed
smartmontools-7.2_1            S.M.A.R.T. disk monitoring tools
tripwire-2.4.3.7_1             File system security and verification program
ytree-2.03                     DOS-XTREE(tm) look-a-like file manager

It is a ratrace.

 

[Holger]

Hmm.


Not sure, I plan to use the workstation part daily and the router only whenever there is a problem or I need to monitor stuff. The router is going to own the WAN connection and get DHCP from the modem and the router should also own the LAN NIC so I can place a firewall there.

If the workstation is the host, I want to be able to get out to the Internet by magically forwarding traffic from lo0 through the LAN NIC and then the WAN.

My experience with VNC is that it is fine for "work", but I wouldn't want to use that to watch movies or play games. I'm not playing games, but movies or videos yes and I want to have 0 lag.

I plan to have a single jail at this time, for the router functionality, but I could increase that later. At that point in time, I would most likely want those instances to be on the LAN network with the router.

So maybe vice versa: Assign the physical interface to your jail and run your host on the loopback device? You would simply need to reverse the pf rule I mentioned above.

As for VNC: I assume you don't want to watch videos inside you router jail, do you? On your workstation host, you would not need any VNC.

 

[tOsYZYny]

root@x9srl:/home/firewall # pkg info devcpu-data-1.39 Intel and AMD CPUs microcode updates dnsmasq-2.86,1 Lightweight DNS forwarder, DHCP, and TFTP server gettext-runtime-0.21 GNU gettext runtime libraries and programs gmp-6.2.1 Free library for arbitrary precision arithmetic got-0.60 Game of Trees version control system iftop-1.0.p4 Display bandwidth usage on an interface by host indexinfo-0.3.1 Utility to regenerate the GNU info page index libidn2-2.3.2 Implementation of IDNA2008 internationalized domain names libunistring-0.9.10_1 Unicode string library nettle-3.7.3 Low-level cryptographic library pftop-0.7_9 Utility for real-time display of statistics for pf pkg-1.17.5 Package manager readline-8.1.1 Library for editing command lines as they are typed smartmontools-7.2_1 S.M.A.R.T. disk monitoring tools tripwire-2.4.3.7_1 File system security and verification program ytree-2.03 DOS-XTREE(tm) look-a-like file manager

That seems awfully light, what function does that Jail / VM serve, just DNS forwarding / filtering?

I use coredns with a bunch of blacklists, but that combines both resolution for my internal network with blocking external traffic. Would you mind sharing your entire setup? I imagine that you need pf in there somewhere along with DHCP. If I interpret your setup, I would surmise that you have a separate jail for each function (DHCP, firewall, NAS / SMB, etc.)

So maybe vice versa: Assign the physical interface to your jail and run your host on the loopback device? You would simply need to reverse the pf rule I mentioned above.

Ok, that makes sense then and then I can use the NICs I have today without needing to get another one which would be great because I only have 4 ports in my office, 2 are in use and with my new setup, 3 would be in use.

As for VNC: I assume you don't want to watch videos inside you router jail, do you? On your workstation host, you would not need any VNC.

Nope, you're right, not unless I'm into ascii art, if it looks good that way, maybe it'd be a thing. Ok, cool.

 

[Phishfry]

That seems awfully light, what function does that Jail / VM serve, just DNS forwarding / filtering?

No Jails. That is what I am saying. Do it bare metal. Start simple.
Jails add unneeded complications. Keep it simple.

Some tasks are ideal for jails and some are not. Router in a jail is not ideal in my book.

 

[Phishfry]

This article would have you think otherwise.

Virtualize Your Network on FreeBSD with VNET - Klara Systems

Learn how to use VNET for network isolation in FreeBSD Jails. This article covers testing firewall changes and proper jail networking.

klarasystems.com

Everyone has had that horror moment when you press enter to commit some firewall configuration and everything stops. You wonder if your local network has gone down, or if the host has had a hiccup and eventually you have to admit that the new firewall rule that you were so sure of has locked you out of the machine.

I agree with this but it also validates my KISS approach. You don't put a webserver on a firewall.
Also backup your firewall rules to rollback before fiddling. Who doesn't do that?

 

[tOsYZYny]

Ok, I will read and think about it more. At the very least, it is going to be substantial effort.

There would always be that potential that while I am sitting physically present at the machine, perhaps I wouldn't be able to SSH in from my host to the guest, and then what do I do? That is a great point, because it happens frequently enough that we need to plan for those what if scenarios.

I do keep my system build process up-to-date (set of shell scripts wrapping freebsd-install), so I hope I can eliminate most of that risk by keeping a spare drive or 2 on standby in case I do lock myself out. Worst case scenario, I power the machine off, put in a known working drive, and I should be good to go.

"Famous last words"

In regards to not putting other services on a firewall, I agree and disagree. I would be sharing the same physical hardware, but isolating it in software. I'd like to more fully utilize the hardware than to have it mostly idle.

 

[Phishfry]

I can eliminate most of that risk by keeping a spare drive or 2 on standby in case I do lock myself out. Worst case scenario, I power the machine off, put in a known working drive, and I should be good to go.

But when you learn to do it yourself you will no long be afraid to test another rule or change a port setting.

I'd like to more fully utilize the hardware than to have it mostly idle.

I disgress here. CPU load at near zero does not bother me one bit.
This is my gatekeeper.

Oh yea. A physical display or serial console is required until you reach level 4. Domination of ssh.

When you can unplug your firewall from display and move it to shelf you have won.
I use /etc/rc.local for audible confirmation.

firewall@x9srl:~ % cat /etc/rc.local
echo "cdefgab" > /dev/speaker

Startup speaker first in /etc/rc.conf

###
##Enable Startup Sound ##
kld_list="speaker"
###

For exit. Descending notes.
/etc/rc.shutdown.local

echo "bagfedc" > /dev/speaker

One quirk here is that rc.shutdown.local only fires on shutdown -r.
If you issue reboot command the script does not run and you get no shutdown tones.
To resolve this make an alias in .cshrc

alias reboot    shutdown -r now

 

[tOsYZYny]

firewall@x9srl:~ % cat /etc/rc.local echo "cdefgab" > /dev/speaker

Awesome, I enjoyed that. I use beep for some scripts to notify when they're done. That is a cool hack in regards to shutdown / reboot. It sounds corny, but it is simple, yet effective for conveying state.

Back to the original topic at hand, I am able to assign a physical interface to a jail thanks to:

VNET Jails with a dedicated ethernet interface

I have been reading up on VNET Jails and looking at the work in usr/src/share/examples/jails reading about jng (is this jails next generation?) and jib. Plus I have read the README and a good article on FreeBSD Journal plus some forum posts. My question is can I use a dedicated network...

forums.freebsd.org

So, then, it is more or less a question of setting up the loopback device / nat and all the glue to make it work.

 

[Erichans]

An addition: Routing and Firewalling VLANS with FreeBSD - January 6, 2021

 

[tOsYZYny]

I am trying to create a simplistic example of what I'm trying to do:

Following the example here:

VNET Jails with a dedicated ethernet interface

I have been reading up on VNET Jails and looking at the work in usr/src/share/examples/jails reading about jng (is this jails next generation?) and jib. Plus I have read the README and a good article on FreeBSD Journal plus some forum posts. My question is can I use a dedicated network...

forums.freebsd.org

# Assume igb1 is physically connected to the 192.168.20.0/24 subnet.
# Create a vnet jail.
jail -c -n testjail vnet persist

# Move igb1 to the vnet jail.
ifconfig igb1 vnet testjail

# Now you cannot see igb1 on the host.
ifconfig igb1

# But you can see it in the jail.
jexec testjail ifconfig igb1

# You can assign an IP address to igb1 in the jail and
# communicate with other hosts on the 192.168.20.0/24 subnet.
jexec testjail ifconfig igb1 192.168.20.1/24
jexec testjail ping 192.168.20.3

I was able to ping other hosts (in the jail) on the same network after I restarted. I couldn't figure out how to fix my configuration, so I restarted to clear the jails ...

As expected, the host cannot ping other hosts on the network because it doesn't see igb1 any longer. So, I need to figure out how to do that, maybe nat will help? I couldn't do:

       route add default 192.168.20.1

It said network is unreachable.


This is what was suggested earlier to do in the guest.

       nat on igb1 from {lo1:network} to any -> (igb1)

 

[tOsYZYny]

This article would have you think otherwise.

Virtualize Your Network on FreeBSD with VNET - Klara Systems

Learn how to use VNET for network isolation in FreeBSD Jails. This article covers testing firewall changes and proper jail networking.

klarasystems.com

I agree with this but it also validates my KISS approach. You don't put a webserver on a firewall.
Also backup your firewall rules to rollback before fiddling. Who doesn't do that?

I overlooked that article, I think that is exactly what I need.

So, I will create an epair interface/device, assign one half to my router and move over the real physical interfaces, and assign IPs to both sides of the epair interface. Then, inside of the jail, I will NAT and also use pf to restrict what the host can do.

Ok, I think I understand conceptually now. That I think was the mental hurdle I had to overcome.

Consolidating this into a single box will save me about $7 -> $10 annually and reduce my CO2 emissions by 200 lbs whilst probably increasing my headaches until I figure this stuff out.

I think this will open new avenues to explore, I've always wanted to run each network service in a jail so I could control precisely what each is doing or can do. Though, I doubt I will see much benefit there, in reality, my concern is more on the mobile device side where apps collect all sorts of data and have unrestricted access to anything and everything and use DNS over HTTPS so blocks become more difficult without terminating network access altogether.

 

[tOsYZYny]

Ok, I'm getting somewhere.

1. created an epair interface, assigned IPs to both sides, move one side to guest (it shouldn't matter which side, it is just a patch cable, right)
2. moved physical interface to jail
3. enabled forwarding in jail
4. added default route in host as the guest epair IP (192.0.2.2, host is 192.0.2.1)

The host can ping the guest and the guest the host. The guest can get to the Internet / network and ping others hosts in my LAN.

But the host can no longer ping any other hosts in the LAN or get to the Internet.

I was thinking that at this point, I need to NAT, so I create a basic pf ruleset:

set block-policy reject
set skip on lo

nat on igb1 from 192.0.2.0/24 to any -> (igb1)
pass out all keep state

But after starting pf, that still doesn't help.

EDIT: #1
tcpdump indicates that the traffic is going through the epair0b interface, so the routing I think is working.

EDIT: #2
tcpdump on my real, hardware router (that is what I'm trying to ping) reveals that my guest is not NATing, the ip address is still showing at 192.0.2.1, the host IP.

EDIT: #3
I just realized that the pf rules that are loaded are that of the hosts (I haven't created a separate FS for this jail, so that will likely solve this headache). The hosts rules are for my workstation needs and do no NATing whatsoever. I'm trying to fix that.

EDIT: #4
It works!

Now, off to building a "clean" example!