Routing between jails and physical interfaces

[korund]

I'm trying to get the following scheme up and running:

What would be a good read to start except of jail() man page?

 

[`Orum]

What are you using for NAT?

I have a setup somewhat like this, and I NAT the jails to the internet with pf(), and control access to/between them with firewall rules. By NATing by source IP address, you can control which interface the traffic exits from as well.

 

[korund]

What are you using for NAT?
I have a setup somewhat like this, and I NAT the jails to the internet with pf(), and control access to/between them with firewall rules. By NATing by source IP address, you can control which interface the traffic exits from as well.

Nothing for now, waiting to upgrade 11-REL to 11-REL-p1 tomorrow. But will use pf for sure. Could you paste your config for the reference?

 

[`Orum]

Could you paste your config for the reference?

Sure. For example, on my router, I run a jailed dns/unbound at 192.168.0.220 (hostname dns), and a separately jailed dns/nsd on 192.168.0.219 (hostname auth-dns). Clients query unbound, which has my stub zone redirected to nsd, but is otherwise the recursive server for internet domain names. The relevant rules in pf.conf() look something like this, with both unbound and nsd on the $srv_if network. Also, I currently only have one WAN connection on $ext_if with a single IP address, so adjust your NAT rules to route accordingly if you want things routed out different interfaces or IP addresses:

nat on $ext_if from { $int_netwk, $srv_netwk } to !<privt> -> ($ext_if)

pass in  quick            inet proto { tcp, udp }                                           to dns      port 53 keep state
pass out quick on $loo_if inet proto { tcp, udp } from { $loo_if:network, $srv_if:network } to dns      port 53 keep state
pass     quick on $loo_if inet proto        udp   from dns                                  to auth-dns port 53 keep state
pass out quick on $ext_if inet proto { tcp, udp }                                           to          port 53 keep state

Edit: One point I should clarify in these rules. If you look closely, you'll notice in the nat rules I use $srv_netwk and in the pass rules I use $srv_if:network. There is a very good reason for this! Why?

The $srv_if has numerous IP aliases, and I'm using "set ruleset-optimization basic" earlier in my ruleset. $srv_if:network normally expands each IP address to the network it's on, creating a separate rule. For example, let's pretend the $srv_if interface had the IP addresses of 10.0.0.1/8, and two aliases of 10.0.0.2/32 and 10.0.0.3/32. A ruleset in pf without optimization would do this:

block in quick on $srv_if from any to ! $srv_if:network
# Without basic optimization, this would expand to:
block in quick on $srv_if from any to ! 10/8
block in quick on $srv_if from any to ! 10/8
block in quick on $srv_if from any to ! 10/8

Yes, that's right, you get the same rule three times, once for each alias! So, I use basic ruleset optimization to eliminate the problem in an elegant manner. The issue with nat rules, however, is they are unaffected by such optimization! So, for these, I manually define a macro of the network for the $srv_if:network as $srv_network. Using my earlier IP addresses, this would look like this in a config: srv_network="10.0.0.0/8"

Yes, it's a bit hackish, but I have yet to find a better way around it. If you discover one, I'd love to hear about it.

 

[kpa]

You can always almost regardless of the situation reduce the outbound NAT rule to this:

nat on $ext_if inet from ! ($ext_if) to any -> ($ext_if)

The reason this works and doesn't NAT anything you don't want to be NAT'ed is because it applies only to traffic that is in the outgoing queue of the ext_if interface and is not originating from ext_if (the host system) itself. This catches all traffic from the jails regardless of the addressing scheme used.

 

[`Orum]

You can always almost regardless of the situation reduce the outbound NAT rule to this:

nat on $ext_if inet from ! ($ext_if) to any -> ($ext_if)

Yes, sorry that wasn't apparent in my pasted rules. There are certain networks I explicitly don't want NATed on that router, but unless he is in the same situation your nat rule is much more concise.