PF PF config for double NAT jail host

R

richardnpaul

I think that I'm being somewhat ambitious and I'm finding that I'm getting some horrible issues as a result. Firstly, what I'm trying to achieve.

The way that I have tried to set up this network in the past was that the router was in the DMZ and that it passed some traffic (HTTP/S) through to the Nginx server and another for Teamspeak on a separate VM.
However, I was having issues with routing the traffic (which in the end turned out to be because I was using the network adapters in Hyper-V, rather than the legacy network adapters) so I ended up port forwarding the web ports to the Nginx VM's IP address on the LAN and setting up yet another PF config there.
This has vastly complicated things on the Nginx box because I now have an external network for the LAN, an external network for the private Hyper-V network and an additional `lo1` interface network for the jails (I'm currently trying to host 4 jails on this machine.)

The jails on the Nginx VM need access to the private network and the return traffic to route back through to them because the apps need to contact the DB servers on this private network, then, the jails also need access to the internet for things like package upgrades etc. This sets up a requirement for a double NAT where the jail can speak with both networks. However, with my current configuration this is all horribly broken and things like viming a file break the SSH connection a lot. This currently is horribly broken with Pingdom telling me that my uptime is something like 33%
My configuration is like this:

VM Host running Windows 10 Pro and Hyper-V:

  • This contains my FreeBSD VMs in a storage pool.
  • This has two networks, one internal private network for inter VM communications and one Internet accessible interface.
  • I have a FreeBSD router for the VMs which don't have an internet facing address (So my two DB servers, one MySQL and one PostgreSQL)
  • The VM which has Nginx installed on it also is a jail host and has two interfaces, one on the private network, one on the LAN
Here's my Nginx pf.conf:

Code: 
ext_if = "de1"
int_if = "de0"
jail_if = "lo1"
jail_net = $jail_if:network

web_services = "{ http,https }"
protos = "{ udp,tcp }"

set loginterface $ext_if
set skip on lo0
set skip on $jail_if
set block-policy drop

scrub in all

nat pass on $ext_if from $jail_net to any -> ($ext_if:network)
nat pass on $int_if from $jail_net to any -> ($int_if:network)

#antispoof for $ext_if
rdr pass on $ext_if inet proto tcp to $web_services -> ($jail_if)
rdr pass on $int_if from $jail_if to ($int_if) -> $int_if

block in all
pass out quick on lo0 all
pass out quick on $jail_if all
pass out quick on $ext_if all
pass out quick on $int_if all
pass in quick on $ext_if proto tcp from any to any port $web_services
pass in quick on $ext_if proto $protos from any to any port ssh
pass in quick on $ext_if from $jail_if to any
pass in quick on $int_if from $jail_if to any

This is what I've managed to scrape together from the docs and many posts here and across the internet from Googling 'FreeBSD Jails PF'
 
Your NAT rules are interesting. Are you trying to do 1:1 NAT or NAPT? If it's the latter, you should not be using ':network' as that will give you a CIDR block which I believe attempts 1:1 NAT.

Again, assuming you want to do NAPT, I'd probably write it something like this:
Code: 
# NAT rules are 'first-match', so put more specific rules first (though it shouldn't matter as long as your routing tables are correct)
nat on $int_if from $jail_net to $int_if:network -> ($int_if)
nat on $ext_if from $jail_net to any -> ($ext_if)

Alternatively if you only wanted them to access specific IP addresses you could even use a table instead of the entire network, and could further limit it by ports you wanted to allow them to as well.

Edit: Could you also post your ifconfig output? Your rdr rules probably won't do what you want if you have aliased IPs assigned to your $jail_if.
 
Hi Orum, thanks for your input. Ifconfig:

ifconfig
Code: 
de0: flags=883<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 00:15:5d:00:0a:19
    hwaddr 00:15:5d:00:0a:19
    inet 10.10.10.110 netmask 0xffffff00 broadcast 10.10.10.255
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (100baseTX)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 192.168.10.0 netmask 0xffffff00
    inet 192.168.10.4 netmask 0xffffffff
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    groups: lo
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
    groups: pflog

Maybe, if you wouldn't mind, would you be able to go through this line by line for me. I'd attempt to do the same but as I mentioned in my first post I've hacked this together from internet posts and a sketchy understanding of what it's doing (I didn't know about NATP by the way, but it does actually sound like what I want if I have multiple jails sharing the same resources).

To answer your question about NAT, I have multiple jails all trying to access different IPs on the internal private network. I also have multiple jails which I want to connect via the external network so that they can get out onto the internet and I can thus update them. This is all with the jails on yet another network which is a cloned loopback interface. I'm going to try to force myself to understand what you've put in the previous post so that I can make some more knowledge drive changes to the pf.conf

Firstly it'll give me an understanding which I've not been able to get from the docs and elsewhere, and secondly it'll highlight any disparities between what I think that a rule is doing and what it is actually doing/not doing.

Some extras for you :)

rc.conf
Code: 
...
defaultrouter="192.168.0.1"

ifconfig_de1="inet 192.168.0.160 netmask 255.255.255.0"
ifconfig_de0="inet 10.10.10.110 netmask 255.255.255.0"

gateway_enable="YES"
cloned_interfaces="lo1"
ipv4_addrs_lo1="192.168.10.0/24"

jail.conf
Code: 
interface = "lo1";
ip4.addr = 192.168.10.$ip;

host.hostname = "$name";
path = "/usr/local/jails/$name";

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

# Jail Definitions
richardnpaul {
    $ip = 1;
}
dev_richardnpaul {
    $ip = 2;
}
wishlists {
    $ip = 3;
}
 
Okay, a few things to note here. You're using private, non-local address space on the $jail_if, so that's routable. As such, you shouldn't need any kind of NAT between the jails and the hosts they're trying to reach on your $int_if, assuming that the hosts on the $int_if:network are using this machine (the jail host) as their default router, and they (the other machines on the $int_if:network) don't have routes to 192.168.10.0/24 out some other interface (i.e. assuming they're not multi-homed). So I wouldn't even nat on $int_if at all, as if those assumptions are true, you don't need it. Things are a bit different if your jails were using something in the 127/8 address space, which I often do, but you're not.

Again, assuming what I said earlier is true, and you remove the nat rule on $int_if, you also don't need any rdr rules on it either. You should simply write pass rules instead, as rdr is (usually) used for port forwarding on NATed interfaces, and you won't be doing any NAT except on your $ext_if.

However, you will still need the nat rule on $ext_if, as well as the rdr if you want to expose the jailed host for incoming connections from the Internet. But, you should not use "... -> ($jail_if)" in your rdr rule, as that interface has multiple IP addresses assigned to it. Put in the IP of the specific jail you want traffic redirected to instead.

I encourage you to first rewrite your ruleset yourself, using my suggestions, and try it out, as that's the best way to learn. Then, if you get stuck somewhere, post it here, and I or someone else can help.

Edit: To simplify things even further, you could just assign aliased IPs to your $int_if and give them to the jails, and not have a separate $jail_if.
 
Agh, I meant to update this last night as it was missing some configuration. Tonight I'm currently switching from Hyper-V to Virtualbox to see if I have some of the same issues with networking (latency in the connections intermittently, but persistently For some reason, passing the VHDx itself seems not to have got the current state :'‑( so I'm left rebuilding a lot of these rules any way :D.)

I've realised that I really don't need a lot of this natting or RDR stuff, but I do have multiple jails on the 192.168.10.x on the lo1 interface trying to go to the $int_if but the ins and outs of nginx just need to be handled with a normal pass in rule on the ext_if, because nginx talks to the jails via socket files within the jail's file structure (which also allows me to serve the static assets from the jail FS too.)

When I've figured it all out I'll come back with my config O:‑)
 
This is what I ended up with on the nginx loadbalancer/jail host after the switch to virtualbox (which is much more stable on the network interface side of things I've found so far; however, I've only got one jail deployed so far and have yet to redeploy my personal website.)

Code: 
ext_if = "em0"
int_if = "em1"
jail_if = "lo1"
jail_net = $jail_if:network

web_services = "{ http,https }"
protos = "{ udp,tcp }"

set loginterface $ext_if
set skip on lo0
set skip on $jail_if
set block-policy drop

scrub in all

nat on $int_if from $jail_net to any -> ($int_if)
nat on $ext_if from $jail_net to any -> ($ext_if)

block in all

# Outbound rules
pass out quick on lo0 all keep state
pass out quick on $jail_if all keep state
pass out quick on $int_if all keep state
pass out quick on $ext_if all keep state

# Inbound rules
# SSH rules
pass in quick on $int_if:network proto $protos from any to any port ssh keep state
pass in quick on $ext_if proto $protos from 192.168.0.0/24 to any port ssh keep state
pass in quick on $int_if from $jail_if:network to any keep state
pass in quick on $ext_if proto tcp from any to any port $web_services keep state
 
You probably don't need the "nat on $int_if from $jail_net to any -> ($int_if)" rule at all, as I mentioned earlier, and it will just make things more complicated to troubleshoot and have a (probably small) negative impact on performance.

A better way to think about your $jail_if/lo1 is like another (V)LAN. You almost never need to NAT between two LANs, because you can simply route between them. Since you're using routable adresses on them, there's really no difference between them and another physical or virtual interface, except for maybe MTU (which you could tweak if you really wanted to) and available features (like WOL).
 
You must log in or register to reply here.
Back
Top