FreeBSD Policy Based Routing with ipfw nat + fwd using 2 or more Poor Man's ssh VPNs

[lucdig]

Hello, thanks to the posts that I found on this forum, I could implement a gateway in FreeBSD that allows me to do flexible policy routing through different interfaces.

I'm going to share.

System: FreeBSD freebsd 12.0-STABLE FreeBSD 12.0-STABLE r346132 NEWKERNEL amd64

NEWKERNEL compiled with:

include GENERIC
ident NEWKERNEL

options IPFIREWALL
options IPFIREWALL_DEFAULT_TO_ACCEPT
options IPFIREWALL_NAT
options LIBALIAS

/etc/rc.conf: gateway_enable="YES"
/etc/sysctl.conf: net.inet.ip.fw.one_pass=0 ipfw doesn’t exit at the first match, it continues processing to the next rule

Network Interface: ifconfig hn0
<local_ip_hn0> = 172.19.167.89 (FreeBSD)
<local_netmask_hn0> = 0xfffffff0 = 255.255.255.240 = 28


Routing: netstat -nr4

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            172.19.167.81      UGS         hn0
127.0.0.1          link#1             UH          lo0
172.19.167.80/28   link#2             U           hn0
172.19.167.89      link#2             UHS         lo0

Initial ipfw state: ipfw list

65535 allow ip from any to any

Using 2 ssh connections to setup 2 tun interfaces. <remote host1> and <remote host2> are Linux, so the commands are ip link set… ip addr add… iptables -t nat…

ssh -x -l root -p <remote_port1> -f -T -N -C -M -S /tmp/test100 -o Tunnel=yes -w 100:100 <remote_host1>
ssh -x -l root -p <remote_port1> -T -S /tmp/test100 <remote_host1> 'ip link set dev tun100 up'
ssh -x -l root -p <remote_port1> -T -S /tmp/test100 <remote_host1> 'ip addr add 10.100.100.2/32 peer 10.100.100.1/32 dev tun100'
ssh -x -l root -p <remote_port1> -T -S /tmp/test100 <remote_host1> 'iptables -t nat -A POSTROUTING -s 10.100.100.1/32 -j MASQUERADE'
ifconfig tun100 10.100.100.1/32 10.100.100.2

ssh -x -l root -p <remote_port2> -f -T -N -C -M -S /tmp/test99 -o Tunnel=yes -w 99:99 <remote_host2>
ssh -x -l root -p <remote_port2> -T -S /tmp/test99 <remote_host2> 'ip link set dev tun99 up'
ssh -x -l root -p <remote_port2> -T -S /tmp/test99 <remote_host2> 'ip addr add 10.99.99.2/32 peer 10.99.99.1/32 dev tun99'
ssh -x -l root -p <remote_port2> -T -S /tmp/test99 <remote_host2> 'iptables -t nat -A POSTROUTING -s 10.99.99.1/32 -j MASQUERADE'
ifconfig tun99 10.99.99.1/32 10.99.99.2

Configuring nat on tun99 and tun100:

ipfw nat 100 config if tun100
ipfw nat 99 config if tun99

Adding ipfw rules to nat-and-route HTTP generated by FreeBSD to tun99 and HTTPS generated by FreeBSD to tun100, nat-and-route icmp traffic generated by external host to 8.8.8.8 through tun100:

00001 skipto 65534 tcp from <local_ip_hn0> to < remote_host1> < remote_port1> out
00001 skipto 65534 tcp from <local_ip_hn0> to < remote_host2> < remote_port2> out
00002 skipto 65534 tcp from < remote_host1> < remote_port1> to <local_ip_hn0> in
00002 skipto 65534 tcp from < remote_host2> < remote_port2> to <local_ip_hn0> in
00003 skipto 65534 ip from 172.19.167.80/28 to 172.19.167.80/28

00010 skipto 61100 tcp from 172.19.167.89 to any 443 out
00011 skipto 60099 tcp from 172.19.167.89 to any 80 out
00012 skipto 61100 icmp from 172.19.167.82 to 8.8.8.8 out

40099 nat 99 ip from any to any in via tun99
40100 nat 100 ip from any to any in via tun100

50000 skipto 65534 ip from any to any

60099 nat 99 ip from any to any out
60100 fwd 10.99.99.2 ip from any to any out
60101 skipto 65534 ip from any to any

61100 nat 100 ip from any to any out
61101 fwd 10.100.100.2 ip from any to any out
61102 skipto 65534 ip from any to any

65535 allow ip from any to any

Rules 1 and 2: don’t change the routing for the 2 ssh connections:

00001 skipto 65534 tcp from <local_ip_hn0> to < remote_host1> < remote_port1> out
00001 skipto 65534 tcp from <local_ip_hn0> to < remote_host2> < remote_port2> out
00002 skipto 65534 tcp from < remote_host1> < remote_port1> to <local_ip_hn0> in
00002 skipto 65534 tcp from < remote_host2> < remote_port2> to <local_ip_hn0> in

Rule 3: don’t change the routing between hosts of the same local network:

00003 skipto 65534 ip from 172.19.167.80/28 to 172.19.167.80/28

Rules 40XXX: packets coming in via tun99 and tun100 must be processed by the nat instances, checking if some entry exists in the internal nat table.

40099 nat 99 ip from any to any in via tun99
40100 nat 100 ip from any to any in via tun100

Rule 50000: after nat for the incoming packets via tun99 or tun100, no more processing is needed, skipto allow

50000 skipto 65534 ip from any to any

Rules 60XXX: nat through interface tun99, forward to the peer ip address of tun99, skipto allow

60099 nat 99 ip from any to any out
60100 fwd 10.99.99.2 ip from any to any out
60101 skipto 65534 ip from any to any

Rules 61XXX: nat through interface tun100, forward to the peer ip address of tun100, skipto allow

61100 nat 100 ip from any to any out
61101 fwd 10.100.100.2 ip from any to any out
61102 skipto 65534 ip from any to any

Rules 10, 11: traffic generated by 172.19.167.89 - outgoing HTTP to tun99, outgoing HTTPS to tun100

00010 skipto 61100 tcp from 172.19.167.89 to any 443 out
00011 skipto 60099 tcp from 172.19.167.89 to any 80 out

Rule 12: icmp traffic generated by 172.19.167.82 (172.19.167.82 has FreeBSD 172.19.167.89 as default gateway) to 8.8.8.8 is nat-ed and sent through interface tun100

00012 skipto 61100 icmp from 172.19.167.82 to 8.8.8.8 out

You can add ipfw rules for flexible Policy Routing between rule 3 and 40099. Be careful with the order of the rules, because net.inet.ip.fw.one_pass=0.

Hope it helps, enjoy