Routing Bastille Jails with loopback (bastille0) strategy to Tailscale Network

[srey]

Hi !

I'm a Ubuntu user from long time, but i choose freebsd for my homelab, because stability, security, learning, but also jumping into new rabbit hole during summer ...

I resume my objectives into one big schema with (false) ip to represent routing.

In two words i have :
- one machine run Freebsd 13.1 at home, a tiny machine (Dell Small Factor Form / homelab), on my home network 192.168.1.xx with gateway 192.168.1.254 running Bastille Jails using a loopback strategy on interface bastille0
- one VPS that run Ubuntu and Docker containers running on Docker Network (webnetwork bridge) 172.17.0.1/16

With Tailscale VPN (overlay network tailscale0) installed on these two machine, they exist on the same network with 100.121.116.111 (homelab) and 100.64.200.29 (vps)

I imagine that both type of containers are :

- a) exposed on tailscale network and
- b) exposed to internet with a reverse proxy (Traeffik on docker container) that listen on public ip (135.180.90.110)

Everything run well with Bastille, usring RDR rules, i see that jails are well exposed to localhost (127.0.0.1) on my homelab.

For example, i'm running a photoprism jails exposed to 2342, and sockstat on homelab return :

root photoprism 70095 7 tcp4 127.0.0.1:2342 *:*

But there is something i don't understand, how could i redirect localhost to tailscale network, in other word : 127.0.0.1:2342 on loopback0 is same as 100.121.116.111:2342 to tailscale0.

Actually this is not the case, 100.121.116.111:2342 is not accessible. I don't understand how do reroute lo0 to tailscale0 ?

In my mind, after that i could redirect traffic to tailscale ip using traeffik reverse proxy without too much problem after that.

Because i'm a beginer with freebsd/pf world, and knowing only the basic of routing in general, I don't know if this is the good way to expose thing.

Command netstat -rn return :

Internet:
Destination Gateway Flags Netif Expire
default 192.168.1.254 UGS re0
10.0.0.1 link#3 UH bastille
100.64.200.29 link#5 UHS tailscal
100.66.237.121 link#5 UHS tailscal
100.83.98.8 link#5 UHS tailscal
100.83.174.105 link#5 UHS tailscal
100.100.100.100 link#5 UHS tailscal
100.121.116.111 link#5 UH lo0
127.0.0.1 link#2 UH lo0
192.168.1.0/24 link#1 U re0
192.168.1.84 link#1 UHS lo0

The bastille rdr photoprism list return
rdr pass on re0 inet proto tcp from any to any port = 2342 -> 10.0.0.1 port 2342

ifconfig -a return the multiple interfaces

re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
ether c0:25:a5:86:18:33
inet6 fe80::c225:a5ff:fe86:1833%re0 prefixlen 64 scopeid 0x1
inet 192.168.1.84 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bastille0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 fe80::1%bastille0 prefixlen 64 scopeid 0x3
inet 10.0.0.1 netmask 0xffffffff
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
groups: pflog
tailscale0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 1280
options=80000<LINKSTATE>
inet 100.121.116.111 netmask 0xffffffff broadcast 100.121.116.111
inet6 fd7a:115c:a1e0:ab12:4843:cd96:6279:746f prefixlen 48
groups: tun
nd6 options=101<PERFORMNUD,NO_DAD>
Opened by PID 95231

pf.conf

ext_if="re0"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo

table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"

block in all
pass out quick keep state
antispoof for $ext_if inet
pass in inet proto tcp from any to any port ssh flags S/SA keep state

rc.conf

clear_tmp_enable="YES"
syslogd_flags="-ss"
sendmail_enable="NONE"
hostname="xxx"
keymap="fr.kbd"
ifconfig_re0="DHCP"
ifconfig_re0_ipv6="inet6 accept_rtadv"
sshd_enable="YES"
powerd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
tailscaled_enable="YES"
bastille_enable="YES"
pf_enable="YES"
pflog_enable="YES"
cloned_interfaces="lo1"
ifconfig_lo1_name="bastille0"
gateway_enable="YES"

I use ansible and the great work of eoli3n (https://eoli3n.eu.org/2021/06/14/jails-part-3.html) to do all the things with Ansible :

/etc/devfs.rules

<!-- BEGIN ANSIBLE MANAGED vnet -->
[bastille_vnet=13]
add path 'bpf*' unhide
<!-- END ANSIBLE MANAGED vnet -->

Thanks for your help !

 

[patmaddox]

But there is something i don't understand, how could i redirect localhost to tailscale network, in other word : 127.0.0.1:2342 on loopback0 is same as 100.121.116.111:2342 to tailscale0.

Actually this is not the case, 100.121.116.111:2342 is not accessible. I don't understand how do reroute lo0 to tailscale0 ?

In my mind, after that i could redirect traffic to tailscale ip using traeffik reverse proxy without too much problem after that.

I'm not totally sure what you're trying to accomplish here, but I'll take my best stab at it.

It seems like you want to:

1. Run a service on bastille jail 1
2. Connect to that service from VPS using Tailscale
3. Expose the service to public internet via the VPS

Is that correct?

If so, it looks like the trouble you're having is on point #2, making the jail service available to the VPS. In that case, I think you have two basic options:

1. Bind the service to the Tailscale interface in jail 1. Now the VPS can connect directly to it.
2. Run a reverse proxy on the jail 1 interface, and have it reverse proxy to the service running on 127.0.0.1

In either case, you need some service running on the Tailscale interface. It's just a matter of whether you bind to it directly, or reverse proxy.

 

[srey]

Thanks patmaddox for your help, you're right about 1/2/3 step.

I continue to look about answer with tailscale/wireguard on different post/forum/reddit, and it seems that runing jail using VNet, running TailScale/Wireguard, is a better approach than installing tailscale directly using port on the freebsd serv.

I try to be more clear about my objective on this freebsd server :

- having a tailscale jail (TJ) connected with the VPS
- running a reverse proxy into this (TJ) to (=> i miss this point in my config your right !)
- set services (SJ) that run in other jails accessible from VPS

So this is more or less what you say at the end of your post.

My question is, how do you configure your different jails (SJ and TJ) to be connected on the same network when you run with VNET on each ?

 

[VladiBG]

https://freebsdfoundation.org/wp-content/uploads/2020/03/Jail-vnet-by-Examples.pdf

 

[patmaddox]

Ah, gotcha. I would not configure PF to connect the jails. I would use if_bridge(4) and epair(4). Create a bridge on the host machine. Create an epair for each jail, connect one side to the bridge and pass the other side to the jail. Give each one an IP in the jail. Then configure traefik to bind to tailscale IP, and have it reverse proxy requests to the appropriate jail IP.

You'll also need a way to connect to the internet from jail1. You could do that by adding the host's NIC to the bridge - and now the host will have direct access to the jails, which may be what you want, may not be. If you don't, then you'll want another bridge that only has jail1 epair and the host NIC.

 

[srey]

Ah, gotcha. I would not configure PF to connect the jails. I would use if_bridge(4) and epair(4). Create a bridge on the host machine. Create an epair for each jail, connect one side to the bridge and pass the other side to the jail. Give each one an IP in the jail. Then configure traefik to bind to tailscale IP, and have it reverse proxy requests to the appropriate jail IP.

You'll also need a way to connect to the internet from jail1. You could do that by adding the host's NIC to the bridge - and now the host will have direct access to the jails, which may be what you want, may not be. If you don't, then you'll want another bridge that only has jail1 epair and the host NIC.

Thanks this is more clear for me

It seems Bastille directly manage the creation of pair with -V option et -B for linking to an external bridge.
https://bastille.readthedocs.io/en/....html#virtual-network-vnet-on-external-bridge
https://bastille.readthedocs.io/en/latest/chapters/networking.html#virtual-network-vnet-on-external-bridge

I'm more interested by the second option patmaddox, to separate networks, this is more something like this if i understand well.

Need to found the good command now