I'm using Bastille to manage several jails, and things have been working great for a while now. But recently I decided to upgrade the server and its jails from 12.2-RELEASE to 13.0-RELEASE. The upgrade on the server went without a hitch, and upgrading the base jail through Bastille was also not an issue. But I had some trouble connecting to pkg.freebsd.org when trying to upgrade the jails themselves. Eventually, I was able to get all the jails using a loopback interface to connect and now they don't seem to have any issues.
But I have a couple of jails with a VNET connection and they are giving me problems. One uses a tun interface (transmission), but the other doesn't (radarr). And for troubleshooting, I created a new VNET jail through Bastille, but that also could not connect. For example, in a VNET jail, this is an error I get:
I'll include the relevant config files.
/etc/rc.conf
/etc/sysctl.conf
/etc/devfs.rules
/usr/local/bastille/jails/radarr/jail.conf
/usr/local/bastille/jails/transmission/jail.conf
/etc/pf.conf
The two rdr rules are commented out as I'd added the rdr-anchor rule after the upgrade. Apparently, I hadn't upgraded Bastille in a while. As it happens, adding the rdr rule dynamically was no problem for caddy, but when I tried the same with transmission, I got this error:
Another message I'm getting on stdout when updating the pf rules is this:
I'm afraid I'm not sure why either of those occur.
I have a pretty good feeling that my problem is with my pf config though, because if I disable pf, I can then access the internet from within the VNET jails.
But I have a couple of jails with a VNET connection and they are giving me problems. One uses a tun interface (transmission), but the other doesn't (radarr). And for troubleshooting, I created a new VNET jail through Bastille, but that also could not connect. For example, in a VNET jail, this is an error I get:
Code:
root@transmission:~ # host pkg.freebsd.org
;; connection timed out; no servers could be reachedI'll include the relevant config files.
/etc/rc.conf
Code:
clear_tmp_enable="YES"
sendmail_enable="NONE"
hostname="miniserver"
ifconfig_re0="inet 192.168.0.250 netmask 255.255.255.0"
defaultrouter="192.168.0.1"
sshd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
#NTP Server
ntpd_enable="YES"
# ntp time synchronization at startup
ntpd_sync_on_start="YES"
bastille_enable="YES"
cloned_interfaces="lo1"
ifconfig_lo1_name="bastille0"
pf_enable="YES"
gateway_enable="YES"
#Allow OpenVPN in jails
cloned_interfaces+="tun"
#ifconfig_tun="10.8.0.0/24"
jail_sysvipc_allow="YES"/etc/sysctl.conf
Code:
security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
security.bsd.see_jail_proc=0
security.bsd.unprivileged_read_msgbuf=0
security.bsd.unprivileged_proc_debug=0
vfs.zfs.min_auto_ashift=12
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_onlyip=0
net.link.bridge.pfil_member=0/etc/devfs.rules
Code:
[bastille_vnet=13]
add path 'bpf*' unhide/usr/local/bastille/jails/radarr/jail.conf
Code:
radarr {
devfs_ruleset = 13;
enforce_statfs = 2;
exec.clean;
exec.consolelog = /var/log/bastille/radarr_console.log;
exec.start = '/bin/sh /etc/rc';
exec.stop = '/bin/sh /etc/rc.shutdown';
host.hostname = radarr;
mount.devfs;
mount.fstab = /usr/local/bastille/jails/radarr/fstab;
path = /usr/local/bastille/jails/radarr/root;
securelevel = 2;
allow.mlock = 1;
allow.raw_sockets = 1;
vnet;
vnet.interface = e0b_bastille1;
exec.prestart += "jib addm bastille1 re0";
exec.poststop += "jib destroy bastille1";
}/usr/local/bastille/jails/transmission/jail.conf
Code:
transmission {
devfs_ruleset = 13;
enforce_statfs = 2;
exec.clean;
exec.consolelog = /var/log/bastille/transmission_console.log;
exec.start = '/bin/sh /etc/rc';
exec.stop = '/bin/sh /etc/rc.shutdown';
host.hostname = transmission;
mount.devfs;
mount.fstab = /usr/local/bastille/jails/transmission/fstab;
path = /usr/local/bastille/jails/transmission/root;
securelevel = 2;
vnet;
vnet.interface = e0b_bastille0;
exec.prestart += "jib addm bastille0 re0";
exec.poststop += "jib destroy bastille0";
}/etc/pf.conf
Code:
ext_if="re0"
tun_if="tun0"
caddy_ip="10.0.0.100"
transmission_ip="192.168.0.251"
tun_net="10.8.0.0/24"
set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo
table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"
transmission_jail_udp = "{ openvpn }" # is in /etc/services
nat on $ext_if from $tun_net to any -> $ext_if
## inbound to container example
## rdr pass inet proto tcp from any to any port {80, 443} -> 10.17.89.45
#rdr pass inet proto tcp from any to any port {80, 443} -> $caddy_ip
#rdr pass inet proto tcp from any to any port {9091, 51413} -> $transmission_ip
pass quick on $tun_if
block in all
pass out quick keep state
antispoof for $ext_if inet
pass in inet proto tcp from any to any port ssh flags S/SA keep stateThe two rdr rules are commented out as I'd added the rdr-anchor rule after the upgrade. Apparently, I hadn't upgraded Bastille in a while. As it happens, adding the rdr rule dynamically was no problem for caddy, but when I tried the same with transmission, I got this error:
Code:
❯ sudo bastille rdr transmission tcp 9091 9091
stdin:2: syntax error
pfctl: Syntax error in config file: pf rules not loadedAnother message I'm getting on stdout when updating the pf rules is this:
Code:
❯ sudo service pf status
/etc/rc.conf: cloned_interfaces+=tun: not found
/etc/rc.conf: cloned_interfaces+=tun: not foundI'm afraid I'm not sure why either of those occur.
I have a pretty good feeling that my problem is with my pf config though, because if I disable pf, I can then access the internet from within the VNET jails.
