EDIT: PLEASE NOTE that I've now got things mostly working; the one remaining problem that I know of is that the jail host can't ping the jails (while other machines on my intranet can ping them). The config shown in this initial post is now out of date; the new config (and further details about the current remaining issue) can be found further downthread. Thank you!
I've been trying to get a jail up using VNET. I have been running into problems with network connectivity between the jail and the outside world (both directions). Through various stab-in-the-dark-ish changes, I've been able to get connectivity to fail with various different symptoms, but I've been unable to ever get it working.
The current symptoms:
ping:
* From the jail host to the jail: "ping: sendto: Host is down".
* From the jail to the jail host: same thing.
route get:
* On the jail host for the jail: Says "interface: host". That surprised me, as my uneducated assumption would be that it should show the interface as the bridge, or the endpoint of the bridge, or pretty much anything *other* than "host".
* In the jail for the jail host: Takes a long time, but eventually comes back with "interface: vnet0".
ssh (with sshd up and running on both):
* From the host to the jail: It seems to just hang, but eventually comes back with "Operation timed out".
* From the jail to the host: Same thing.
I am new to jails (though as proof-of-concept in preparation for what I'm doing now, I was able to successfully get them up using simpler forms of connectivity, e.g. the loopback/alias things, both with Bastille and with "raw" jails). I am new to VNET. As for networking in general, I am definitely no expert, but I'm not a newbie and I feel that I have a working grasp of at least some of the basics.
I am using FreeBSD-13.1-RELEASE-p2 and Bastille. My machine has two different physical IP interfaces, one of which I've named "host" and am using for the "main" part of the machine (the "jail host"), and the other I've named "jails" and am using (well... trying to use) for the jails. I've got IPv4 set up for both, and IPv6 for neither.
I've been trying to piece this all together from a variety of sources, including the Bastille documentation, "Absolute FreeBSD" by Michael W. Lucas, "FreeBSD Mastery: Jails" by the same author, and a lot of googling.
The current config for my machine is as follows. I'd appreciate any help very much. Thanks in advance.
(jail host) /etc/rc.conf:
(jail host) ifconfig (much of which was set up automatically by bastille):
(jail host) /etc/devfs.rules (set manually, without understanding, based on documentation):
(jail host) a few sysctl values that I set manually, without understanding, based on documentation:
(jail host) /usr/local/etc/bastille/bastille.conf (exactly the default config, except for two ZFS lines):
(jail host) /usr/local/bastille/jails/fstab (created automatically by bastille):
(jail host) /usr/local/bastille/jails/jail.conf (created automatically by bastille):
(jail) /etc/rc.conf:
(jail) ifconfig:
I've been trying to get a jail up using VNET. I have been running into problems with network connectivity between the jail and the outside world (both directions). Through various stab-in-the-dark-ish changes, I've been able to get connectivity to fail with various different symptoms, but I've been unable to ever get it working.
The current symptoms:
ping:
* From the jail host to the jail: "ping: sendto: Host is down".
* From the jail to the jail host: same thing.
route get:
* On the jail host for the jail: Says "interface: host". That surprised me, as my uneducated assumption would be that it should show the interface as the bridge, or the endpoint of the bridge, or pretty much anything *other* than "host".
* In the jail for the jail host: Takes a long time, but eventually comes back with "interface: vnet0".
ssh (with sshd up and running on both):
* From the host to the jail: It seems to just hang, but eventually comes back with "Operation timed out".
* From the jail to the host: Same thing.
I am new to jails (though as proof-of-concept in preparation for what I'm doing now, I was able to successfully get them up using simpler forms of connectivity, e.g. the loopback/alias things, both with Bastille and with "raw" jails). I am new to VNET. As for networking in general, I am definitely no expert, but I'm not a newbie and I feel that I have a working grasp of at least some of the basics.
I am using FreeBSD-13.1-RELEASE-p2 and Bastille. My machine has two different physical IP interfaces, one of which I've named "host" and am using for the "main" part of the machine (the "jail host"), and the other I've named "jails" and am using (well... trying to use) for the jails. I've got IPv4 set up for both, and IPv6 for neither.
I've been trying to piece this all together from a variety of sources, including the Bastille documentation, "Absolute FreeBSD" by Michael W. Lucas, "FreeBSD Mastery: Jails" by the same author, and a lot of googling.
The current config for my machine is as follows. I'd appreciate any help very much. Thanks in advance.
(jail host) /etc/rc.conf:
Code:
clear_tmp_enable="YES"
syslogd_flags="-ss"
sendmail_enable="NONE"
hostname="silicon.vestertopia.net"
ifconfig_bce0_name="host"
ifconfig_bce1_name="jails"
ifconfig_host="inet 192.168.0.14 netmask 255.255.128.0"
ifconfig_jails="up"
defaultrouter="192.168.0.1"
sshd_enable="YES"
ntpd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
smartd_enable="YES"
gateway_enable="YES"(jail host) ifconfig (much of which was set up automatically by bastille):
Code:
host: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 84:2b:2b:66:dd:c3
inet 192.168.0.14 netmask 0xffff8000 broadcast 192.168.127.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
jails: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
ether 84:2b:2b:66:dd:c4
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
jailsbridge: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 58:9c:fc:10:7a:17
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: e0a_bastille0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 2000
member: jails flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 2 priority 128 path cost 20000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
e0a_bastille0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: vnet host interface for Bastille jail pocjail
options=8<VLAN_MTU>
ether 02:20:98:66:dd:c4
hwaddr 02:ea:dc:ca:c3:0a
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>(jail host) /etc/devfs.rules (set manually, without understanding, based on documentation):
Code:
[bastille_vnet=13]
add path 'bpf*' unhide(jail host) a few sysctl values that I set manually, without understanding, based on documentation:
Code:
net.link.bridge.pfil_member: 0
net.link.bridge.pfil_bridge: 0
net.link.bridge.pfil_onlyip: 0(jail host) /usr/local/etc/bastille/bastille.conf (exactly the default config, except for two ZFS lines):
Code:
#####################
## [ BastilleBSD ] ##
#####################
## default paths
bastille_prefix="/usr/local/bastille" ## default: "/usr/local/bastille"
bastille_backupsdir="${bastille_prefix}/backups" ## default: "${bastille_prefix}/backups"
bastille_cachedir="${bastille_prefix}/cache" ## default: "${bastille_prefix}/cache"
bastille_jailsdir="${bastille_prefix}/jails" ## default: "${bastille_prefix}/jails"
bastille_releasesdir="${bastille_prefix}/releases" ## default: "${bastille_prefix}/releases"
bastille_templatesdir="${bastille_prefix}/templates" ## default: "${bastille_prefix}/templates"
bastille_logsdir="/var/log/bastille" ## default: "/var/log/bastille"
## bastille scripts directory (assumed by bastille pkg)
bastille_sharedir="/usr/local/share/bastille" ## default: "/usr/local/share/bastille"
## bootstrap archives, which components of the OS to install.
## base - The base OS, kernel + userland
## lib32 - Libraries for compatibility with 32 bit binaries
## ports - The FreeBSD ports (3rd party applications) tree
## src - The source code to the kernel + userland
## test - The FreeBSD test suite
## this is a whitespace separated list:
## bastille_bootstrap_archives="base lib32 ports src test"
bastille_bootstrap_archives="base" ## default: "base"
## default timezone
bastille_tzdata="" ## default: empty to use host's time zone
## default jail resolv.conf
bastille_resolv_conf="/etc/resolv.conf" ## default: "/etc/resolv.conf"
## bootstrap urls
bastille_url_freebsd="[URL]http://ftp.freebsd.org/pub/FreeBSD/releases/[/URL]" ## default: "[URL]http://ftp.freebsd.org/pub/Fre[/URL]
eBSD/releases/"
bastille_url_hardenedbsd="[URL]http://installer.hardenedbsd.org/pub/hardenedbsd/[/URL]" ## default: "[URL]https://installer.hardenedbsd[/URL].
org/pub/HardenedBSD/releases/"
bastille_url_midnightbsd="[URL]https://www.midnightbsd.org/ftp/MidnightBSD/releases/[/URL]" ## default: "[URL]https://www.midni[/URL]
ghtbsd.org/pub/MidnightBSD/releases/"
## ZFS options
bastille_zfs_enable="YES" ## default: ""
bastille_zfs_zpool="zroot" ## default: ""
bastille_zfs_prefix="bastille" ## default: "${bastille_zfs_zpool}/bastille"
bastille_zfs_options="-o compress=lz4 -o atime=off" ## default: "-o compress=lz4 -o atime=off"
## Export/Import options
bastille_compress_xz_options="-0 -v" ## default "-0 -v"
bastille_decompress_xz_options="-c -d -v" ## default "-c -d -v"
bastille_compress_gz_options="-1 -v" ## default "-1 -v"
bastille_decompress_gz_options="-k -d -c -v" ## default "-k -d -c -v"
## Networking
bastille_network_loopback="bastille0" ## default: "bastille0"
bastille_network_shared="" ## default: ""
bastille_network_gateway="" ## default: ""
bastille_url_midnightbsd="[URL]https://www.midnightbsd.org/ftp/MidnightBSD/releases/[/URL]" ## default: "[URL]https://www.midni[/URL]
ghtbsd.org/pub/MidnightBSD/releases/"
## ZFS options
bastille_zfs_enable="YES" ## default: ""
bastille_zfs_zpool="zroot" ## default: ""
bastille_zfs_prefix="bastille" ## default: "${bastille_zfs_zpool}/bastille"
bastille_zfs_options="-o compress=lz4 -o atime=off" ## default: "-o compress=lz4 -o atime=off"
## Export/Import options
bastille_compress_xz_options="-0 -v" ## default "-0 -v"
bastille_decompress_xz_options="-c -d -v" ## default "-c -d -v"
bastille_compress_gz_options="-1 -v" ## default "-1 -v"
bastille_decompress_gz_options="-k -d -c -v" ## default "-k -d -c -v"
## Networking
bastille_network_loopback="bastille0" ## default: "bastille0"
bastille_network_shared="" ## default: ""
bastille_network_gateway="" ## default: ""
bastille_network_gateway6="" ## default: ""
## Default Templates
bastille_template_base="default/base" ## default: "default/base"
bastille_template_empty="" ## default: "default/empty"
bastille_template_thick="default/thick" ## default: "default/thick"
bastille_template_clone="default/clone" ## default: "default/clone"
bastille_template_thin="default/thin" ## default: "default/thin"
bastille_template_vnet="default/vnet" ## default: "default/vnet"(jail host) /usr/local/bastille/jails/fstab (created automatically by bastille):
Code:
/usr/local/bastille/releases/13.1-RELEASE /usr/local/bastille/jails/pocjail/root/.bastille nullfs ro 0 0(jail host) /usr/local/bastille/jails/jail.conf (created automatically by bastille):
Code:
pocjail {
devfs_ruleset = 13;
enforce_statfs = 2;
exec.clean;
exec.consolelog = /var/log/bastille/pocjail_console.log;
exec.start = '/bin/sh /etc/rc';
exec.stop = '/bin/sh /etc/rc.shutdown';
host.hostname = pocjail;
mount.devfs;
mount.fstab = /usr/local/bastille/jails/pocjail/fstab;
path = /usr/local/bastille/jails/pocjail/root;
securelevel = 2;
vnet;
vnet.interface = e0b_bastille0;
exec.prestart += "jib addm bastille0 jails";
exec.prestart += "ifconfig e0a_bastille0 description \"vnet host interface for Bastille jail pocjail\"";
exec.poststop += "jib destroy bastille0";
}(jail) /etc/rc.conf:
Code:
syslogd_flags="-ss"
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
cron_flags="-J 60"
ifconfig_e0b_bastille0_name="vnet0"
ifconfig_vnet0="inet 192.168.14.1/17"
defaultrouter="192.168.0.1"(jail) ifconfig:
Code:
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 0e:20:98:66:dd:c4
hwaddr 02:ea:dc:ca:c3:0b
inet 192.168.14.1 netmask 0xffff8000 broadcast 192.168.127.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Last edited:
