All the vnet literature talks about a vnet jail having it's own separate ip stack. I and most everyone else interpreted this to mean that the vnet jail's stack was connected directly to the host's external interface facing the public internet WITHOUT the host's firewall knowing anything about that vnet traffic.
This interpretation is not correct. All bidirectional external interface traffic passes through the hosts firewall including vnet traffic before its handed off to the vnet stack. After many years playing around with vnet starting at release 9.0 and each following release 10.0, 11.0 and 12.1 where I finally got vnet jails to work using just native FreeBSD jail, bridge, and epair set of commands.
I am running FreeBSD RELEASE 12.1-p6 on real hardware. em0 is the host interface connected to the public network with a dynamic ip address obtained from the ISP by DHCP. To populate my working vnet jail directory tree I did this. It's called a full jail because it has a complete install of the FreeBSD 12.1 directory tree. This method can be duplicated using any of the Freebsd RELEASE versions as new RELEASES are published.
Keep in mind that the version of Freebsd running on the host must be the same version used to populate the vnet jail directory tree as the only sure way of eliminating software conflicts and operational functions from causing errors or jail crashes. When a vnet jail crashes the contents of any open files will be lost so be sure to make a backup copy of the host's /etc/jail.conf file.
This is a simple way to populate your full directory tree for non-vnet jails and vnet jails.
/etc/jail.conf on the host
If you want more vnet jails, just duplicate the testjail statements in /etc/jail.conf and enter different jailname
as well as unpack base.txz to new jail directory tree.
Change the IP address in
to 10.0.101.55. Increase the 100 by 1 for each new jail. All the jails on bridge0 will be able to access the public
internet and each other. If you don't want your vnet jails to access the public internet, just be able to access
each other then don't assign the public interface to the bridge. Put this in the hosts /etc/rc.conf
Vnet jails can have their own firewall. But you need to know about these mandatory requirements.
ipfw, pf, and ipfilter firewalls are vnet aware but you have to run the same firewall on the host as you want to run in your vnet jails.
Beware: ipfw in a vnet jail defaults to intermixing it's log records with the hosts ipfw log records.
If you don't want this to occur add firewall_logif="YES" to the vnet's rc.conf where you have the other ipfw firewall statements. This will cause the vnet jails ipfw firewall to log to /var/log/security in the vnet jail.
If you want your vnet jail to use a firewall within the vnet jail itself, than take notice; you have to create a unique devfs_ruleset to enable the devices used internally by the firewall you want to use.
# devfsrules to allow dhclient(8) and tcpdump to work inside a vnet jail.
[bpf=50]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_jail
add path 'bpf*' unhide
# devfsrules for ipfilter to function in a vnet jail.
[vnet_ipf=60]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_jail
add path ipl unhide
add path ipl0 unhide
add path ipf unhide
add path ipauth unhide
add path ipnat unhide
add path ipstate unhide
# used by ipstate
#add path kmem unhide
#add path kernel unhide
# devfsrules for pf to function in a vnet jail.
[vnet_pf=70]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_jail
add path 'bpf*' unhide
add path pf unhide
add path pflog unhide
add path pfsync unhide
# Been a long time since I played with ipfw in a vnet jail.
# Thinking rule 4 is all the vnet jail needs to use ipfw firewall.
Very important detail; The host firewall must NAT the private ipv4 addresses used.
This interpretation is not correct. All bidirectional external interface traffic passes through the hosts firewall including vnet traffic before its handed off to the vnet stack. After many years playing around with vnet starting at release 9.0 and each following release 10.0, 11.0 and 12.1 where I finally got vnet jails to work using just native FreeBSD jail, bridge, and epair set of commands.
I am running FreeBSD RELEASE 12.1-p6 on real hardware. em0 is the host interface connected to the public network with a dynamic ip address obtained from the ISP by DHCP. To populate my working vnet jail directory tree I did this. It's called a full jail because it has a complete install of the FreeBSD 12.1 directory tree. This method can be duplicated using any of the Freebsd RELEASE versions as new RELEASES are published.
Keep in mind that the version of Freebsd running on the host must be the same version used to populate the vnet jail directory tree as the only sure way of eliminating software conflicts and operational functions from causing errors or jail crashes. When a vnet jail crashes the contents of any open files will be lost so be sure to make a backup copy of the host's /etc/jail.conf file.
This is a simple way to populate your full directory tree for non-vnet jails and vnet jails.
Code:
# download the base.txz file to the host
cd /usr
fetch -avrA http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/12.1-RELEASE/base.txz
# unpack base.txz to directory tree. Modify the directory path location and change testjail to the name you want to call your jail.
mkdir -p /usr/jails/testjail
cd /usr/jails
xzdec base.txz | tar --unlink -xpJf - -C /usr/jails/testjail
# Prep jail directory. Copy required files from host to vnet jail and turn off sendmail in the vnet jail.
cp /etc/localtime testjail/etc
cp /etc/resolv.conf testjail/etc
echo "sendmail_enable="none"" > testjail/etc/rc.conf
echo "sendmail_submit_enable="none"" >> testjail/etc/rc.conf
echo "sendmail_outbound_enable="none"" >> testjail/etc/rc.conf
echo "sendmail_msp_queue_enable="none"" >> testjail/etc/rc.conf/etc/jail.conf on the host
Code:
#
# Using manual command method FreeBSD 12.1
# with assigned ip address for epairb and bridge.
# Start and stop vnet jail works without crashing the host because
# of the embedded sleep command that work around the tear-down bug that
# is now fixed in the soon to be published FreeBSD 13 RELEASE.
# From within the vnet jail can ping the bridge private ip,
# host public ip and the public internet. ping -c 2 1.1.1.1 0% packet loss
# The ipv6 addresses are assigned by your isp and is the full public routable prefix
# ending with a unique number for that interface.
#
# Prep the bridge.
# Add these lines to the hosts /etc/rc.conf & reboot to enable.
cloned_interfaces="bridge0"
ifconfig_bridge0="inet 10.0.100.1/24 addm em0 up"
ifconfig_bridge0="descr vnet-jail-bridge"
# Assigns a description able to be seen with ifconfig issued from the host
ifconfig_bridge0_ipv6="inet6 26ab:3c02:e003:16d:afaf::2 prefixlen 64"
#
# You can use any of the private reserved ipv4 address ranges.
#
# Or you can issue these commands from the host console
ifconfig bridge0 create up
ifconfig bridge0 inet 10.0.100.1/24 addm em0
ifconfig bridge0 descr vnet-jail-bridge
ifconfig bridge0 ipv6 inet6 26ab:3c02:e003:16d:afaf::2 prefixlen 64
#
#
# Using native jail command for start and stop of vnet jail.
# -v = verbose outputs log of what start process is really doing
# [ jail -vc jailname ] to start, [ jail -vr jailname ] to stop
# [ service jail [start stop] jailname ] works also.
#
# [ jexec jailname login -f root ] to login to the vnet jail from host
# [ jls ] to see what jails are running
# Change every where you see "testjail" to the name of your jail.
testjail {
host.hostname = "vnet_testjail";
path = "/usr/jails/testjail";
exec.consolelog = "/var/log/vnet_testjail.console.log";
mount.devfs = "true";
devfs_ruleset = "4";
vnet = "new";
vnet.interface = "epair1b";
exec.prestart = "ifconfig epair1 create up";
exec.prestart += "ifconfig epair1a up descr vnet-testjail
exec.prestart += "ifconfig bridge0 addm epair1a";
exec.start = "/bin/sh /etc/rc";
exec.start += "ifconfig epair1b inet 10.0.100.55 netmask 255.255.255.0";
exec.start += "route add default 10.0.100.1";
exec.start += "ifconfig epair1b inet6 26ab:3c02:e003:16d:afaf::2000"
exec.start += "route add -inet6 default 26ab:3c02:e003:16d:afaf::2"
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.poststop = "ifconfig epair1b -vnet testjail";
exec.poststop += "ifconfig bridge0 deletem epair1a";
exec.poststop += "sleep 2";
exec.poststop += "ifconfig epair1a destroy";
}If you want more vnet jails, just duplicate the testjail statements in /etc/jail.conf and enter different jailname
as well as unpack base.txz to new jail directory tree.
Change the IP address in
exec.start += "ifconfig epair1b inet 10.0.100.55 netmask 255.255.255.0";to 10.0.101.55. Increase the 100 by 1 for each new jail. All the jails on bridge0 will be able to access the public
internet and each other. If you don't want your vnet jails to access the public internet, just be able to access
each other then don't assign the public interface to the bridge. Put this in the hosts /etc/rc.conf
Code:
cloned_interfaces="bridge0"
ifconfig_bridge0="inet 10.0.100.1/24 up" [ the addm em0 option is missing.]Vnet jails can have their own firewall. But you need to know about these mandatory requirements.
ipfw, pf, and ipfilter firewalls are vnet aware but you have to run the same firewall on the host as you want to run in your vnet jails.
Beware: ipfw in a vnet jail defaults to intermixing it's log records with the hosts ipfw log records.
If you don't want this to occur add firewall_logif="YES" to the vnet's rc.conf where you have the other ipfw firewall statements. This will cause the vnet jails ipfw firewall to log to /var/log/security in the vnet jail.
If you want your vnet jail to use a firewall within the vnet jail itself, than take notice; you have to create a unique devfs_ruleset to enable the devices used internally by the firewall you want to use.
# devfsrules to allow dhclient(8) and tcpdump to work inside a vnet jail.
[bpf=50]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_jail
add path 'bpf*' unhide
# devfsrules for ipfilter to function in a vnet jail.
[vnet_ipf=60]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_jail
add path ipl unhide
add path ipl0 unhide
add path ipf unhide
add path ipauth unhide
add path ipnat unhide
add path ipstate unhide
# used by ipstate
#add path kmem unhide
#add path kernel unhide
# devfsrules for pf to function in a vnet jail.
[vnet_pf=70]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_jail
add path 'bpf*' unhide
add path pf unhide
add path pflog unhide
add path pfsync unhide
# Been a long time since I played with ipfw in a vnet jail.
# Thinking rule 4 is all the vnet jail needs to use ipfw firewall.
Very important detail; The host firewall must NAT the private ipv4 addresses used.
Last edited:
