Hi,
I am having issues setting up network on a dedicated server.
Basic network tests are failing (cf. end of this message)
1/ Here is main objectives
2/ Global solution
Here is what I had in mind, but maybe this is not the best way of doing things (don't hesitate to says so)
3/ Detailed implementation
Based on "2/ Global solution", here is what I have done so far.
HOST rc.conf :
HOST /boot/loader.conf :
HOST IPFW.rules
IOCAGE JAIL CONFIG
JTEST ifconfig
JTEST ipfw
JTEST rc.conf :
Network tests on HOST
Network tests on JTEST
I am having issues setting up network on a dedicated server.
Basic network tests are failing (cf. end of this message)
1/ Here is main objectives
- Secure both HOST and JAILS
- Jails must be able to access public IP (Internet)
- Jails MUST NOT be able to see any other jail than themself
- NB : I will have as many jails as there are services to be available on my public IP
Code:
[ jail vnet0 (10.0.1.2) ]
↓
[ bridge0 (10.0.1.1) ]
↓
[ igb0 (217.x.x.x) ]
↓
Internet2/ Global solution
Here is what I had in mind, but maybe this is not the best way of doing things (don't hesitate to says so)
- HOST server : Use IPFW to secure this machine
- HOST server : Use IOCAGE to manage jails
- HOST server : Manage public services, by routing ports to the jail providing the required service (for example port 80 and 443 would be routed to JAIL1 which provides public website)
3/ Detailed implementation
Based on "2/ Global solution", here is what I have done so far.
- network interface : igb0 with public IP 217.182.1.2
- network bridge interface: bridge0 with local IP : 10.0.1.1
- bridge members : igb0 and all the jails' vnet interfaces (vnet0.*)
Code:
HOST# ifconfig
igb0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,HWSTATS>
ether ac:1f:6b:00:c5:0c
inet 217.182.1.2 netmask 0xffffff00 broadcast 217.182.1.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=0
ether 58:9c:fc:10:e4:6c
inet 10.0.1.1 netmask 0xffffff00 broadcast 10.0.1.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet0.10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000
member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 20000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.10: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: associated with jail: JTEST_14_2-RELEASE as nic: epair0b
options=8<VLAN_MTU>
ether ae:1f:6b:b2:4b:e5
hwaddr 02:b5:25:93:bc:0a
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>HOST rc.conf :
Code:
HOST# cat /etc/rc.conf (network/firewall parts)
ifconfig_igb0="DHCP"
firewall_enable="YES"
firewall_type="open"
firewall_script="/etc/ipfw_configure.rules"
iocage_enable="YES"HOST /boot/loader.conf :
Code:
HOST# cat /boot/loader.conf (relevant parts)
vnet.enable="YES"
ipfw_load="YES"
ipfw_nat_load="YES"
security.jail.allow_raw_sockets=1
security.jail.chflags_allowed=1
net.inet.ip.forwarding=1
net.inet.ip.fw.verbose=1
net.inet.ip.fw.verbose_limit=5HOST IPFW.rules
Code:
HOST# cat ipfw_configure.rules
ipfw -q -f flush
ipfw add 10 allow all from any to any via lo0
ipfw add 20 allow tcp from any to any established
ipfw add 30 allow ip from any to any out via igb0 keep-state
ipfw add 100 nat 1 ip from 10.0.1.0/24 to any out via igb0
ipfw add 1001 allow tcp from any to any dst-port 22 in via igb0- I've setup a test jail named JTEST_14.2-RELEASE. Here is the current jail config :
IOCAGE JAIL CONFIG
Code:
HOST# iocage get all JTEST_14.2-RELEASE
CONFIG_VERSION:33
allow_chflags:0
allow_mlock:0
allow_mount:0
allow_mount_devfs:0
allow_mount_fdescfs:0
allow_mount_fusefs:0
allow_mount_linprocfs:0
allow_mount_linsysfs:0
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0
allow_mount_zfs:0
allow_nfsd:0
allow_quotas:0
allow_raw_sockets:1
allow_set_hostname:1
allow_socket_af:0
allow_sysvipc:0
allow_tun:0
allow_vmm:0
assign_localhost:0
available:readonly
basejail:0
boot:1
bpf:0
children_max:0
cloned_release:14.2-RELEASE
comment:none
compression:on
compressratio:readonly
coredumpsize:off
count:1
cpuset:off
cputime:off
datasize:off
dedup:off
defaultrouter:none
defaultrouter6:auto
depends:none
devfs_ruleset:4
dhcp:0
enforce_statfs:2
exec_clean:1
exec_created:/usr/bin/true
exec_fib:0
exec_jail_user:root
exec_poststart:/usr/bin/true
exec_poststop:/usr/bin/true
exec_prestart:/usr/bin/true
exec_prestop:/usr/bin/true
exec_start:/bin/sh /etc/rc
exec_stop:/bin/sh /etc/rc.shutdown
exec_system_jail_user:0
exec_system_user:root
exec_timeout:60
host_domainname:none
host_hostname:JTEST-14.2-RELEASE
host_hostuuid:JTEST_14.2-RELEASE
host_time:1
hostid:00000000-0000-0000-0000-ac1f6b00c50c
hostid_strict_check:0
interfaces:vnet0:bridge0
ip4:new
ip4_addr:vnet0|10.0.1.2/24
ip4_saddrsel:1
ip6:new
ip6_addr:none
ip6_saddrsel:1
ip_hostname:0
jail_zfs:0
jail_zfs_dataset:iocage/jails/JTEST_14.2-RELEASE/data
jail_zfs_mountpoint:none
last_started:2025-04-11 22:24:01
localhost_ip:none
login_flags:-f root
mac_prefix:ae1f6b
maxproc:off
memorylocked:off
memoryuse:off
min_dyn_devfs_ruleset:1000
mount_devfs:1
mount_fdescfs:1
mount_linprocfs:0
mount_procfs:0
mountpoint:readonly
msgqqueued:off
msgqsize:off
nat:0
nat_backend:ipfw
nat_forwards:none
nat_interface:none
nat_prefix:172.16
nmsgq:off
notes:none
nsem:off
nsemop:off
nshm:off
nthr:off
openfiles:off
origin:readonly
owner:root
pcpu:off
plugin_name:none
plugin_repository:none
priority:99
pseudoterminals:off
quota:none
readbps:off
readiops:off
release:14.2-RELEASE-p2
reservation:none
resolver:/etc/resolv.conf
rlimits:off
rtsold:0
securelevel:2
shmsize:off
stacksize:off
state:up
stop_timeout:30
swapuse:off
sync_state:none
sync_target:none
sync_tgt_zpool:none
sysvmsg:new
sysvsem:new
sysvshm:new
template:0
type:jail
used:readonly
vmemoryuse:off
vnet:1
vnet0_mac:ae1f6bb24be5 ae1f6bb24be6
vnet0_mtu:auto
vnet1_mac:none
vnet1_mtu:auto
vnet2_mac:none
vnet2_mtu:auto
vnet3_mac:none
vnet3_mtu:auto
vnet_default_interface:auto
vnet_default_mtu:1500
vnet_interfaces:none
wallclock:off
writebps:off
writeiops:offJTEST ifconfig
Code:
JTEST# ifconfig
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair0b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=8<VLAN_MTU>
ether ae:1f:6b:b2:4b:e6
hwaddr 02:16:92:c4:aa:0b
inet 10.0.1.2 netmask 0xffffff00 broadcast 10.0.1.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>JTEST ipfw
Code:
JTEST# ipfw show
00100 0 0 allow ip from any to any via lo0
00200 0 0 deny ip from any to 127.0.0.0/8
00300 0 0 deny ip from 127.0.0.0/8 to any
00400 0 0 deny ip from any to ::1
00500 0 0 deny ip from ::1 to any
00600 0 0 allow ipv6-icmp from :: to ff02::/16
00700 0 0 allow ipv6-icmp from fe80::/10 to fe80::/10
00800 0 0 allow ipv6-icmp from fe80::/10 to ff02::/16
00900 0 0 allow ipv6-icmp from any to any icmp6types 1
01000 0 0 allow ipv6-icmp from any to any icmp6types 2,135,136
65000 547 44256 allow ip from any to any
65535 0 0 deny ip from any to anyJTEST rc.conf :
Code:
JTEST# cat /etc/rc.conf (network/firewall parts)
firewall_enable="YES"
firewall_type="open"Network tests on HOST
Code:
HOST# ping 127.0.0.1
PING 127.0.0.1 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.030 ms
Code:
HOST# ping 217.182.1.2
PING 217.182.1.2 (217.182.1.2): 56 data bytes
64 bytes from 217.182.1.2: icmp_seq=0 ttl=64 time=0.029 ms
Code:
HOST# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=112 time=4.810 ms
Code:
HOST# ping 10.0.1.1
PING 10.0.1.1 (10.0.1.1): 56 data bytes
(.... Never responds ....)
Code:
HOST# ping 10.0.1.2
PING 10.0.1.2 (10.0.1.2): 56 data bytes
(.... Never responds ....)
Code:
HOST# netstat -rn
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 217.182.1.254 UGS igb0
10.0.1.0/24 link#5 U bridge0
10.0.1.1 link#3 UHS lo0
127.0.0.1 link#3 UH lo0
217.182.1.0/24 link#1 U igb0
217.182.1.2 link#3 UHS lo0
Internet6:
Destination Gateway Flags Netif Expire
::/96 link#3 URS lo0
::1 link#3 UHS lo0
::ffff:0.0.0.0/96 link#3 URS lo0
fe80::%lo0/10 link#3 URS lo0
fe80::%lo0/64 link#3 U lo0
fe80::1%lo0 link#3 UHS lo0
ff02::/16 link#3 URS lo0Network tests on JTEST
Code:
JTEST# ping 127.0.0.1
PING 127.0.0.1 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.026 ms
Code:
JTEST# ping 10.0.1.2
PING 10.0.1.2 (10.0.1.2): 56 data bytes
64 bytes from 10.0.1.2: icmp_seq=0 ttl=64 time=0.031 ms
Code:
JTEST# ping 10.0.1.1
PING 10.0.1.1 (10.0.1.1): 56 data bytes
(.... Never responds ....)
Code:
JTEST# ping 217.182.1.2
PING 217.182.1.2 (217.182.1.2): 56 data bytes
ping: sendto: Network is unreachable
ping: sendto: Network is unreachable
ping: sendto: Network is unreachable
Code:
JTEST# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: Network is unreachable
ping: sendto: Network is unreachable
ping: sendto: Network is unreachable
Code:
HOST# netstat -rn
Routing tables
Internet:
Destination Gateway Flags Netif Expire
10.0.1.0/24 link#7 U epair0b
10.0.1.2 link#4 UHS lo0
127.0.0.1 link#4 UH lo0
Internet6:
Destination Gateway Flags Netif Expire
::/96 link#4 URS lo0
::1 link#4 UHS lo0
::ffff:0.0.0.0/96 link#4 URS lo0
fe80::%lo0/10 link#4 URS lo0
fe80::%lo0/64 link#4 U lo0
fe80::1%lo0 link#4 UHS lo0
ff02::/16 link#4 URS lo0
