Jails with IPFW and no internet inside jails

U

Uriel2000

Hello.

I'm trying to set up basic jails on FreeBSD 13.0-RELEASE. After 3 days of struggling with no internet access inside jails, I give up with troubleshooting and reach out here for help.

This is my configuration and what I've achieved so far:

/etc/rc.conf:

Code: 
root@freebsd13:~# cat /etc/rc.conf
hostname="freebsd13"
zfs_enable="YES"
jail_enable="YES"
firewall_enable="YES"
firewall_nat_enable="YES"
firewall_nat_interface="vtnet0"
firewall_type="/etc/ipfw.rules"
dumpdev="NO"
sendmail_enable="NONE"
ntpd_enable="YES"
sshd_enable="YES"
syslogd_flags="-ss"
ifconfig_vtnet0="DHCP"
cloned_interfaces="lo1"
ifconfig_lo1="inet 10.0.0.1 netmask 255.255.255.0"
ifconfig_lo1_alias1="inet 10.0.0.2 netmask 255.255.255.255"
ifconfig_lo1_alias1="inet 10.0.0.3 netmask 255.255.255.255"

/etc/sysctl.conf:

Code: 
vfs.zfs.min_auto_ashift=12
net.inet.ip.fw.one_pass=0

/etc/jails.conf:

Code: 
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
securelevel = 2;

jail1 {
  host.hostname = "jail1.local";
  path = "/jails/jail1";
  ip4.addr = "lo1|10.0.0.2";
  persist;
}

jail2 {
  host.hostname = "jail2.local";
  path = "/jails/jail2";
  ip4.addr = "lo1|10.0.0.3";
  persist;
}

And finally, /etc/ipfw.rules:
I did use this guide: https://paulgorman.org/technical/freebsd-jails.txt.html

Code: 
nat 1 config if vtnet0 reset
add 10 allow all from any to any via lo0
add 25 allow all from any to any via lo1
add 50 deny ip from any to any not antispoof in
add 100 nat 1 ip from any to any via vtnet0 in
add 110 check-state
add 900 skipto 10000 udp from any to any out via vtnet0 keep-state
add 910 skipto 10000 tcp from any to any out via vtnet0 keep-state
#add 1000 skipto 10000 udp from any to any 53 out via vtnet0 keep-state
#add 1010 skipto 10000 tcp from any to any 53 out via vtnet0 keep-state
#add 1100 skipto 10000 tcp from any to any 80,443 out via vtnet0 keep-state
#add 1300 skipto 10000 udp from any to any 123 out via vtnet0 keep-state
add 2000 allow tcp from any to any established
add 5200 allow tcp from any to any 80 keep-state
add 5300 allow tcp from any to any 443 keep-state
add 6100 allow tcp from any to me 22 in keep-state
add 6200 allow tcp from me to any 25 out keep-state
add 6300 allow icmp from any to any via vtnet0 out keep-state
add 9999 deny all from any to any
add 10000 nat 1 ip from any to any via vtnet0 out
add 10001 allow ip from any to any

Commands output/debug information:

Code: 
root@freebsd13:~# ifconfig -a
vtnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
        ether 52:54:00:ea:99:22
        inet 192.168.100.242 netmask 0xffffff00 broadcast 192.168.100.255
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 10.0.0.1 netmask 0xffffff00
        inet 10.0.0.2 netmask 0xffffffff
        inet 10.0.0.3 netmask 0xffffffff
        inet6 fe80::1%lo1 prefixlen 64 scopeid 0x3
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

root@freebsd13:~# netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.100.1      UGS      vtnet0
10.0.0.1           link#3             UH          lo1
10.0.0.2           link#3             UH          lo1
10.0.0.3           link#3             UH          lo1
127.0.0.1          link#2             UH          lo0
192.168.100.0/24   link#1             U        vtnet0
192.168.100.242    link#1             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#2                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0
fe80::%lo1/64                     link#3                        U           lo1
fe80::1%lo1                       link#3                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0

root@freebsd13:~# ipfw list
00010 allow ip from any to any via lo0
00025 allow ip from any to any via lo1
00050 deny ip from any to any not antispoof in
00100 allow ip from any to any via lo0
00100 nat 1 ip from any to any via vtnet0 in
00110 check-state :default
00200 deny ip from any to 127.0.0.0/8
00300 deny ip from 127.0.0.0/8 to any
00400 deny ip from any to ::1
00500 deny ip from ::1 to any
00600 allow ipv6-icmp from :: to ff02::/16
00700 allow ipv6-icmp from fe80::/10 to fe80::/10
00800 allow ipv6-icmp from fe80::/10 to ff02::/16
00900 allow ipv6-icmp from any to any icmp6types 1
00900 skipto 10000 udp from any to any out via vtnet0 keep-state :default
00910 skipto 10000 tcp from any to any out via vtnet0 keep-state :default
01000 allow ipv6-icmp from any to any icmp6types 2,135,136
02000 allow tcp from any to any established
05200 allow tcp from any to any 80 keep-state :default
05300 allow tcp from any to any 443 keep-state :default
06100 allow tcp from any to me 22 in keep-state :default
06200 allow tcp from me to any 25 out keep-state :default
06300 allow icmp from any to any via vtnet0 out keep-state :default
09999 deny ip from any to any
10000 nat 1 ip from any to any via vtnet0 out
10001 allow ip from any to any
65535 deny ip from any to any

With this configuration I can't bootstrap pkg database:

Code: 
root@freebsd13:~# jls
   JID  IP Address      Hostname                      Path
     1  10.0.0.2        jail1.local                   /jails/jail1
     2  10.0.0.3        jail2.local                   /jails/jail2
root@freebsd13:~# jexec 1
root@jail1:/ # cd
root@jail1:~ # pkg
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:13:amd64/quarterly, please wait...
pkg: Error fetching http://pkg.FreeBSD.org/FreeBSD:13:amd64/quarterly/Latest/pkg.txz: No address record
Address resolution failed for http://pkg.FreeBSD.org/FreeBSD:13:amd64/quarterly.
Consider changing PACKAGESITE.
root@jail1:~ #

Some additional info about routing table inside the jail:

Code: 
root@jail1:~ # netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
10.0.0.2           link#3             UH          lo1

I think the problem are IPFW rules, or the routing settings inside jail.

I would like to uderstand the problem than keep up further tinerking for days and learning by mistakes or erros in configuration.

Thank you in advance!
 
SirDice said:
"No address record" means a failure in DNS settings. Is /etc/resolv.conf configured correctly in the jail?
Click to expand...
I forgot about that resolv.conf needs also to be configured inside the jail itself. My mistake.
Now pkg runs without a problem.

Thank you!

I still have doubts about IPFW rules that I use. Should they look this way? Should I use skipto for jails rules? Any additional recommendations and insights?
 
You must log in or register to reply here.
Back
Top