No routing between jails in different private subnets

amiramix · May 18, 2016

When both jails are in the same subnet 192.168.1.0/24 everything works as expected, however when one jail is in a different private subnet 10.33.1.0/16 it doesn't see the jail in the other subnet. I can ping both jails from the host. What could be the reason? Some data.

Working jail:

Code:

root@app2:/ # netstat -rn
Routing tables

Internet:
Destination  Gateway  Flags  Netif Expire
192.168.1.76  link#4  UHS  lo0

Code:

root@app2:/ # ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
  ether 00:25:90:ae:e8:bc
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
  ether 00:25:90:ae:e8:bc
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
  options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
  ether 00:25:90:ae:e8:bc
  inet 192.168.1.76 netmask 0xffffffff broadcast 192.168.1.76
  media: Ethernet autoselect
  status: active
  laggproto lacp lagghash l2,l3,l4
  laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
  laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

Not working jail:

Code:

root@pjp1:/ # netstat -rn
Routing tables

Internet:
Destination  Gateway  Flags  Netif Expire
10.33.1.40  link#4  UHS  lo0

Code:

root@pjp1:/ # ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
  ether 00:25:90:ae:e8:bc
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
  ether 00:25:90:ae:e8:bc
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
  options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
  ether 00:25:90:ae:e8:bc
  inet 10.33.1.40 netmask 0xffffffff broadcast 10.33.1.40
  media: Ethernet autoselect
  status: active
  laggproto lacp lagghash l2,l3,l4
  laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
  laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

On the host:

Code:

root@somehost:~ # netstat -rn
Routing tables

Internet:
Destination  Gateway  Flags  Netif Expire
default  192.168.1.1  UGS  lagg0
10.33.1.40  link#4  UHS  lo0
10.33.1.40/32  link#4  U  lagg0
127.0.0.1  link#3  UH  lo0
192.168.1.0/24  link#4  U  lagg0
192.168.1.10  link#4  UHS  lo0
192.168.1.76  link#4  UHS  lo0   
192.168.1.76/32  link#4  U  lagg0   
(... other jails)
   
Internet6:   
Destination  Gateway  Flags  Netif Expire   
::/96  ::1  UGRS  lo0   
::1  link#3  UH  lo0   
::ffff:0.0.0.0/96  ::1  UGRS  lo0   
fe80::/10  ::1  UGRS  lo0   
fe80::%lo0/64  link#3  U  lo0   
fe80::1%lo0  link#3  UHS  lo0   
ff01::%lo0/32  ::1  U  lo0   
ff02::/16  ::1  UGRS  lo0   
ff02::%lo0/32  ::1  U  lo0

Code:

root@somehost:~ # ifconfig   
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500   
  options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>   
  ether 00:25:90:ae:e8:bc   
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>   
  media: Ethernet autoselect (1000baseT <full-duplex>)   
  status: active   
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500   
  options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>   
  ether 00:25:90:ae:e8:bc
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
  options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
  inet6 ::1 prefixlen 128
  inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
  inet 127.0.0.1 netmask 0xff000000
  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
  ether 00:25:90:ae:e8:bc
  inet 192.168.1.10 netmask 0xffffff00 broadcast 192.168.1.255
  inet 192.168.1.76 netmask 0xffffffff broadcast 192.168.1.76 
  (... other jails)
  inet 10.33.1.40 netmask 0xffffffff broadcast 10.33.1.40
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect
  status: active
  laggproto lacp lagghash l2,l3,l4
  laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
  laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

SirDice · May 19, 2016

Did you enable routing? Add to /etc/rc.conf:

Code:

gateway_enable="YES"

amiramix · May 19, 2016

No. The documentation says it's to forward packets between interfaces, and all jails have their IPs aliased to the same interface. And so far routing between jails in the same subnet works without issues. Do you think that would help? Is there any other magic knob that I should enable in rc.conf?

SirDice · May 19, 2016

amiramix said:
The documentation says it's to forward packets between interfaces,

No, it's to forward packets between networks.

And so far routing between jails in the same subnet works without issues.

There is no routing involved in that case.

amiramix · May 19, 2016

I understand that would need to be in the host's rc.conf, not jail's, or both?

SirDice · May 19, 2016

Only on the host's. A jail can't modify these settings.

amiramix · May 19, 2016

No, it didn't help. Should I restart the host?

SirDice · May 19, 2016

Simply adding that line to /etc/rc.conf won't make it active.

amiramix · May 19, 2016

Well, it makes it active for services. I thought that parameters that are active only after the restart are in /boot/loader.conf. But anyway, restarting the host didn't help either. This is my rc.conf so far:

Code:

root@somehost:~ # cat /etc/rc.conf
hostname="somehost.somedomain.com"
keymap="pl_PL.ISO8859-2.kbd"
ifconfig_em0="up"
ifconfig_em1="up"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport em0 laggport em1 192.168.1.10/24"
defaultrouter="192.168.1.1"
syslogd_flags="-s -s"
sshd_enable="YES"
ntpd_enable="YES"
devfs_load_rulesets="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
jail_enable="YES"
rpcbind_enable="YES"
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 4"
nfsv4_server_enable="YES"
nfsuserd_enable="YES"
mountd_flags="-r"
gateway_enable="YES"

The interesting thing is that I can ping the new jail from another host using the IP 10.33.1.40, so there is routing in the network, it's just the routing between 192.168.1.0 and 10.33.0.0 jails that's broken.

BTW I just observed another interesting thing. The ping is much longer to the 10.33.1.40 jail than to 192.168.1.0 jails:

Code:

g@crayon2:~ % ping 192.168.1.50   
PING 192.168.1.50 (192.168.1.50): 56 data bytes   
64 bytes from 192.168.1.50: icmp_seq=0 ttl=64 time=0.085 ms   
64 bytes from 192.168.1.50: icmp_seq=1 ttl=64 time=0.096 ms   
64 bytes from 192.168.1.50: icmp_seq=2 ttl=64 time=0.088 ms   
64 bytes from 192.168.1.50: icmp_seq=3 ttl=64 time=0.075 ms   
64 bytes from 192.168.1.50: icmp_seq=4 ttl=64 time=0.092 ms

Code:

g@crayon2:~ % ping 192.168.1.60
PING 192.168.1.60 (192.168.1.60): 56 data bytes   
64 bytes from 192.168.1.60: icmp_seq=0 ttl=64 time=0.094 ms   
64 bytes from 192.168.1.60: icmp_seq=1 ttl=64 time=0.124 ms   
64 bytes from 192.168.1.60: icmp_seq=2 ttl=64 time=0.093 ms   
64 bytes from 192.168.1.60: icmp_seq=3 ttl=64 time=0.740 ms   
64 bytes from 192.168.1.60: icmp_seq=4 ttl=64 time=0.096 ms

Code:

g@crayon2:~ % ping 10.33.1.40
PING 10.33.1.40 (10.33.1.40): 56 data bytes   
64 bytes from 10.33.1.40: icmp_seq=0 ttl=63 time=0.615 ms   
64 bytes from 10.33.1.40: icmp_seq=1 ttl=63 time=0.635 ms   
64 bytes from 10.33.1.40: icmp_seq=2 ttl=63 time=0.587 ms   
64 bytes from 10.33.1.40: icmp_seq=3 ttl=63 time=0.602 ms   
64 bytes from 10.33.1.40: icmp_seq=4 ttl=63 time=0.615 ms
64 bytes from 10.33.1.40: icmp_seq=5 ttl=63 time=0.630 ms
64 bytes from 10.33.1.40: icmp_seq=6 ttl=63 time=0.615 ms

dave · May 23, 2016

You would need to add static routes to different subnets, no?

amiramix · May 23, 2016

Why? The ping reaches the destination albeit with a delay. I guess if a static route was needed then it wouldn't be reach the destination at all? And the main problem is with routing between jails in different subnets but running on the same host and aliased to the same network interface. Wouldn't the kernel know the routing between those jails without me setting up a static route explicitly?

SirDice · May 23, 2016

Correct, the static routes are already there. They're implicitly set because they're directly connected networks.

No routing between jails in different private subnets

amiramix

SirDice

Administrator

amiramix

SirDice

Administrator

amiramix

SirDice

Administrator

amiramix

SirDice

Administrator

amiramix

dave

amiramix

SirDice

Administrator