Solved NATted outbound jail traffic sometimes uses backup CARP IP on LACP/LAGG interface

dch

dch

Developer
Outgoing traffic from a jail via PF NAT over a LAGG/LACP sometimes has the backup CARP IP address assigned to it. As this IP is only active on the "other" server, the return TCP connection traffic never actually gets back to our CARP master, and the other server sees spurious TCP connections.This is very reproducible and appears to be deterministic, like a round robin IP allocation.

Hopefully this is some misconfiguration on my part - what am I doing wrong?

FreeBSD 11.0R p10 amd64

rc.conf settings:

Code: 
# /etc/rc.conf network
ifconfig_igb0="up"
ifconfig_igb1="up"
cloned_interfaces="${cloned_interfaces} lagg0"

defaultrouter="1.2.3.81"
ipv6_defaultrouter="1:2:3:4::1"

ifconfig_lagg0="inet 1.2.3.83/28 laggproto lacp laggport igb0 laggport igb1"
ifconfig_lagg0_ipv6="inet6 1:2:3:4::83/64"

# carp on
kld_list="${kld_list} carp"
ifconfig_lagg0_aliases="\
  inet  vhid 1 advskew 100 pass pw1 1.2.3.84/32 \
  inet6 vhid 2 advskew 100 pass pw2 1:2:3:4::84/64 \
  inet  vhid 3 advskew   0 pass pw3 1.2.3.85/32 \
  inet6 vhid 4 advskew   0 pass pw4 1:2:3:4::85/64 \
"

# jail networks use their own separate cloned if
cloned_interfaces="${cloned_interfaces} lo1"
ifconfig_lo1_aliases="inet 10.241.0.0-15/16"

pf.conf settings are minimal as I'd thought I was blocking some connections:

Code: 
# /etc/pf.conf
# macros
protocols = "{ tcp, udp, icmp }"
tcp_services    = "{ domain, http, https, smtp" }
udp_services    = "{ domain }"

# interfaces
extl_if="lagg0"
intl_if="lo0"
jail_if="lo1"

# networks
intl_net = $intl_if:network
jail_net = $jail_if:network
internet = $extl_if:network

# limits
# bigger state tables help erlang receive sockets faster
# https://blog.tyk.nu/blog/fun-with-freebsd-listen-queue-overflow/
set limit { states 200000, frags 80000, src-nodes 80000 }
set timeout { adaptive.start 180000, adaptive.end 200000 }

# clean packets are happy packets
scrub in all

# jails are allowed outbound connections but not inbound
nat on $extl_if proto $protocols from   $jail_net to any -> ($extl_if)

# o ye of little faith
pass in all
pass out all

Active ifconfig settings:

Code: 
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
 options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
   ether 78:45:c4:fa:d2:99
   inet 1.2.3.82 netmask 0xfffffff0 broadcast 1.2.3.95
*   inet 1.2.3.84 netmask 0xffffffff broadcast 1.2.3.84 vhid 1
*   inet 1.2.3.85 netmask 0xffffffff broadcast 1.2.3.85 vhid 3 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   inet6 fe80::7a45:c4ff:fefa:d299%lagg0 prefixlen 64 scopeid 0x4
   inet6 1:2:3:4::82 prefixlen 64
*   inet6 1:2:3:4::84 prefixlen 64 vhid 2
*   inet6 1:2:3:4::85 prefixlen 64 vhid 4
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
   media: Ethernet autoselect
   status: active
*   carp: MASTER vhid 1 advbase 1 advskew 0
*   carp: BACKUP vhid 3 advbase 1 advskew 100 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
*   carp: MASTER vhid 2 advbase 1 advskew 0
*   carp: BACKUP vhid 4 advbase 1 advskew 100
   groups: lagg
   laggproto lacp lagghash l2,l3,l4
*   laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
*   laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

I've had to remove the IP aliases marked with !!!!!!! so that outbound traffic actually makes it back.

The resulting active pf.conf shows indeed it's round robin:

Code: 
#  pfctl -vnf /etc/pf.conf
protocols = "{ tcp, udp, icmp }"
tcp_services = "{ domain, http, https, smtp }"
udp_services = "{ domain }"
extl_if = "lagg0"
intl_if = "lo0"
jail_if = "lo1"
intl_net = "lo0:network"
jail_net = "lo1:network"
internet = "lagg0:network"
set limit states 200000
set limit frags 80000
set limit src-nodes 80000
set timeout adaptive.start 180000
set timeout adaptive.end 200000
scrub in all fragment reassemble
nat on lagg0 inet proto tcp from 10.241.0.0/16 to any -> (lagg0) round-robin
nat on lagg0 inet proto tcp from 10.241.0.1 to any -> (lagg0) round-robin
nat on lagg0 inet proto tcp from 10.241.0.2 to any -> (lagg0) round-robin
nat on lagg0 inet proto tcp from 10.241.0.3 to any -> (lagg0) round-robin
nat on lagg0 inet proto tcp from 10.241.0.6 to any -> (lagg0) round-robin
nat on lagg0 inet proto tcp from 10.241.0.7 to any -> (lagg0) round-robin
nat on lagg0 inet proto tcp from 10.241.0.8 to any -> (lagg0) round-robin
nat on lagg0 inet proto tcp from 10.241.0.9 to any -> (lagg0) round-robin
nat on lagg0 inet proto tcp from 10.241.0.12 to any -> (lagg0) round-robin
nat on lagg0 inet proto tcp from 10.241.0.15 to any -> (lagg0) round-robin
nat on lagg0 inet proto tcp from 10.241.0.13 to any -> (lagg0) round-robin
nat on lagg0 inet proto tcp from 10.241.0.10 to any -> (lagg0) round-robin
nat on lagg0 inet proto tcp from 10.241.0.14 to any -> (lagg0) round-robin
nat on lagg0 inet proto tcp from 10.241.0.4 to any -> (lagg0) round-robin
nat on lagg0 inet proto tcp from 10.241.0.11 to any -> (lagg0) round-robin
nat on lagg0 inet proto tcp from 10.241.0.5 to any -> (lagg0) round-robin
nat on lagg0 inet proto udp from 10.241.0.0/16 to any -> (lagg0) round-robin
nat on lagg0 inet proto udp from 10.241.0.1 to any -> (lagg0) round-robin
nat on lagg0 inet proto udp from 10.241.0.2 to any -> (lagg0) round-robin
nat on lagg0 inet proto udp from 10.241.0.3 to any -> (lagg0) round-robin
nat on lagg0 inet proto udp from 10.241.0.6 to any -> (lagg0) round-robin
nat on lagg0 inet proto udp from 10.241.0.7 to any -> (lagg0) round-robin
nat on lagg0 inet proto udp from 10.241.0.8 to any -> (lagg0) round-robin
nat on lagg0 inet proto udp from 10.241.0.9 to any -> (lagg0) round-robin
nat on lagg0 inet proto udp from 10.241.0.12 to any -> (lagg0) round-robin
nat on lagg0 inet proto udp from 10.241.0.15 to any -> (lagg0) round-robin
nat on lagg0 inet proto udp from 10.241.0.13 to any -> (lagg0) round-robin
nat on lagg0 inet proto udp from 10.241.0.10 to any -> (lagg0) round-robin
nat on lagg0 inet proto udp from 10.241.0.14 to any -> (lagg0) round-robin
nat on lagg0 inet proto udp from 10.241.0.4 to any -> (lagg0) round-robin
nat on lagg0 inet proto udp from 10.241.0.11 to any -> (lagg0) round-robin
nat on lagg0 inet proto udp from 10.241.0.5 to any -> (lagg0) round-robin
nat on lagg0 inet proto icmp from 10.241.0.0/16 to any -> (lagg0) round-robin
nat on lagg0 inet proto icmp from 10.241.0.1 to any -> (lagg0) round-robin
nat on lagg0 inet proto icmp from 10.241.0.2 to any -> (lagg0) round-robin
nat on lagg0 inet proto icmp from 10.241.0.3 to any -> (lagg0) round-robin
nat on lagg0 inet proto icmp from 10.241.0.6 to any -> (lagg0) round-robin
nat on lagg0 inet proto icmp from 10.241.0.7 to any -> (lagg0) round-robin
nat on lagg0 inet proto icmp from 10.241.0.8 to any -> (lagg0) round-robin
nat on lagg0 inet proto icmp from 10.241.0.9 to any -> (lagg0) round-robin
nat on lagg0 inet proto icmp from 10.241.0.12 to any -> (lagg0) round-robin
nat on lagg0 inet proto icmp from 10.241.0.15 to any -> (lagg0) round-robin
nat on lagg0 inet proto icmp from 10.241.0.13 to any -> (lagg0) round-robin
nat on lagg0 inet proto icmp from 10.241.0.10 to any -> (lagg0) round-robin
nat on lagg0 inet proto icmp from 10.241.0.14 to any -> (lagg0) round-robin
nat on lagg0 inet proto icmp from 10.241.0.4 to any -> (lagg0) round-robin
nat on lagg0 inet proto icmp from 10.241.0.11 to any -> (lagg0) round-robin
nat on lagg0 inet proto icmp from 10.241.0.5 to any -> (lagg0) round-robin
pass in all flags S/SA keep state
pass out all flags S/SA keep state
 
You must log in or register to reply here.
Back
Top