WireGuard peers can't reach each other on IPv6

Moviuro · Jun 30, 2020

Hi all,

Since 2018 I have a FreeBSD server acting as a WireGuard "server". This server has an IPv6 tunnel, courtesy of Hurricane Electric (https://tunnelbroker.net) and I have successfully distributed those IPv6 addresses to the VPN peers, so they now live in the future. Every single one of them when it visits e.g. https://icanhazip.com gets a correct IPv6 connection. From some other random IPv6 host, I can also access via IPv6 some sshd server on the VPN peers. However, those peers can't reach each other over IPv6 even though they can access IPv6 addresses that are given to jails (or to the host) on the server; and they have perfect IPv4 connectivity.

In short: addressing works, routing mostly works, firewalling is probably not a factor (can't see anything on pflog0), IPv4 works.

Code:

server# ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=81249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER>
        ether aa:aa:aa:aa:aa
        inet my.ipv4.add.ress netmask 0xffffff00 broadcast my.ipv4.add.255
        inet6 fe80::eea8:6bff:fef1:c8a3%em0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 10.10.10.1 netmask 0xffffff00
        inet 10.10.40.40 netmask 0xffffff00
        inet 10.10.10.2 netmask 0xffffff00
        inet 10.10.10.25 netmask 0xffffff00
        inet 10.10.10.5 netmask 0xffffff00
        inet6 fe80::1%lo1 prefixlen 64 scopeid 0x3
        inet6 my:ipv6:prefix:7765::1 prefixlen 64
        inet6 my:ipv6:prefix:7765::6d61:696c prefixlen 64
        inet6 my:ipv6:prefix:7765:7472:7970:6f70:686f prefixlen 64
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo2: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 10.10.20.1 netmask 0xffffff00
        inet6 fe80::1%lo2 prefixlen 64 scopeid 0x4
        inet6 my:ipv6:prefix:6261::1 prefixlen 64
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo3: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 10.10.30.1 netmask 0xffffff00
        inet 10.10.30.18 netmask 0xffffff00
        inet 10.10.30.4 netmask 0xffffff00
        inet 10.10.30.3 netmask 0xffffff00
        inet6 fe80::1%lo3 prefixlen 64 scopeid 0x5
        inet6 my:ipv6:prefix:6f74::1 prefixlen 64
        inet6 my:ipv6:prefix:6f74:7379:6e63:7468:696e prefixlen 64
        inet6 my:ipv6:prefix:6f74:0:7069:7261:7465 prefixlen 64
        inet6 my:ipv6:prefix:6f74::7a:6e63 prefixlen 64
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo4: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 10.10.40.1 netmask 0xffffff00
        inet 10.10.40.53 netmask 0xffffff00
        inet6 fe80::1%lo4 prefixlen 64 scopeid 0x6
        inet6 my:ipv6:prefix:444d::1 prefixlen 64
        inet6 my:ipv6:prefix:444d::ffff prefixlen 64
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280
        options=80000<LINKSTATE>
        tunnel inet my.ipv4.add.ress --> 216.66.84.46
        inet6 w:h:a:t:e:v:e:r::2 --> w:h:a:t:e:v:e:r::1 prefixlen 128
        inet6 fe80::eea8:6bff:fef1:c8a3%gif0 prefixlen 64 scopeid 0x7
        groups: gif
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
        groups: pflog
wg0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1420
        options=80000<LINKSTATE>
        inet 10.5.0.1 --> 10.5.0.1 netmask 0xffffff00
        inet6 my:ipv6:prefix:7670::1 prefixlen 64
        groups: tun
        nd6 options=101<PERFORMNUD,NO_DAD>
        Opened by PID 42057

Bash:

server# route -6 peer1.fqdn
   route to: peer1.fqdn
destination: my:ipv6:prefix:7670::
       mask: ffff:ffff:ffff:ffff::
        fib: 0
  interface: wg0
      flags: <UP,DONE>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1420         1         0
server# route -6 get peer2.fqdn 
   route to: my:ipv6:prefix:7670:746f:786f
destination: my:ipv6:prefix:7670::
       mask: ffff:ffff:ffff:ffff::
        fib: 0
  interface: wg0
      flags: <UP,DONE>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1420         1         0

Bash:

server # ping6 -c1 peer1.fqdn
PING6(56=40+8+8 bytes) my:ipv6:prefix:7670::1 --> my:ipv6:prefix:7670:16:19ff:fe25:308
16 bytes from my:ipv6:prefix:7670:16:19ff:fe25:308, icmp_seq=0 hlim=64 time=35.798 ms

--- peer1.fqdn ping6 statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 35.798/35.798/35.798/0.000 ms
server # ping6 -c1 peer2.fqdn
PING6(56=40+8+8 bytes) my:ipv6:prefix:7670::1 --> my:ipv6:prefix:7670::746f:786f
16 bytes from my:ipv6:prefix:7670::746f:786f, icmp_seq=0 hlim=64 time=10.885 ms

--- peer2.fqdn ping6 statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 10.885/10.885/10.885/0.000 ms

peer1# ping -6 -c1 my:ipv6:prefix:7670::1
PING my:ipv6:prefix:7670::1(my:ipv6:prefix:7670::1) 56 data bytes
64 bytes from my:ipv6:prefix:7670::1: icmp_seq=1 ttl=64 time=19.1 ms

--- my:ipv6:prefix:7670::1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 19.052/19.052/19.052/0.000 ms

peer2# ping -6 -c1 my:ipv6:prefix:7670::1
PING my:ipv6:prefix:7670::1(my:ipv6:prefix:7670::1) 56 data bytes
64 bytes from my:ipv6:prefix:7670::1: icmp_seq=1 ttl=64 time=11.0 ms

--- my:ipv6:prefix:7670::1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 10.987/10.987/10.987/0.000 ms

peer2# ping -c4 peer1.fqdn
PING peer1.fqdn(peer1.fqdn (my:ipv6:prefix:7670:16:19ff:fe25:308)) 56 data bytes
From my:ipv6:prefix:7670::1 (my:ipv6:prefix:7670::1) icmp_seq=1 Destination unreachable: Address unreachable
From my:ipv6:prefix:7670::1 (my:ipv6:prefix:7670::1) icmp_seq=2 Destination unreachable: Address unreachable
From my:ipv6:prefix:7670::1 (my:ipv6:prefix:7670::1) icmp_seq=3 Destination unreachable: Address unreachable
From my:ipv6:prefix:7670::1 (my:ipv6:prefix:7670::1) icmp_seq=4 Destination unreachable: Address unreachable

--- peer1.fqdn ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3003ms

peer1# ping -c4 10.5.0.35
PING 10.5.0.35 (10.5.0.35) 56(84) bytes of data.
From 10.5.0.1 icmp_seq=1 Redirect Host(New nexthop: 35.0.5.10)
64 bytes from 10.5.0.35: icmp_seq=1 ttl=63 time=31.0 ms
From 10.5.0.1 icmp_seq=2 Redirect Host(New nexthop: 35.0.5.10)
64 bytes from 10.5.0.35: icmp_seq=2 ttl=63 time=27.4 ms

--- 10.5.0.35 ping statistics ---
2 packets transmitted, 2 received, +2 errors, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 27.406/29.187/30.969/1.781 ms

While peer2# ping peer1 fails:

Bash:

server# tcpdump -tttnei wg0 icmp6 and host my:ipv6:prefix:7670::746f:786f and host my:ipv6:prefix:7670:16:19ff:fe25:308
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg0, link-type NULL (BSD loopback), capture size 262144 bytes
 00:00:00.000000 AF IPv6 (28), length 108: my:ipv6:prefix:7670::746f:786f > my:ipv6:prefix:7670:16:19ff:fe25:308: ICMP6, echo request, seq 1, length 64
 00:00:01.002030 AF IPv6 (28), length 108: my:ipv6:prefix:7670::746f:786f > my:ipv6:prefix:7670:16:19ff:fe25:308: ICMP6, echo request, seq 2, length 64
 00:00:01.001903 AF IPv6 (28), length 108: my:ipv6:prefix:7670::746f:786f > my:ipv6:prefix:7670:16:19ff:fe25:308: ICMP6, echo request, seq 3, length 64
 00:00:01.002068 AF IPv6 (28), length 108: my:ipv6:prefix:7670::746f:786f > my:ipv6:prefix:7670:16:19ff:fe25:308: ICMP6, echo request, seq 4, length 64
 
 peer2# tcpdump -tttnei wg0 icmp6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg0, link-type RAW (Raw IP), capture size 262144 bytes
 00:00:00.000000 ip: my:ipv6:prefix:7670::746f:786f > my:ipv6:prefix:7670:16:19ff:fe25:308: ICMP6, echo request, seq 1, length 64
 00:00:00.011833 ip: my:ipv6:prefix:7670::1 > my:ipv6:prefix:7670::746f:786f: ICMP6, destination unreachable, unreachable address my:ipv6:prefix:7670:16:19ff:fe25:308, length 112
 00:00:00.990138 ip: my:ipv6:prefix:7670::746f:786f > my:ipv6:prefix:7670:16:19ff:fe25:308: ICMP6, echo request, seq 2, length 64
 00:00:00.012115 ip: my:ipv6:prefix:7670::1 > my:ipv6:prefix:7670::746f:786f: ICMP6, destination unreachable, unreachable address my:ipv6:prefix:7670:16:19ff:fe25:308, length 112
 00:00:00.989201 ip: my:ipv6:prefix:7670::746f:786f > my:ipv6:prefix:7670:16:19ff:fe25:308: ICMP6, echo request, seq 3, length 64
 00:00:00.011541 ip: my:ipv6:prefix:7670::1 > my:ipv6:prefix:7670::746f:786f: ICMP6, destination unreachable, unreachable address my:ipv6:prefix:7670:16:19ff:fe25:308, length 112
 00:00:00.990160 ip: my:ipv6:prefix:7670::746f:786f > my:ipv6:prefix:7670:16:19ff:fe25:308: ICMP6, echo request, seq 4, length 64
 00:00:00.011830 ip: my:ipv6:prefix:7670::1 > my:ipv6:prefix:7670::746f:786f: ICMP6, destination unreachable, unreachable address my:ipv6:prefix:7670:16:19ff:fe25:308, length 112
 
 peer1# tcpdump -tttnei wg0 icmp6
 # empty

Mjölnir · Jun 30, 2020

Do you have ipv6_gateway_enable=yes in rc.conf(5)?

Moviuro · Jun 30, 2020

mjollnir said:
Do you have ipv6_gateway_enable=yes in rc.conf(5)?

Obviously yes, because else my peers wouldn't have otherwise perfectly functional IPv6 connectivity.

xtaz · Jul 2, 2020

I've spoken to you on IRC about this. The issue is fixed by setting the following sysctls:

Code:

net.inet.ip.redirect=0
net.inet6.ip6.redirect=0

What happens is if you have wireguard peers A-B-C where B is acting as a server and A tries to contact C, B gets confused somewhere and sends back an ICMP redirect message to say C can not be reached through B, please send your packet to B instead. And it just gets in a loop.

Setting those sysctls stops this behavior and it works, as you have confirmed on IRC.

The question is why? Is this a bug in the tun interface or kernel or something else? It appears that this block of code is to blame: https://github.com/freebsd/freebsd/blob/master/sys/netinet6/ip6_forward.c#L264

xtaz · Jul 2, 2020

Related PR now after discussions: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247718

WireGuard peers can't reach each other on IPv6

Moviuro

Mjölnir

Moviuro

xtaz

xtaz