FIB's with IPv6

mumu · Dec 6, 2024

I have a VPS that has a working IPv6 connection. Now I want to add Hurricane Electric(HE) to the mix, so I could have certain applications go through the VPS and and certain others through HE.

Code:

OS: FreeBSD 14.1-RELEASE
/boot/loader.conf:
net.fibs="5"
net.add_addr_allfibs="0"

For HE, here's what I've tried to do..

Code:

setfib 1 ifconfig gif0 create up
setfib 1 ifconfig gif0 tunnel 1.2.3.4 216.x.x.x mtu 1480
setfib 1 ifconfig gif0 inet6 2001:470:xx:xx::2 2001:470:xx:xx::1 prefixlen 128
setfib 1 route -6n add -host 2001:470:xx:xx::2 -iface gif0
setfib 1 route -6n add default 2001:470:xx:xx::1
add net default: gateway 2001:470:xx:xx::1 fib 1: Invalid argument <-- this error

Then I tried adding the default route this way:

Code:

setfib 1 route -6n add default -iface gif0 2001:470:xx:xx::1
add net default: gateway gif0 fib 1

However netstat -rn6F1 doesn't show the default route and obviously ping/ping6 fails with 'no route to host'.

So how can I use FIB's to add a second IPv6 gateway through HE ?

SirDice · Dec 6, 2024

With the HE tunnel I used to have I used a gateway like so: route -6 add default -iface gif0 The gateway doesn't really need an address, there's only one way out of the tunnel (the other tunnel end-point).

SirDice · Dec 6, 2024

Code:

setfib 1 route -6n add -host 2001:470:xx:xx::2 -iface gif0

Not needed. The 2001:470:xx:xx::2 address is set on the interface, this already creates an implicit route.

mumu · Dec 6, 2024

SirDice said:
With the HE tunnel I used to have I used a gateway like so: route -6 add default -iface gif0 The gateway doesn't really need an address, there's only one way out of the tunnel (the other tunnel end-point).

Ok, tried that, no errors, but ping still fails.

Code:

# setfib 1 netstat -rn6            
Routing tables (fib: 1)

Internet6:
Destination                  Gateway                    Flags     Netif Expire
::/96                             link#2                        URS         lo0
default                         link#6                        US           gif0
::1                                 link#2                        UHS        lo0
::ffff:0.0.0.0/96              link#2                        URS        lo0
fe80::%lo0/10               link#2                       URS         lo0
ff02::/16                        link#2                       URS         lo0

Code:

# setfib 1 ping6 2001:470:xx:xx::1
PING(56=40+8+8 bytes) 2001:470:xx:xx::2 --> 2001:470:xx:xx::1
ping6: sendmsg: No route to host
ping: wrote 2001:470:xx:xx::1 16 chars, ret=-1
ping6: sendmsg: No route to host
ping: wrote 2001:470:xx:xx::1 16 chars, ret=-1
^C
--- 2001:470:xx:xx::1 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss

SirDice · Dec 6, 2024

Right, I remember I was never able to ping the other end of the tunnel. Try something like setfib 1 ping -6 www.google.com

Also ping6(8) doesn't really exist anymore, both IPv4 and IPv6 have been rolled into one ping(8) command. /usr/sbin/ping6 still exists, but it's hardlinked to /usr/sbin/ping. If you need IPv4 or IPv6 specifically you can use ping -4 ... or ping -6 ...

mumu · Dec 6, 2024

SirDice said:
Right, I remember I was never able to ping the other end of the tunnel. Try something like setfib 1 ping -6 www.google.com

Also ping6(8) doesn't really exist anymore, both IPv4 and IPv6 have been rolled into one ping(8) command. /usr/sbin/ping6 still exists, but it's hardlinked to /usr/sbin/ping. If you need IPv4 or IPv6 specifically you can use ping -4 ... or ping -6 ...

I get 'ping: cannot resolve www.google.com: Name does not resolve' error.

I think this has to do with gif(4) being a virtual interface and the errors happening is probably due to gif(4) referencing the main physical interface (fib 0) which obviously won't work. Meaning FIB's only work with physical interfaces than virtual one's.

SirDice · Dec 6, 2024

mumu said:
I get 'ping: cannot resolve www.google.com: Name does not resolve' error.

Right, it does assume name resolving actually works (/etc/resolve.conf?). Pinging an IPv6 address might be better, ping 2001:4860:4860::8888 for example (Google's IPv6 equivalent of 8.8.8.8), at the very least an address somewhere beyond that tunnel endpoint address and your own assigned prefix. I suspect the tunnel endpoint isn't attached to anything, it's completely fictitious. You can't ping it, or use it as a gateway address. Also, the IPv6 tunnel addresses are different from the range(s) (you can get a /64 and/or a /48 prefix) you've been assigned.

mumu · Dec 7, 2024

SirDice said:
Right, it does assume name resolving actually works (/etc/resolve.conf?). Pinging an IPv6 address might be better, ping 2001:4860:4860::8888 for example (Google's IPv6 equivalent of 8.8.8.8), at the very least an address somewhere beyond that tunnel endpoint address and your own assigned prefix. I suspect the tunnel endpoint isn't attached to anything, it's completely fictitious. You can't ping it, or use it as a gateway address. Also, the IPv6 tunnel addresses are different from the range(s) (you can get a /64 and/or a /48 prefix) you've been assigned.

Tried that and still failed..

Code:

# setfib 1 ping -I gif0 2001:470:xx:xx::1
PING(56=40+8+8 bytes) 2001:470:xx:xx::2 --> 2001:470:xx:xx::1
ping: sendmsg: No route to host
ping: wrote 2001:470:xx:xx::1 16 chars, ret=-1
ping: sendmsg: No route to host
ping: wrote 2001:470:xx:xx::1 16 chars, ret=-1
ping: sendmsg: No route to host
ping: wrote 2001:470:xx:xx::1 16 chars, ret=-1
^C
--- 2001:470:xx:xx::1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
# setfib 1 ping -I gif0 2001:4860:4860::8888
PING(56=40+8+8 bytes) 2001:470:xx:xx::2 --> 2001:4860:4860::8888
ping: sendmsg: No route to host
ping: wrote 2001:4860:4860::8888 16 chars, ret=-1
ping: sendmsg: No route to host
ping: wrote 2001:4860:4860::8888 16 chars, ret=-1
^C
--- 2001:4860:4860::8888 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss

Code:

# setfib 1 netstat -rn6                    
Routing tables (fib: 1)

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             link#2                        URS         lo0
default                           link#6                        US         gif0
::1                               link#2                        UHS         lo0
::ffff:0.0.0.0/96                 link#2                        URS         lo0
fe80::%lo0/10                     link#2                        URS         lo0
ff02::/16                         link#2                        URS         lo0
#

mumu · Dec 7, 2024

Also adding this in for context..

Code:

# setfib 1 route -6n get 2001:4860:4860::8888
   route to: 2001:4860:4860::8888
destination: ::
       mask: ::
        fib: 1
  interface: gif0
      flags: <UP,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1480         1         0
#

The 'flags' is missing GATEWAY. Here's the same command on the main fib(0):

Code:

# route -6n get 2001:4860:4860::8888 
   route to: 2001:4860:4860::8888
destination: ::
       mask: ::
    gateway: 2001:dfx:yy:zz::1
        fib: 0
  interface: vtnet0
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0 
#

mumu · Dec 8, 2024

After further testing, I'm suspecting that FIB's don't work with virtual interfaces as it seems to somehow reference the physical interface (fib0) for outbound/inbound traffic which obviously wouldn't work.

IAmPaigeAT · 2025-04-19T23:59:28+0100

mumu said:
After further testing, I'm suspecting that FIB's don't work with virtual interfaces as it seems to somehow reference the physical interface (fib0) for outbound/inbound traffic which obviously wouldn't work.

Confirmed and I actually think I know what's going on but first let me demonstrate to make sure we're on the same page:

- Same FIB:

Bash:

➜  stelleri ifconfig epair127 create
epair127a
➜  stelleri ifconfig epair127a inet6 fcff:ffff:127::a/64
➜  stelleri ifconfig epair127b inet6 fcff:ffff:127::b/64
➜  stelleri ping fcff:ffff:127::b
PING(56=40+8+8 bytes) fcff:ffff:127::b --> fcff:ffff:127::b
16 bytes from fcff:ffff:127::b, icmp_seq=0 hlim=64 time=0.029 ms
16 bytes from fcff:ffff:127::b, icmp_seq=1 hlim=64 time=0.072 ms
16 bytes from fcff:ffff:127::b, icmp_seq=2 hlim=64 time=0.078 ms

- Destroy it, assign b to FIB 127 then assign addresses:

Bash:

➜  stelleri ifconfig epair127a destroy
➜  stelleri ifconfig epair127 create                   
epair127a
➜  stelleri ifconfig epair127b fib 127
➜  stelleri ifconfig epair127b inet6 fcff:ffff:127::b/64
➜  stelleri ifconfig epair127a inet6 fcff:ffff:127::a/64

When you assign a new address to b while its still assigned to fib 127 and tcpdump it at the same time you can see it trying to negotiate NDP:

Bash:

00:00:05.110781 02:a5:c4:9d:c5:0b > 33:33:ff:00:00:0c, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ff00:c: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fcff:ffff:127::c
          unknown option (14), length 8 (1):
            0x0000:  d007 2415 0ca6
        0x0000:  3333 ff00 000c 02a5 c49d c50b 86dd 6000  33............`.
        0x0010:  0000 0020 3aff 0000 0000 0000 0000 0000  ....:...........
        0x0020:  0000 0000 0000 ff02 0000 0000 0000 0000  ................
        0x0030:  0001 ff00 000c 8700 6d9d 0000 0000 fcff  ........m.......
        0x0040:  ffff 0127 0000 0000 0000 0000 000c 0e01  ...'............
        0x0050:  d007 2415 0ca6                           ..$...
 00:00:00.067474 02:a5:c4:9d:c5:0b > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 90: (hlim 1, next-header Options (0) payload length: 36) fe80::a5:c4ff:fe9d:c50b > ff02::16: HBH (padn)(rtalert: 0x0000)  [icmp6 sum ok] ICMP6, multicast listener report v2, 1 group record(s) [gaddr ff02::1:ff00:c to_ex { }]
        0x0000:  3333 0000 0016 02a5 c49d c50b 86dd 6000  33............`.
        0x0010:  0000 0024 0001 fe80 0000 0000 0000 00a5  ...$............
        0x0020:  c4ff fe9d c50b ff02 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0016 3a00 0100 0502 0000 8f00  ......:.........
        0x0040:  e7ae 0000 0001 0400 0000 ff02 0000 0000  ................
        0x0050:  0000 0000 0001 ff00 000c                 ..........
 00:00:02.316002 02:a5:c4:9d:c5:0b > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 90: (hlim 1, next-header Options (0) payload length: 36) fe80::a5:c4ff:fe9d:c50b > ff02::16: HBH (padn)(rtalert: 0x0000)  [icmp6 sum ok] ICMP6, multicast listener report v2, 1 group record(s) [gaddr ff02::1:ff00:c to_ex { }]
        0x0000:  3333 0000 0016 02a5 c49d c50b 86dd 6000  33............`.
        0x0010:  0000 0024 0001 fe80 0000 0000 0000 00a5  ...$............
        0x0020:  c4ff fe9d c50b ff02 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0016 3a00 0100 0502 0000 8f00  ......:.........
        0x0040:  e7ae 0000 0001 0400 0000 ff02 0000 0000  ................
        0x0050:  0000 0000 0001 ff00 000c                 ..........

But something you might notice that stands out to me when you look at ndp -a

Bash:

➜  stelleri diff same_fib diff_fib
17d16
< fcff:ffff:10::b                      02:5c:fd:97:c4:0b epair1a 23h49m3s  S R
24,27c23,24
< fcff:ffff:127::a                     02:cf:d0:88:d7:0a epair127a permanent R
< fe80::cf:d0ff:fe88:d70a%epair127a    02:cf:d0:88:d7:0a epair127a permanent R
< fe80::cf:d0ff:fe88:d70b%epair127b    02:cf:d0:88:d7:0b epair127b permanent R
< fcff:ffff:127::b                     02:cf:d0:88:d7:0b epair127b permanent R
---
> fcff:ffff:127::a                     02:4d:a5:29:85:0a epair127a permanent R
> fe80::4d:a5ff:fe29:850a%epair127a    02:4d:a5:29:85:0a epair127a permanent R

My suspicion is that NDP isn't working correctly, it might be possible to substitute a static NDP entry (would require a static MAC for the interface in addition to the entry in rc.conf, assuming it's possible to configure NDP from rc.conf..) I'll have to dig into this a little more but I'm fairly certain now it has something to do with NDP and I'll tell you why:

- The fact that this is the same host, you notice how ping wants to send `fcff:ffff:127::b --> fcff:ffff:127::b` note that specifying ping -S fcff:ffff:127::a fcff:ffff:127::b also does not work
- There's nothing in the man page for ndp that would lead me to believe that FIBs have ever been a consideration
- IPv4 works fine for this, you can even assign a /31 on a pair which has no broadcast

Code:

➜  stelleri ifconfig epair127a 172.16.0.0/31
➜  stelleri ifconfig epair127b 172.16.0.1/31 fib 127
➜  stelleri ping 172.16.0.1
PING 172.16.0.1 (172.16.0.1): 56 data bytes
64 bytes from 172.16.0.1: icmp_seq=0 ttl=64 time=0.028 ms
^C
--- 172.16.0.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.028/0.028/0.028/0.000 ms
➜  stelleri arp -a
....
? (172.16.0.0) at 02:4d:a5:29:85:0a on epair127a permanent [ethernet]
? (172.16.0.1) at 02:4d:a5:29:85:0b on epair127a expires in 1189 seconds [ethernet]
? (172.16.0.0) at 02:4d:a5:29:85:0a on epair127b expires in 1189 seconds [ethernet]
? (172.16.0.1) at 02:4d:a5:29:85:0b on epair127b permanent [ethernet]

When I assign 172.16.0.1 though to b I notice it just sends a arp for itself:

Code:

00:00:04.037441 02:4d:a5:29:85:0a > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len
 4), Request who-has 172.16.0.1 tell 172.16.0.0, length 28
0x0000:  ffff ffff ffff 024d a529 850a 0806 0001  .......M.)......
0x0010:  0800 0604 0001 024d a529 850a ac10 0000  .......M.)......
0x0020:  0000 0000 0000 ac10 0001                 ..........
 00:00:00.000005 02:4d:a5:29:85:0b > 02:4d:a5:29:85:0a, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len
4), Reply 172.16.0.1 is-at 02:4d:a5:29:85:0b, length 28
0x0000:  024d a529 850a 024d a529 850b 0806 0001  .M.)...M.)......
0x0010:  0800 0604 0002 024d a529 850b ac10 0001  .......M.)......
0x0020:  024d a529 850a ac10 0000                 .M.)......

So that's apparently working correctly and so I'm pretty firm on this has to have something with NDP

but it might need to be triaged a little more unless somebody knows exactly what is going on ....

* EDIT
I just realized the OP's interface (gif) is layer 3 encapsulation so I don't know how that works off hand (whether it still uses ndp or not) there's apparently something wrong in any case, also I sent an e-mail to freebsd-net to ask about this.

FIB's with IPv6

mumu

SirDice

Administrator

SirDice

Administrator

mumu

SirDice

Administrator

mumu

SirDice

Administrator

mumu

mumu

mumu

IAmPaigeAT