ipv6 not working in vnet jail on ipv6 only vps

N

nantex

Howdy,

I am still having problems getting ipv6 with a vnet jail working.

This is the situation: I am using a Hetzner VPS, I installed FreeBSD 14.1. The VPS is an ipv6-only instance. Hetzner does not support dynamic ipv6 assignment, one has to manually configure ipv6 addresses.

The host is reachable by v6, it is also able to reach v6 destinations on the internet.

The problem is, that the vnet jail is unreachable, whatever I do.

I tried several setups:

1) Pretty standard: bridge0 has vtnet0 (external interface) and the epair as member, no ip address configured. The jail's epair has an ipv6 /64 address of the same prefix as the host configured. When the jail comes up (bult with bastille) it has no default route. Whatever I try to add, doesn't work and is unreachable. I tried the ip of the host, the link local ip of the bridge etc - to no avail.

Host interfaces:

Code: 
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
        ether 96:00:03:d9:54:0b
        inet 100.65.251.74 netmask 0xffffffff broadcast 100.65.251.74
        inet6 2a01:4f8:c013:6513::1 prefixlen 64
        inet6 fe80::9400:3ff:fed9:540b%vtnet0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=0
        ether 58:9c:fc:10:de:49
        inet6 fe80::5a9c:fcff:fe10:de49%bridge0 prefixlen 64 scopeid 0x3
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: e0a_pubnix flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000
        groups: bridge
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
e0a_pubnix: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:e9:c0:e1:3c:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Host routes:

Code: 
Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             link#2                        URS         lo0
default                           fe80::1%vtnet0                UGS      vtnet0
::1                               link#2                        UHS         lo0
::ffff:0.0.0.0/96                 link#2                        URS         lo0
2a01:4f8:c013:6513::/64           link#1                        U        vtnet0
2a01:4f8:c013:6513::1             link#2                        UHS         lo0
fe80::%lo0/10                     link#2                        URS         lo0
fe80::%vtnet0/64                  link#1                        U        vtnet0
fe80::9400:3ff:fed9:540b%lo0      link#2                        UHS         lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0
fe80::%bridge0/64                 link#3                        U       bridge0
fe80::5a9c:fcff:fe10:de49%lo0     link#2                        UHS         lo0
ff02::/16                         link#2                        URS         lo0

Jail interface:

Code: 
vnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:e9:c0:e1:3c:0b
        inet6 2a01:4f8:c013:6513::2 prefixlen 64
        inet6 fe80::e9:c0ff:fee1:3c0b%vnet0 prefixlen 64 scopeid 0x6
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Jail routes:

Code: 
Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             link#7                        URS         lo0
::1                               link#7                        UHS         lo0
::ffff:0.0.0.0/96                 link#7                        URS         lo0
2a01:4f8:c013:6513::/64           link#6                        U         vnet0
2a01:4f8:c013:6513::2             link#7                        UHS         lo0
fe80::%lo0/10                     link#7                        URS         lo0
fe80::%vnet0/64                   link#6                        U         vnet0
fe80::e9:c0ff:fee1:3c0b%lo0       link#7                        UHS         lo0
fe80::%lo0/64                     link#7                        U           lo0
fe80::1%lo0                       link#7                        UHS         lo0
ff02::/16                         link#7                        URS         lo0

Ping from jail to host external ip:

Code: 
ping6 2a01:4f8:c013:6513::1
PING(56=40+8+8 bytes) 2a01:4f8:c013:6513::2 --> 2a01:4f8:c013:6513::1
^C
--- 2a01:4f8:c013:6513::1 ping statistics ---
6 packets transmitted, 0 packets received, 100.0% packet loss
root@pubnix:/ # ping6 -c1 2a01:4f8:c013:6513::1
PING(56=40+8+8 bytes) 2a01:4f8:c013:6513::2 --> 2a01:4f8:c013:6513::1

--- 2a01:4f8:c013:6513::1 ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss

Tcpdump during the ping on the bridge0 interface:

Code: 
10:52:35.573829 IP6 2a01:4f8:c013:6513::2 > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has 2a01:4f8:c013:6513::1, length 32
10:52:36.579107 IP6 2a01:4f8:c013:6513::2 > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has 2a01:4f8:c013:6513::1, length 32
10:52:37.582565 IP6 2a01:4f8:c013:6513::2 > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has 2a01:4f8:c013:6513::1, length 32

2) I also tried to set it up so, that the external interface does NOT have an inet6 config, but instead the bridge0 interface has it (like replace ifconfig_vtnet0_ipv6 with ifconfig_bridge0_ipv6). But in this setup the whole host is unreachable. So I didn't further investigate this one.


3) I also tried to give the host its address with a mask of /128, give the jail its address with the /64 mask, give the bridge an ip address with /64 and use this as default router in the jail. This works insofar as the host is reachable, but the jail is not.

Host interfaces:

Code: 
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
        ether 96:00:03:da:9e:0d
        inet6 2a01:4f8:c013:6513::1 prefixlen 128
        inet6 fe80::9400:3ff:feda:9e0d%vtnet0 prefixlen 64 scopeid 0x1
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=0
        ether 58:9c:fc:10:de:49
        inet6 fe80::5a9c:fcff:fe10:de49%bridge0 prefixlen 64 scopeid 0x3
        inet6 2a01:4f8:c013:6513::10 prefixlen 64
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: e0a_pubnix flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000
        groups: bridge
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
e0a_pubnix: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:e9:c0:e1:3c:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Jail interface:

Code: 
vnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:e9:c0:e1:3c:0b
        inet6 2a01:4f8:c013:6513::2 prefixlen 64
        inet6 fe80::e9:c0ff:fee1:3c0b%vnet0 prefixlen 64 scopeid 0x6
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Jail routes:

Code: 
Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             link#7                        URS         lo0
default                           2a01:4f8:c013:6513::10        UGS       vnet0
::1                               link#7                        UHS         lo0
::ffff:0.0.0.0/96                 link#7                        URS         lo0
2a01:4f8:c013:6513::/64           link#6                        U         vnet0
2a01:4f8:c013:6513::2             link#7                        UHS         lo0
fe80::%lo0/10                     link#7                        URS         lo0
fe80::%vnet0/64                   link#6                        U         vnet0
fe80::e9:c0ff:fee1:3c0b%lo0       link#7                        UHS         lo0
fe80::%lo0/64                     link#7                        U           lo0
fe80::1%lo0                       link#7                        UHS         lo0
ff02::/16                         link#7                        URS         lo0

Tcpdump output looks identical to the above setup.

So, I have no idea how to solve this. Has anyone an idea what might be wrong?


PS: one thing about the "v6 only part": as can be seen, the ext if has an ipv4 address, but it's no internet address, it is only being used to reach the hetzner cloud meta data service.
 
nantex said:
Pretty standard: bridge0 has vtnet0 (external interface) and the epair as member
Click to expand...
Not showing on any of the outputs you've posted, it only shows the epair interface as a member. Then bridge0 isn't connected to vtnet0 on the host, so it has no connection to the outside world.
 
k, I tried to fix it, now on the host the interfaces look like this:

Code: 
vtnet0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=c00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
        ether 96:00:03:dc:97:54                                                                                                                            
        inet6 2a01:4f8:c013:6513::1 prefixlen 64                                                                                                           
        inet6 fe80::9400:3ff:fedc:9754%vtnet0 prefixlen 64 scopeid 0x1                                                                                     
        media: Ethernet autoselect (10Gbase-T <full-duplex>)                                                                                               
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=0
        ether 58:9c:fc:10:de:49
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: e0a_pubnix flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000
        member: vtnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 2000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
e0a_pubnix: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:e9:c0:e1:3c:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

So, bridge0 has 2 members: the host interface and the jail's epair. Inside the jail I have:

Code: 
root@pubnix:/ # ifconfig vnet0                                                       
vnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8<VLAN_MTU>                                                          
        ether 02:e9:c0:e1:3c:0b                                                      
        inet6 2a01:4f8:c013:6513::2 prefixlen 64                                                                                                           
        inet6 fe80::e9:c0ff:fee1:3c0b%vnet0 prefixlen 64 scopeid 0x6                 
        groups: epair                                                                                                                                      
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)                          
        status: active                                                               
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>                                    
root@pubnix:/ # netstat -rnfinet6                                            
Routing tables

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             link#7                        URS         lo0
default                           fe80::1%vnet0                 UGS       vnet0
..

But the jail is still unreachable.
 
I also tried another option, which the bastille maintainer suggested: to have no ipv6 address on vtnet0 and instead put it on bridge0, but in this case I can't add fe80::1%vtnet0 as default route (invalid argument it says).
 
Ok, I followed this HOWTO, now I have this config:

Host:

Code: 
cloned_interfaces="bridge0"
create_args_bridge0="inet6 auto_linklocal -ifdisabled addm vtnet0"
ifconfig_bridge0="inet 10.55.0.143 netmask 255.255.255.0"
ifconfig_bridge0_ipv6="inet6 2a01:4f8:c013:6513::1 prefixlen 64 auto_linklocal"
ifconfig_vtnet0="up -tso -vlanhwtso DHCP"
ipv6_defaultrouter="fe80::1%bridge0"

Interfaces now:
Code: 
vtnet0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=800b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
        ether 96:00:03:dc:ff:78
        inet 100.65.251.74 netmask 0xffffffff broadcast 100.65.251.74
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=0
        ether 96:00:03:dc:ff:78
        inet 10.55.0.143 netmask 0xffffff00 broadcast 10.55.0.255
        inet6 fe80::9400:3ff:fedc:ff78%bridge0 prefixlen 64 scopeid 0x3
        inet6 2a01:4f8:c013:6513::1 prefixlen 64
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: epjd.h flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000
        member: vtnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 2000
        groups: bridge
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epjd.h: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:3b:79:d9:d1:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

The bridge0 and vtnet0 interfaces now share the mac addres. Inside the jail:

Code: 
epjd.j: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:3b:79:d9:d1:0b
        inet 10.1.1.2 netmask 0xffffff00 broadcast 10.1.1.255
        inet6 2a01:4f8:c013:6513::beef prefixlen 64
        inet6 fe80::3b:79ff:fed9:d10b%epjd.j prefixlen 64 scopeid 0x6
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

# default route:
Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             link#4                        URS         lo0
default                           2a01:4f8:c013:6513::1         UGS      epjd.j

Now I can reach the jail ipv6 from the host and vice versa. I can also reach the host from the internet. So, this is a good step forward.

However, I'm still unable to reach the jail from the outside. I'm not sure what to use as default gw inside the jail. I tried 2a01:4f8:c013:6513::1, which is reachable, that is, I can ping it from the jail. I also tried fe80::1%epjd.j, this didn't work as well.

When I tcpdump while pinging from the internet, I can see the packets on bridge0, but not inside the jail.

Code: 
tcpdump -pni bridge0 not port 22
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on bridge0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:42:25.365222 IP6 2003:d7:f4a:4a00:83c3:79e:1aee:9c71 > 2a01:4f8:c013:6513::beef: ICMP6, echo request, id 289, seq 1096, length 64
12:42:26.389786 IP6 2003:d7:f4a:4a00:83c3:79e:1aee:9c71 > 2a01:4f8:c013:6513::beef: ICMP6, echo request, id 289, seq 1097, length 64
12:42:27.414032 IP6 2003:d7:f4a:4a00:83c3:79e:1aee:9c71 > 2a01:4f8:c013:6513::beef: ICMP6, echo request, id 289, seq 1098, length 64
...

PS: regarding the changed interface names: I switched tools for debugging purposes and use jaildk for now. There's no rc.conf for the jail to show anymore, because the script does all the work.
 
You must log in or register to reply here.
Back
Top