Solved Please advise with bhyve IPv6 configuration

Argentum said:
Thank you. I will try it later and use fixed V6 address on the main machine. As I wrote, this is just a test machine in my home network and I am trying to simulate the conditions in Hetzner. Do not want to go and fiddle there with the settings and eventually land on hard reset.
Click to expand...
Definitely, the conditions should be reproduced and properly tested!
 
Okay, lets understand how it all works...

When a machine boots up, it knows nothing about the network.
But, when an interface has auto_linklocal enabled,an IPv6 address is created automatically. These addresses are created from the MAC address, they start with fe80: and are linklocal to that network, cannot be routed onwards:
Code: 
# ifconfig build
build: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        inet6 fe80::438:92ff:fe08:ac43%build prefixlen 64 scopeid 0x4

Then, when accept_rtadv is enabled, a router advertisement may be received from upstream, or we may run rtsol to explicitely request one:
Code: 
# tcpdump -nibuild icmp6
17:21:05.910891 IP6 fe80::438:92ff:fe08:ac43 > ff02::2: ICMP6, router solicitation, length 16
17:21:06.123552 IP6 fe80::438:92ff:fe08:ac41 > ff02::1: ICMP6, router advertisement, length 88

That "router solicitation" request is sent from our linklocal address, and it is sent to a broadcast address ff02::2 where the routers listen. A router in charge will then answer and send the router advertisement - which contains our real routeable prefix.
The kernel can now configure that network onto the interface. And also the kernel does now know the IPv6 address of the router - because that is the IP from which the router advertisement came! So this can be configured as the default route. In the example that would be fe80::438:92ff:fe08:ac41%build - so we get:
Code: 
# netstat -rn6 | grep ^default
default                           fe80::438:92ff:fe08:ac41%build UG        build

And now we're up and connective.
The problem here is: that address we got for our router, is again only a linklocal address! This is good enough for a route, because we get to the next hop, and that's all we need to. But it is only good for ad-hoc automated configuration, because such an IP may change on occasion - it should not be used in rc.conf to configure a default-gateway persistently.

Therefore, my hoster (scaleway) tells me
  • your ipv6 prefix is 2001:bc8:xxxx:yyyy::/64
  • and your nexthop/gateway is always nr. 1, i.e. 2001:bc8:xxxx:yyyy::1
That means, I cannot use the nr. 1 myself, because thats already occupied, but I could configure a defaultroute as follows (without configuring the /64 network):
Code: 
route add -6 -host 2001:bc8:xxxx:yyyy::1 -iface bce0
route add -6 -net default 2001:bc8:xxxx:yyyy::1
 
PMc said:
Definitely, the conditions should be reproduced and properly tested!
Click to expand...
Hello!

Looks like I have made some progress, but still no score:
  1. Configured the main machine to the static address and an actual gateway - it works
  2. Performed all the steps described before
  3. The VM can ping the router now, and all the internal addresses, but cannot ping outside world.
Code: 
root@Argentum ~# netstat -rn6 | grep ^2001:7d0
2001:7d0:xxxx:xxxx::/64           link#1                        U          bge0
2001:7d0:xxxx:xxxx::1             link#2                        UHS         lo0
2001:7d0:xxxx:xxxx::100/120       link#3                        U      vm-inter
2001:7d0:xxxx:xxxx::101           link#3                        UHS         lo0
2001:7d0:xxxx:xxxx:xxxx:xxxx:xxxx:1785 link#1                    UHS         lo0

In the VM the traceroute6 ends at 2001:7d0:xxxx:yyyy::101 - the internal switch address.

Setting sysctl net.inet6.ip6.forwarding=1 did not kill the V6 network any more.

And there is still this error:

Code: 
root@Argentum ~# route add -6 -reject -net 2001:7d0:xxxx:yyyy::/64
add net 2001:7d0:xxxx:yyyy::/64 fib 0: route already in table
 
Argentum said:
Hello!

Looks like I have made some progress, but still no score:
  1. Configured the main machine to the static address and an actual gateway - it works
  2. Performed all the steps described before
  3. The VM can ping the router now, and all the internal addresses, but cannot ping outside world.
Code: 
root@Argentum ~# netstat -rn6 | grep ^2001:7d0
2001:7d0:xxxx:xxxx::/64           link#1                        U          bge0
2001:7d0:xxxx:xxxx::1             link#2                        UHS         lo0
2001:7d0:xxxx:xxxx::100/120       link#3                        U      vm-inter
2001:7d0:xxxx:xxxx::101           link#3                        UHS         lo0
2001:7d0:xxxx:xxxx:xxxx:xxxx:xxxx:1785 link#1                    UHS         lo0

In the VM the traceroute6 ends at 2001:7d0:xxxx:yyyy::101 - the internal switch address.

Setting sysctl net.inet6.ip6.forwarding=1 did not kill the V6 network any more.

And there is still this error:

Code: 
root@Argentum ~# route add -6 -reject -net 2001:7d0:xxxx:yyyy::/64
add net 2001:7d0:xxxx:yyyy::/64 fib 0: route already in table
Click to expand...
If you have static address and defaultroute, that /64 on bge0 should not be there anymore.
Next step would be to watch the ping with tcpdump -ni tap0 icmp6 and tcpdump -ni bce0 icmp6 and see how far they get.

Also, route -6 -nv get <dest-ip> gives very usefull information about where some packet is going.
 
PMc said:
If you have static address and defaultroute, that /64 on bge0 should not be there anymore.
Next step would be to watch the ping with tcpdump -ni tap0 icmp6 and tcpdump -ni bce0 icmp6 and see how far they get.

Also, route -6 -nv get <dest-ip> gives very usefull information about where some packet is going.
Click to expand...
Thank you for your feedback. I will debug this later, but as a quick response:

Code: 
root@Helium ~# route -6 -nv get google.com
RTA_DST: inet6 2a00:1450:4026:804::200e; RTA_IFP: link ; RTM_GET: Report Metrics: len 240, pid: 0, seq 1, errno 0, flags
:<UP,GATEWAY,HOST,STATIC>
locks:  inits:
sockaddrs: <DST,IFP>
 2a00:1450:4026:804::200e link#0
   route to: 2a00:1450:4026:804::200e
destination: ::
       mask: ::
    gateway: 2001:7d0:xxxx:xxxx::101
        fib: 0
  interface: vtnet0
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

locks:  inits:
sockaddrs: <DST,GATEWAY,NETMASK,IFP,IFA>
 :: 2001:7d0:xxxx:xxxx::101 :: vtnet0:58.9c.fc.e.34.cb 2001:7d0:xxxx:xxxx::102

Also, the main machine has static address, configured in the rc.conf

Code: 
ifconfig_bge0_ipv6="inet6  2001:7d0:xxxx:xxxx:xxxx:xxxx:xxxx:1785 prefixlen 64"
ipv6_defaultrouter="2001:7d0:xxxx:xxxx::1"

From main machine:

Code: 
root@Argentum ~# route -6 -nv get google.com
RTA_DST: inet6 2a00:1450:4026:804::200e; RTA_IFP: link ; RTM_GET: Report Metrics: len 240, pid: 0, seq 1, errno 0, flags:<UP,GATEWAY,HOST,STATIC>
locks:  inits:
sockaddrs: <DST,IFP>
 2a00:1450:4026:804::200e link#0
   route to: 2a00:1450:4026:804::200e
destination: ::
       mask: ::
    gateway: 2001:7d0:xxxx:xxxx::1
        fib: 0
  interface: bge0
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

locks:  inits:
sockaddrs: <DST,GATEWAY,NETMASK,IFP,IFA>
 :: 2001:7d0:xxxx:xxxx::1 :: bge0:64.xx.xx.xx.xx.85 2001:7d0:xxxx:xxxx:xxxx:xxxx:xxxx:1785

Code: 
root@Argentum ~# ping google.com
PING6(56=40+8+8 bytes) 2001:7d0:xxxx:xxxx:xxxx:xxxx:xxxx:1785 --> 2a00:1450:4026:804::200e
16 bytes from 2a00:1450:4026:804::200e, icmp_seq=0 hlim=58 time=4.744 ms
16 bytes from 2a00:1450:4026:804::200e, icmp_seq=1 hlim=58 time=4.023 ms
16 bytes from 2a00:1450:4026:804::200e, icmp_seq=2 hlim=58 time=4.351 ms
16 bytes from 2a00:1450:4026:804::200e, icmp_seq=3 hlim=58 time=3.502 ms

Code: 
root@Argentum ~# traceroute6 -n google.com
traceroute6 to google.com (2a00:1450:4026:805::200e) from 2001:7d0:xxxx:xxxx:xxxx:xxxx:xxxx:1785, 64 hops max, 28 byte packets
 1  2001:7d0:xxxx:xxxx::1  1.165 ms  0.982 ms  0.728 ms
 2  * * *
 3  * * *
 4  2001:2000:3080:82d::1  4.009 ms  3.147 ms  3.245 ms
 5  2001:4860:1:1::2197  4.795 ms
    2001:2034:0:13f::1  41.101 ms  4.973 ms
 6  2001:4860:1:1::2196  4.490 ms  11.840 ms  5.104 ms
 7  2a00:1450:808e::1  4.976 ms  5.540 ms
    2a00:1450:8082::1  5.144 ms
 8  2a00:1450:4026:805::200e  4.363 ms
    2001:4860:0:1::44e4  5.724 ms  5.928 ms

And from VM, the trace ends in 101:

Code: 
root@Helium ~# traceroute6 -n google.com
traceroute6 to google.com (2a00:1450:4026:804::200e) from 2001:7d0:xxxx:xxxx::102, 64 hops max, 28 byte packets
 1  2001:7d0:xxxx:xxxx::101  0.210 ms  0.162 ms  0.116 ms
root@Helium ~ [SIGINT]#
 
Argentum said:
Thank you for your feedback. I will debug this later, but as a quick response:
Click to expand...
Take Your time.
Now let me show You how to work with this information (because it's of not much use here for me, but rather for You to see what is going to happen):
Argentum said:
Code: 
root@Helium ~# route -6 -nv get google.com
[...]
   route to: 2a00:1450:4026:804::200e
[...]
    gateway: 2001:7d0:xxxx:xxxx::101
        fib: 0
  interface: vtnet0
[..]
sockaddrs: <DST,GATEWAY,NETMASK,IFP,IFA>
 :: 2001:7d0:xxxx:xxxx::101 :: vtnet0:58.9c.fc.e.34.cb 2001:7d0:xxxx:xxxx::102
Click to expand...
These are the important informations here:
  • the "gateway" is tne next hop that the packet is expected to reach
  • the "fib" is only important if you use these
  • the "interface" is our local interface on this machine where the packet should go out through
  • the "IFA" (last line, last item: 2001:7d0:xxxx:xxxx::102) is the IP-address on the sending interface that goes into the packet as the sender address (if there is none yet).
With this information we can then start tcpdump on that interface and check if the packets do actually go out there and have that sender address.

Then we go to our next hop and do the same: check with tcpdump if the packets do actually come in from the expected interface, look at route [-6] -nv get ... where that host wants to send them, and then check if they actually go out there.

And, obviousely, then we do the same thing backwards for the replies: watch them come in on the outbound interface, see which destination address they carry, check with 'route get' what this host wants to do with such a destination address, and verify with tcpdump that it actually happens.
 
PMc said:
Take Your time.
Click to expand...
Took my time - cool! It is working now. I can ping my desktop routed VM from outside world.

Thank you for your help!

...




... and the solution was to configure IPV6 static route to my upstream router.

Code: 
Route to 2001:7d0:xxxx:xxxx::100/120

But here another question rises - how can I do this in Hetzner where I do not have access to the upstream router. Or is there some sort of route preconfigured so that it can route to subnets inside /64 network?

Marking this thread solved.
 
Argentum said:
... and the solution was to configure IPV6 static route to my upstream router.

Code: 
Route to 2001:7d0:xxxx:xxxx::100/120

But here another question rises - how can I do this in Hetzner where I do not have access to the upstream router. Or is there some sort of route preconfigured so that it can route to subnets inside /64 network?
Click to expand...
Hm, it seems You built somthing that is not exactly what I imagined... ;)
I thought the upstream router should send the entire /64 into your host - as that is what Hetzner will most likely do.
 
PMc said:
Hm, it seems You built somthing that is not exactly what I imagined... ;)
I thought the upstream router should send the entire /64 into your host - as that is what Hetzner will most likely do.
Click to expand...
But in my home network the upstream router must obviously know, which computer has this new /120 subnet. That was my reasoning and it worked. I own the router and have access to it. So I configured the route.

But in general, I am not sure if that works in Hetzner.
 
Argentum said:
Hello!

Seems that I need some help here.

The main machine is FreeBSD13.1 with Bhyve guests. I am using sysutils/vm-bhyve for VM-s management.
On the main machine I have network configured (and working):


Code: 
inet6 2a01:xxx:xx:xxxx::10 prefixlen 64

Also I have bridge configured and accessible form the world

Code: 
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether ea:b6:29:db:cf:77
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 2a01:xxx:xx:xxxx::1 prefixlen 64

V4 guests are working. And have a net configuration:
Code: 
vtnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether 58:9c:fc:0f:26:bc
        inet6 fe80::5a9c:fcff:fe0f:26bc%vtnet0 prefixlen 64 scopeid 0x1
        inet 192.168.1.100 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

The V4 configuration comes from my local DHCP server. V4 traffic goes through NAT configured in pf:

Code: 
nat on $ext_if from {192.168.1.0/24} to any -> ($ext_if)

Need advice here how to configure bhyve machine to be accessible over V6?
Click to expand...
Dear argentum
how do you add ipv6 and ipv4 to vm-public switch in bhybe at same time ? please show me the step ? thanks.
 
You must log in or register to reply here.
Back
Top