Solved Please advise with bhyve IPv6 configuration

Argentum

Argentum

Hello!

Seems that I need some help here.

The main machine is FreeBSD13.1 with Bhyve guests. I am using sysutils/vm-bhyve for VM-s management.
On the main machine I have network configured (and working):


Code: 
inet6 2a01:xxx:xx:xxxx::10 prefixlen 64

Also I have bridge configured and accessible form the world

Code: 
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether ea:b6:29:db:cf:77
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 2a01:xxx:xx:xxxx::1 prefixlen 64

V4 guests are working. And have a net configuration:
Code: 
vtnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether 58:9c:fc:0f:26:bc
        inet6 fe80::5a9c:fcff:fe0f:26bc%vtnet0 prefixlen 64 scopeid 0x1
        inet 192.168.1.100 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

The V4 configuration comes from my local DHCP server. V4 traffic goes through NAT configured in pf:

Code: 
nat on $ext_if from {192.168.1.0/24} to any -> ($ext_if)

Need advice here how to configure bhyve machine to be accessible over V6?
 
Run rtadvd on the "vm-public" interface on the host, this will announce the global prefix.
Then run rtsold (assuming its FreeBSD, other OS's have a similar alternative) in the guest, to request an address from the announced prefix.
 
driesm said:
Run rtadvd on the "vm-public" interface on the host, this will announce the global prefix.
Then run rtsold (assuming its FreeBSD, other OS's have a similar alternative) in the guest, to request an address from the announced prefix.
Click to expand...
  1. Started rtadvd on vm-public
  2. rtsold was running - restarted
Still no success.

Code: 
root@cobalt ~# less /etc/rtadvd.conf
vm-public:\
                   :addr="2a01:xxx:xx:xxxx::":prefixlen#64:

I am missing something here.
 
1) Check the output of "rtadvctl -vv show", is it advertising correctly on vm-public?
Try running rtadvd without config file, it is only needed if you pass "-s" to rtadvd or very specific reasons.

2) lets check if 1) is set-up correctly first
 
driesm said:
1) Check the output of "rtadvctl -vv show", is it advertising correctly on vm-public?
Try running rtadvd without config file, it is only needed if you pass "-s" to rtadvd or very specific reasons.

2) lets check if 1) is set-up correctly first
Click to expand...
Hello:

Seems that it is not advertising on vm-public.
I had in the /etc/rtadvd.conf the following lines:

Code: 
vm-public:\
                   :addr="2a01:xxx:xx:xxxx::":prefixlen#64:

Code: 
root@cobalt ~# rtadvctl -vv show
igb0: flags=<UP,TRANSITIVE,PERSIST> status=<RA_RECV> mtu 1500
        DefaultLifetime: 30m
        MinAdvInterval/MaxAdvInterval: 3m20s/10m
        AdvLinkMTU: <none>, Flags: <none>, Preference: medium
        ReachableTime: 0s, RetransTimer: 0s, CurHopLimit: 64
        AdvIfPrefixes: yes
        Next RA send: Sat Jun 11 16:04:54 2022
        Last RA send: never
        Prefixes (1):
          2a01:xxx:xx:xxxx::/64 (KERNEL, vltime=30d, pltime=7d, flags=LA)

        Counters
         RA burst counts: 3 (interval: 16s)
         RS wait counts: 0
        Outputs
         RA: 0
        Inputs
         RA: 0 (normal)
         RA: 0 (inconsistent)
         RS: 0

lo0: flags=<UP> status=<INACTIVE>
vm-public: flags=<UP> status=<INACTIVE>
tap0: flags=<UP> status=<INACTIVE>

Removed the configuration file and restarted, but it is all the same.

After rtadvctl enable vm-public it is there, but still no connection from the VM.

Code: 
lo0: flags=<UP> status=<INACTIVE>
vm-public: flags=<UP,TRANSITIVE,PERSIST> status=<INACTIVE> mtu 1500
        DefaultLifetime: 30m
        MinAdvInterval/MaxAdvInterval: 3m20s/10m
        AdvLinkMTU: <none>, Flags: <none>, Preference: medium
        ReachableTime: 0s, RetransTimer: 0s, CurHopLimit: 64
        AdvIfPrefixes: yes
        Next RA send: Sat Jun 11 16:15:45 2022
        Last RA send: never
        Prefixes (1):
          2a01:xxx:xx:xxxx::/64 (KERNEL, vltime=30d, pltime=7d, flags=LA)

        Counters
         RA burst counts: 3 (interval: 16s)
         RS wait counts: 3
        Outputs
         RA: 0
        Inputs
         RA: 0 (normal)
         RA: 0 (inconsistent)
         RS: 3

tap0: flags=<UP> status=<INACTIVE>
 
Argentum said:
After rtadvctl enable vm-public it is there, but still no connection from the VM.
Click to expand...

Did you restart rtsold in the guest after making these changes? It can take some time to automatically see the change, if you force it to restart it should get an address very fast.

You might want to set the interfaces you want to advertise on in rc.conf, like rtadvd_interfaces="vm-public".
If you also want to advertise on igb0 just space separate the interfaces in the rtadvd_interfaces variable.
Otherwise I don't understand why igb0 is active in advertising, unless you also have it statically configured in rtadvd.conf.

Anyway, it shouldn't take more than this, advertising the prefix on the host and running the router solicitation on the guest side, check firewall, etc.

EDIT: after checking output a bit more closely it seems rtadvd is still not advertising.
A correct output should look like this (I have no config file), see the difference in status

bridge1: flags=<UP,CONFIGURED,PERSIST> status=<RA_SEND> mtu 1500
DefaultLifetime: 30m
MinAdvInterval/MaxAdvInterval: 3m20s/10m
AdvLinkMTU: <none>, Flags: <none>, Preference: medium
ReachableTime: 0s, RetransTimer: 0s, CurHopLimit: 64
AdvIfPrefixes: yes
Next RA send: Sat Jun 11 18:12:23 2022
Last RA send: Sat Jun 11 18:05:27 2022
Prefixes (1):
fd00:0:0:2::/64 (KERNEL, vltime=30d, pltime=7d, flags=LA)

Counters
RA burst counts: 0 (interval: 16s)
RS wait counts: 0
Outputs
RA: 11171
Inputs
RA: 0 (normal)
RA: 0 (inconsistent)
RS: 1031
Click to expand...
 
driesm said:
Did you restart rtsold in the guest after making these changes? It can take some time to automatically see the change, if you force it to restart it should get an address very fast.

You might want to set the interfaces you want to advertise on in rc.conf, like rtadvd_interfaces="vm-public".
If you also want to advertise on igb0 just space separate the interfaces in the rtadvd_interfaces variable.
Otherwise I don't understand why igb0 is active in advertising, unless you also have it statically configured in rtadvd.conf.

Anyway, it shouldn't take more than this, advertising the prefix on the host and running the router solicitation on the guest side, check firewall, etc.

EDIT: after checking output a bit more closely it seems rtadvd is still not advertising.
A correct output should look like this (I have no config file), see the difference in status
Click to expand...
I set the variable in /etc/rc.conf and restarted rtadvd. Now the output is

Code: 
root@cobalt ~ [1]# rtadvctl -vv show
igb0: flags=<UP> status=<INACTIVE>
lo0: flags=<UP> status=<INACTIVE>
vm-public: flags=<UP,TRANSITIVE,PERSIST> status=<INACTIVE> mtu 1500
        DefaultLifetime: 30m
        MinAdvInterval/MaxAdvInterval: 3m20s/10m
        AdvLinkMTU: <none>, Flags: <none>, Preference: medium
        ReachableTime: 0s, RetransTimer: 0s, CurHopLimit: 64
        AdvIfPrefixes: yes
        Next RA send: Sat Jun 11 19:16:32 2022
        Last RA send: never
        Prefixes (1):
          2a01:xxx:xx:xxxx::/64 (KERNEL, vltime=30d, pltime=7d, flags=LA)

        Counters
         RA burst counts: 3 (interval: 16s)
         RS wait counts: 0
        Outputs
         RA: 0
        Inputs
         RA: 0 (normal)
         RA: 0 (inconsistent)
         RS: 0

tap0: flags=<UP> status=<INACTIVE>

In the pf.conf I have the last lines

Code: 
pass in all
pass out all

I can ping the v6 address of the bridge.
 
driesm said:
Can you show ifconfig output of the vm-public interface?
Click to expand...
Yes:
Code: 
root@cobalt ~# ifconfig -v vm-public
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether ea:b6:29:db:cf:77
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 2a01:xxx:xx:xxxx::1 prefixlen 64
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000000
        groups: bridge vm-switch viid-4c918@
        nd6 options=1<PERFORMNUD>
I have a step forward -

sysctl net.inet6.ip6.forwarding=1

And now it seems advertising:
Code: 
root@cobalt ~# rtadvctl -vv show
igb0: flags=<UP> status=<INACTIVE>
lo0: flags=<UP> status=<INACTIVE>
vm-public: flags=<UP,CONFIGURED,PERSIST> status=<RA_SEND> mtu 1500
        DefaultLifetime: 30m
        MinAdvInterval/MaxAdvInterval: 3m20s/10m
        AdvLinkMTU: <none>, Flags: <none>, Preference: medium
        ReachableTime: 0s, RetransTimer: 0s, CurHopLimit: 64
        AdvIfPrefixes: yes
        Next RA send: Sat Jun 11 19:42:18 2022
        Last RA send: Sat Jun 11 19:36:55 2022
        Prefixes (1):
          2a01:xxx:xx:xxxx::/64 (KERNEL, vltime=30d, pltime=7d, flags=LA)

        Counters
         RA burst counts: 0 (interval: 16s)
         RS wait counts: 0
        Outputs
         RA: 4
        Inputs
         RA: 0 (normal)
         RA: 0 (inconsistent)
         RS: 3

tap0: flags=<UP> status=<INACTIVE>

Restarted rtsold on the guest, but still no connection.
 
Argentum said:
root@cobalt ~# rtadvctl -vv show
igb0: flags=<UP> status=<INACTIVE>
lo0: flags=<UP> status=<INACTIVE>
vm-public: flags=<UP,CONFIGURED,PERSIST> status=<RA_SEND> mtu 1500
DefaultLifetime: 30m
MinAdvInterval/MaxAdvInterval: 3m20s/10m
AdvLinkMTU: <none>, Flags: <none>, Preference: medium
ReachableTime: 0s, RetransTimer: 0s, CurHopLimit: 64
AdvIfPrefixes: yes
Next RA send: Sat Jun 11 19:42:18 2022
Last RA send: Sat Jun 11 19:36:55 2022
Prefixes (1):
2a01:xxx:xx:xxxx::/64 (KERNEL, vltime=30d, pltime=7d, flags=LA)

Counters
RA burst counts: 0 (interval: 16s)
RS wait counts: 0
Outputs
RA: 4
Inputs
RA: 0 (normal)
RA: 0 (inconsistent)
RS: 3
Click to expand...

Nice, so seeing the output, it seems to be advertising and it has even received 3 router solicitations from somewhere.
I would try to dig deeper by TCP dumping in the guest and check if the RA's are seen there, and vice versa, if the RS's are seen on the host.

How are you using the host, are you doing routing + client? Then you might need the following rc.conf variables:
ipv6_cpe_wanif="em0" (external iface here)
ipv6_gateway_enable="YES" (underlyingly this will set net.inet6.ip6.forwarding=1 and some extra stuff IIRC)

During this phase you can also use rtsol, not the daemon rtsold, you can run rtsol in the foreground in the VM to print some additional debug info.
 
driesm said:
Nice, so seeing the output, it seems to be advertising and it has even received 3 router solicitations from somewhere.
I would try to dig deeper by TCP dumping in the guest and check if the RA's are seen there, and vice versa, if the RS's are seen on the host.

How are you using the host, are you doing routing + client? Then you might need the following rc.conf variables:
ipv6_cpe_wanif="em0" (external iface here)
ipv6_gateway_enable="YES" (underlyingly this will set net.inet6.ip6.forwarding=1 and some extra stuff IIRC)

During this phase you can also use rtsol, not the daemon rtsold, you can run rtsol in the foreground in the VM to print some additional debug info.
Click to expand...
Thank you for your advice.

Yes, I have enabled the ipv6 gateway on the main machine.

Digging deeper, it looks like routing v6 problem. Please correct me if I am wrong, but the main machine does not know how to route from main interface to the VM-s on the virtual switch.

Assume, the solution would be to put the external interface on the switch, but when I did so some time ago, I received a strict abuse message from the service provider:

Code: 
Dear Mr,

We have detected that your server is using different MAC addresses from those allowed by your Robot account.

Please take all necessary measures to avoid this in the future and to solve the issue.
We also request that you send a short response to us. This response should contain information about how this could have happened and what you intend to do about it.
In the event that the following steps are not completed successfully, your server can be locked at any time after...

Do not even understand why this is an abuse by their policy, but I had to remove the external interface from the switch.

Now, I assume, I need routing, but within prefixlen 64 it does not seem possible to divide it to additional subnets (if I understand the v6 specs properly).

With v4 I have NAT rule for VM-s internet access and required ports forwarded.

Code: 
nat on $ext_if from {192.168.1.0/24} to any -> ($ext_if)

# SSH

rdr pass on $ext_if proto tcp from any to $my_ip port $port_private -> $starter_virt_ip port $port_ssh

But for V6 there is no NAT. Also, experimentally I configured local v6 addresses for interfaces (inet6 fd00::1 prefixlen 8) and I can ping the switch interface from VM.

My idea is to create a PF rule to forward from internal v6 addresses to the external addresses on the main interface. But so far I have not managed to create a correct rule (if this is possible). I gave googled and read the manual, but not been able to create PF rule to move all trafic from external to internal V6 addresses.

Please correct me if I am misunderstanding something.

Additional information - when I try to ping switch v6 addres from VM I get in the dump on the main machine:

Code: 
20:23:45.944768 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2a01:xxx:xx:xxxx::20 > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2a01:xxx:xx:xx::1
          source link-address option (1), length 8 (1): 58:9c:fc:0f:26:bc
            0x0000:  589c fc0f 26bc
 
Could you briefly describe which interfaces you have on your machine and what there function is and how they are configured IPv6 wise?
How do you obtain your global prefix? (DHCPv6, auto configuration? Which prefix length do you get from your ISP? Only one /64?)
Then shortly describe what you try to accomplish, just trying to get your VM's IPv6 accessible?

Just to prevent an X-Y problem.
 
Argentum said:
Code: 
root@cobalt ~# ifconfig -v vm-public
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether ea:b6:29:db:cf:77
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 2a01:xxx:xx:xxxx::1 prefixlen 64
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000000
        groups: bridge vm-switch viid-4c918@
        nd6 options=1<PERFORMNUD>
Click to expand...
I think you are also missing an IPv6 link-local address on your bridge interface which is needed for rtadvd and rtsol.
You can fix this manually by "ifconfig vm-public inet6 auto_linklocal". Take a look at rc.conf to make it permanent on start-up.
 
driesm said:
I think you are also missing an IPv6 link-local address on your bridge interface which is needed for rtadvd and rtsol.
You can fix this manually by "ifconfig vm-public inet6 auto_linklocal". Take a look at rc.conf to make it permanent on start-up.
Click to expand...
Thank you!

Now I have on the bridge
Code: 
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Restarted rtadvd and rtsold, but cannot ping the bridge IP6.

On the VM vtnet0 I have:
Code: 
inet6 2a01:xxx:xx:xxxx:5a9c:fcff:fe0f:26bc prefixlen 64 autoconf

Which I can ping locally, but not from the main machine.

Routing on the main machine seems wrong:

Code: 
oot@cobalt ~ [2]# route get -6 2a01:xxx:xx:xxxx:5a9c:fcff:fe0f:26bc
   route to: 2a01:xxx:xx:xxxx:5a9c:fcff:fe0f:26bc
destination: 2a01:xxx:xx:xxxx::
       mask: ffff:ffff:ffff:ffff::
        fib: 0
  interface: igb0
      flags: <UP,DONE,PINNED>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

Pinging the main machine gives on the bridge dump:
Code: 
21:11:03.211019 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) 2a01:xxx:xx:xxxx:5a9c:fcff:fe0f:26bc > ff02::1:ff00:10: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2a01:xxx:xx:xxxx::10
          source link-address option (1), length 8 (1): 58:9c:fc:0f:26:bc
            0x0000:  589c fc0f 26bc
 
It has been almost half a year, but still I have no solution in this case.

I think I need to clarify a little bit more - this is associated with the service provider (Hetzner) and their strange policy not to allow internal VM-s MAC addresses to be visible on the outside network. (As soon as I put the outside interface on the internal virtual switch the service provider starts sending nasty abuse messages threatening to cut the connection. It is completely unclear why they consider this as an abuse. And yes, we have some reasons to have this server there...)

I have two different installations in a different network which work just fine. The only difference is that outside network interface is a member of internal virtual switch (vm-public).

In this case here I cannot have external interface on the switch. With IP4 I have a rule in the pf.conf to do NAT and then I map internal ports needed to the outside IP:
Code: 
nat on $ext_if from {192.168.1.0/24} to any -> ($ext_if)

With IP6 there is no NAT and in fact there is the whole 64 bit range of internal IP6 addresses which are not routable by specification. However this should still be possible to break the internal 64 bit address range into smaller (say 2) pieces (subnets) and forward traffic from one such local subnet to the external IP6-s. I can configure any number of IP6-s on the external interface and this is OK. The service provider only does not want to see packets coming out with other MAC address than the physical interface.

I have not managed to configure it in such way. Any help would be appreciated.
 
Argentum said:
The service provider only does not want to see packets coming out with other MAC address than the physical interface.
Click to expand...
This is understandable. If you create arbitrary MAC, then the outside bridge tables will fill with these, and there may even be duplicates.

Argentum said:
Now, I assume, I need routing, but within prefixlen 64 it does not seem possible to divide it to additional subnets (if I understand the v6 specs properly).
Click to expand...
It is absolutely possible. But some software will not work with that, specifically concerning prefix delegation will not work.
Normally you have three parts:
  • your network, usually /48, or anything bigger than /64
  • the "sla" (site level aggregation) - thats the piece between 48 and 64, and is basically your internal network number
  • the host part, which should be 64 bits for the fancy 'SLAAC' translation to work.
You can certainly chop another network part off the host part (and treat it just like with IPv4 CIDR), but then there are four parts, and rtadv will not know how to handle this correctly. This is a problem if you have coming&going client systems which get their addresses dynamically assigned, or if you are on a home-network that gets a dynamic prefix delegation from upstream. It should not be a problem for server infrastructure with static addresses+routing

And if all else fails, you can internally use a network from fd00::/7, and translate in ipfw with NPTv6. Then you have the same as NAT does. It's a bit ambitioned to do the math, but I have it working.
 
PMc said:
It is absolutely possible. But some software will not work with that, specifically concerning prefix delegation will not work.
Normally you have three parts:
  • your network, usually /48, or anything bigger than /64
  • the "sla" (site level aggregation) - thats the piece between 48 and 64, and is basically your internal network number
  • the host part, which should be 64 bits for the fancy 'SLAAC' translation to work.
You can certainly chop another network part off the host part (and treat it just like with IPv4 CIDR), but then there are four parts, and rtadv will not know how to handle this correctly. This is a problem if you have coming&going client systems which get their addresses dynamically assigned, or if you are on a home-network that gets a dynamic prefix delegation from upstream. It should not be a problem for server infrastructure with static addresses+routing
Click to expand...
Thank you! I will try.
 
PMc said:
You can certainly chop another network part off the host part (and treat it just like with IPv4 CIDR), but then there are four parts, and rtadv will not know how to handle this correctly. This is a problem if you have coming&going client systems which get their addresses dynamically assigned, or if you are on a home-network that gets a dynamic prefix delegation from upstream. It should not be a problem for server infrastructure with static addresses+routing
Click to expand...
Do you have any good reference or an example how to configure such internal network?
 
Argentum said:
Do you have any good reference or an example how to configure such internal network?
Click to expand...
That's the problem with IPv6: there are a lot of papers explaining the basics of how an address looks like, but very few tutorials about what would be best-practices or how to set up something specific.
So I did a writeup for what I configured and which problems I encountered, but that is concerned about how to deal with dynamic prefixes and getting them distributed throughout a network.

The openvpn wiki has a mention about when you have only one /64, how to split it in halves and use only one half for your hosts, getting the other half free for the tunnel endpoints, but it doesn't get very clear from that.

So, lets consider the most simple case: you have two bhyve, and they have only a tap interface created, no bridge no nothing. Now you need a separate network for each tap interface, because the tap interface in the host and the vtnet0 in the guest together form a logical ethernet.

How do we do this in IPv4: we grab a network, say 192.168.42.0/24, and split it along:
bhyve-1 network 192.168.42.0/28
bhyve-2 network 192.168.42.16/28

and we distribute the IPs - inside the bhyve:
bhyve-1:
ifconfig vtnet0 192.168.42.2/28
route add -net default 192.168.42.1
bhyve-2:
ifconfig vtnet0 192.168.42.18/28
route add -net default 192.168.42.17

and on the host, for the tap:
ifconfig tap0 192.168.42.1/28
ifconfig tap1 192.168.42.17/28

In IPv6 you can do the same. Imagine we get the network 2001:db8::/64 from somewhere upstream routed into our machine.
Usually our machine (the host) will use only one or two of the 2^64 addresses provided. The rest can be split into other networks.

bhyve-1 network 2001:db8::100/120
bhyve-2 network 2001:db8::200/120

bhyve-1:
ifconfig vtnet0 inet6 2001:db8::102/120
route add -6 -net default 2001:db8::101
bhyve-2:
ifconfig vtnet0 inet6 2001:db8::202/120
route add -6 -net default 2001:db8::201

and on the host, for the tap:
ifconfig tap0 inet6 2001:db8::101/120
ifconfig tap1 inet6 2001:db8::201/120

# and the host itself:
ifconfig lo0 inet6 2001:db8::1/128

# and on the host we should also add a void route for the entire /64, so that the remaining addresses do not get routed astray:
route add -6 -reject -net 2001:db8::/64

Finally everything else goes from the host into a default route to the upstream IPv6 gateway. This depends a bit on how that outbound connection to the gateway actually works.

So just grab your laptop, create two guests ad try it out...
 
PMc said:
How do we do this in IPv4: we grab a network, say 192.168.42.0/24, and split it along:
bhyve-1 network 192.168.42.0/28
bhyve-2 network 192.168.42.16/28

and we distribute the IPs - inside the bhyve:
bhyve-1:
ifconfig vtnet0 192.168.42.2/28
route add -net default 192.168.42.1
bhyve-2:
ifconfig vtnet0 192.168.42.18/28
route add -net default 192.168.42.17

and on the host, for the tap:
ifconfig tap0 192.168.42.1/28
ifconfig tap1 192.168.42.17/28

In IPv6 you can do the same. Imagine we get the network 2001:db8::/64 from somewhere upstream routed into our machine.
Usually our machine (the host) will use only one or two of the 2^64 addresses provided. The rest can be split into other networks.

bhyve-1 network 2001:db8::100/120
bhyve-2 network 2001:db8::200/120

bhyve-1:
ifconfig vtnet0 inet6 2001:db8::102/120
route add -6 -net default 2001:db8::101
bhyve-2:
ifconfig vtnet0 inet6 2001:db8::202/120
route add -6 -net default 2001:db8::201

and on the host, for the tap:
ifconfig tap0 inet6 2001:db8::101/120
ifconfig tap1 inet6 2001:db8::201/120

# and the host itself:
ifconfig lo0 inet6 2001:db8::1/128

# and on the host we should also add a void route for the entire /64, so that the remaining addresses do not get routed astray:
route add -6 -reject -net 2001:db8::/64

Finally everything else goes from the host into a default route to the upstream IPv6 gateway. This depends a bit on how that outbound connection to the gateway actually works.

So just grab your laptop, create two guests ad try it out...
Click to expand...
Thank you!

This is exactly what I am going to do - try this out on the test machine. Let you all know about the results later.
 
PMc said:
So just grab your laptop, create two guests ad try it out...
Click to expand...
Seems that I still misunderstood something. Here is what happened:

Started VM (Helium) and configured interface:
Code: 
root@Helium ~ [1]# ifconfig vtnet0 inet6 2001:7d0:xxxx:yyyy::102/120
root@Helium ~# route add -6 -net default 2001:7d0:xxxx:yyyy::101
add net default: gateway 2001:7d0:xxxx:yyyy::101

Configured main machine:

Code: 
root@Argentum ~# ifconfig tap0 inet6 2001:7d0:xxxx:yyyy::101/120
root@Argentum ~# ifconfig lo0 inet6 2001:7d0:xxxx:yyyy::1/128
root@Argentum ~# route add -6 -reject -net 2001:7d0:xxxx:yyyy::/64
add net 2001:7d0:xxxx:yyyy::/64 fib 0: route already in table

After that I could ping main machine from VM and VM from main machine. But the VM was not able to ping outside addresses.

Noticed that forwarding was not set on the main machine. When I entered

Code: 
root@Argentum ~# sysctl net.inet6.ip6.forwarding=1

All the outside IPV6 connectivity was lost.
 
Argentum said:
Seems that I still misunderstood something. Here is what happened:

Started VM (Helium) and configured interface:
Code: 
root@Helium ~ [1]# ifconfig vtnet0 inet6 2001:7d0:xxxx:yyyy::102/120
root@Helium ~# route add -6 -net default 2001:7d0:xxxx:yyyy::101
add net default: gateway 2001:7d0:xxxx:yyyy::101

Configured main machine:

Code: 
root@Argentum ~# ifconfig tap0 inet6 2001:7d0:xxxx:yyyy::101/120
root@Argentum ~# ifconfig lo0 inet6 2001:7d0:xxxx:yyyy::1/128
root@Argentum ~# route add -6 -reject -net 2001:7d0:xxxx:yyyy::/64
add net 2001:7d0:xxxx:yyyy::/64 fib 0: route already in table
Click to expand...
That is an error. The /64 network is already configured, probably per the outside connection.
So what is already in the route table concerning 2001:7d0 ? netstat -rn6 | grep ^2001:7d0

Argentum said:
After that I could ping main machine from VM and VM from main machine. But the VM was not able to ping outside addresses.
Click to expand...
Okay.

Argentum said:
Noticed that forwarding was not set on the main machine. When I entered
Click to expand...
That would be needed, so that one guest can ping the other guest as well.

Argentum said:
Code: 
root@Argentum ~# sysctl net.inet6.ip6.forwarding=1

All the outside IPV6 connectivity was lost.
Click to expand...
And this leads to the question how your outside connection is supposed to work - which I don't know.
I'm using mostly hurricane tunnels, because they delegate the reverse DNS to me, and my hoster doesn't. My hoster, however, would expect me to use a dhcpv6 client to configure the system - and I found, I need to run that dhcpv6 client to request a lease in order for the hoster to enable the traffic flow, but I do not need to allow it to configure my system; I can do that in a different way manually.

You must have a manual from your hoster, explaining how to configure the network and how to get the upstream gateway IPv6.
Or, asking differently: how does the default route on the main machine look like when connectivity works? (And where is that configured?)
 
PMc said:
That is an error. The /64 network is already configured, probably per the outside connection.
So what is already in the route table concerning 2001:7d0 ? netstat -rn6 | grep ^2001:7d0


You must have a manual from your hoster, explaining how to configure the network and how to get the upstream gateway IPv6.
Or, asking differently: how does the default route on the main machine look like when connectivity works? (And where is that configured?)
Click to expand...
The test machine sits in my basement and is connected directly to my fiber optic line.
For IPV6 configuration I have just one line

Code: 
ifconfig_bge0_ipv6="inet6 accept_rtadv"

in the rc.conf. Everything has been working a long time. Also, when I connect bge on the virtual switch, the VM-s can be autoconfigured and everything works. I can also ping individual VM-s from outside.

Also

Code: 
root@Argentum ~# netstat -rn6 | grep ^2001:7d0
2001:7d0:xxxx:yyyy::/64           link#1                        U          bge0
2001:7d0:xxxx:yyyy::1             link#2                        UHS         lo0
2001:7d0:xxxx:yyyy::100/120       link#3                        U      vm-inter
2001:7d0:xxxx:yyyy::101           link#3                        UHS         lo0
2001:7d0:xxxx:yyyy:....:....:....:1785 link#1                    UHS         lo0

P.S. vm-inter is a switch. Tried directly with tap also with exactly the same result.
 
Argentum said:
The test machine sits in my basement and is connected directly to my fiber optic line.
For IPV6 configuration I have just one line

Code: 
ifconfig_bge0_ipv6="inet6 accept_rtadv"
Click to expand...
Okay, that's the problem - as I said, rtadv does not play well with splitting /64.

Argentum said:
Also

Code: 
root@Argentum ~# netstat -rn6 | grep ^2001:7d0
2001:7d0:xxxx:yyyy::/64           link#1                        U          bge0
2001:7d0:xxxx:yyyy::1             link#2                        UHS         lo0
2001:7d0:xxxx:yyyy::100/120       link#3                        U      vm-inter
2001:7d0:xxxx:yyyy::101           link#3                        UHS         lo0
2001:7d0:xxxx:yyyy:....:....:....:1785 link#1                    UHS         lo0
Click to expand...
The first line here says that the bge0 does already take occupancy of the entire /64.

This is what accept_rtadv does: it computes the address for the local machine according to SLAAC (that is the one ending with 1785) and configures the network on the interface. The Prefix Length (/64) is contained in the message received from upstream, and therefore cannot be locally changed.

The solution is to not use rtadv. Anyway you may not want to use rtadv when running a server: when you run a (e.g. web)server, you want to publish the IP in DNS, so you cannot use one that is computed by SLAAC and changes whenever the network card gets swapped. (Okay, you can always add a fixed address additionally.)

If you do not use rtadv, you have to configure the network manually. Therefore you need to know your IPv6 nexthop/gateway address.
 
PMc said:
The solution is to not use rtadv. Anyway you may not want to use rtadv when running a server: when you run a (e.g. web)server, you want to publish the IP in DNS, so you cannot use one that is computed by SLAAC and changes whenever the network card gets swapped. (Okay, you can always add a fixed address additionally.)

If you do not use rtadv, you have to configure the network manually. Therefore you need to know your IPv6 nexthop/gateway address.
Click to expand...
Thank you. I will try it later and use fixed V6 address on the main machine. As I wrote, this is just a test machine in my home network and I am trying to simulate the conditions in Hetzner. Do not want to go and fiddle there with the settings and eventually land on hard reset.
 
You must log in or register to reply here.
Back
Top