Vlan configuration for Bhyve

D

Dodge

Hi there,

currently I have a FreeBSD 12.3 Server for a school project. The goal of this project is to create a website from which you can create virtual machines. This works atm. The other goal of the project is to create bhyve switches with vlan id's so that the machines with the same vlan id can ping each other etc... However the server only has 1 physical ethernet interface. How would I go about creating virtual interfaces that get their Ip addresses from the physical interface so I can create switches connected to these virtual interfaces to give the virtual machines their Ip address and tag their network traffic?

Thanks so much in advance already!!
 
Dodge said:
However the server only has 1 physical ethernet interface. How would I go about creating virtual interfaces that get their Ip addresses from the physical interface so I can create switches connected to these virtual interfaces to give the virtual machines their Ip address and tag their network traffic?
Click to expand...
This is automatically done if you use vm switch ... from sysutils/vm-bhyve.

If you do this by hand, then just create a vlan(4) interface with the correct VLAN ID and linked to your physical interface. Then add that vlan(4) interface to a bridge(4). Attach your VMs to that bridge. Create a vlan(4) interface and bridge(8) for each of the VLANs you need.

Code: 
root@hosaka:~ # vm switch list
NAME     TYPE      IFACE       ADDRESS  PRIVATE  MTU   VLAN  PORTS
servers  standard  vm-servers  -        no       9000  11    lagg0
public   standard  vm-public   -        no       9000  10    lagg0
Those correspond with a vm-public and vm-servers bridge interface:
Code: 
root@hosaka:~ # ifconfig vm-public
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        ether d6:3f:41:5c:c3:9f
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap12 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 23 priority 128 path cost 2000000
        member: tap8 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 19 priority 128 path cost 2000000
        member: tap7 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 18 priority 128 path cost 2000000
        member: tap6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 17 priority 128 path cost 2000000
        member: tap5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 16 priority 128 path cost 2000000
        member: lagg0.10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 10 priority 128 path cost 2000000
        groups: bridge vm-switch viid-4c918@
        nd6 options=9<PERFORMNUD,IFDISABLED>

For this one lagg0.10 is the vlan(4) interface:
Code: 
root@hosaka:~ # ifconfig lagg0.10
lagg0.10: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: vm-vlan-public-lagg0.10
        options=4000000<NOMAP>
        ether 00:25:90:f1:58:39
        groups: vlan vm-vlan viid-bdfd6@
        vlan: 10 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Those vlan(4) and bridge(4) combinations are created by vm(8) ( vm switch ...), no need to configure anything else.

Crude drawing: igb1, igb2 -> lagg(4) (bundles the two physical interfaces) -> vlan(4) -> bridge(4) -> VM
 
When I use vm switch create to create a virtual switch from to vm-bhyve sysutils then I get this output. I had already created one from before which I linked to the public switch but I dont have any other physical interface. So how would I go about connecting the test switch then because this one atm doesnt work.
Code: 
root@Matthias9:~ # vm switch list
NAME    TYPE      IFACE      ADDRESS  PRIVATE  MTU  VLAN  PORTS
public  standard  vm-public  -        no       -    -     em0
test    standard  vm-test    -        no       -    -     -
 
Last edited by a moderator:
oh this is what interfaces I have now as well when these two switches have been created.
Code: 
root@Matthias9:~ # ifconfig
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=81209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
        ether f4:8e:38:bd:18:98
        inet6 fe80::f68e:38ff:febd:1898%em0 prefixlen 64 scopeid 0x1
        inet6 2a02:a03f:874c:3400:f68e:38ff:febd:1898 prefixlen 64 autoconf
        inet 200.200.200.102 netmask 0xffffff00 broadcast 200.200.200.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vm-public: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 66:a6:04:fc:4b:52
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 200000
        groups: bridge vm-switch viid-4c918@
        nd6 options=9<PERFORMNUD,IFDISABLED>
vm-test: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether ce:68:80:75:5c:59
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000000
        groups: bridge vm-switch viid-098f6@
        nd6 options=1<PERFORMNUD>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet-test-0-test
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:18:7a
        inet6 fe80::5a9c:fcff:fe10:187a%tap0 prefixlen 64 tentative scopeid 0x5
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 42284
 
SirDice said:
This is automatically done if you use vm switch ... from sysutils/vm-bhyve.

If you do this by hand, then just create a vlan(4) interface with the correct VLAN ID and linked to your physical interface. Then add that vlan(4) interface to a bridge(4). Attach your VMs to that bridge. Create a vlan(4) interface and bridge(8) for each of the VLANs you need.

Code: 
root@hosaka:~ # vm switch list
NAME     TYPE      IFACE       ADDRESS  PRIVATE  MTU   VLAN  PORTS
servers  standard  vm-servers  -        no       9000  11    lagg0
public   standard  vm-public   -        no       9000  10    lagg0
Those correspond with a vm-public and vm-servers bridge interface:
Code: 
root@hosaka:~ # ifconfig vm-public
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        ether d6:3f:41:5c:c3:9f
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap12 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 23 priority 128 path cost 2000000
        member: tap8 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 19 priority 128 path cost 2000000
        member: tap7 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 18 priority 128 path cost 2000000
        member: tap6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 17 priority 128 path cost 2000000
        member: tap5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 16 priority 128 path cost 2000000
        member: lagg0.10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 10 priority 128 path cost 2000000
        groups: bridge vm-switch viid-4c918@
        nd6 options=9<PERFORMNUD,IFDISABLED>

For this one lagg0.10 is the vlan(4) interface:
Code: 
root@hosaka:~ # ifconfig lagg0.10
lagg0.10: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: vm-vlan-public-lagg0.10
        options=4000000<NOMAP>
        ether 00:25:90:f1:58:39
        groups: vlan vm-vlan viid-bdfd6@
        vlan: 10 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Those vlan(4) and bridge(4) combinations are created by vm(8) ( vm switch ...), no need to configure anything else.

Crude drawing: igb1, igb2 -> lagg(4) (bundles the two physical interfaces) -> vlan(4) -> bridge(4) -> VM
Click to expand...
I have tried this but to no avail. Could I ask for some more information? Thanks!
 
You could also take a look at how sysutils/chyves does things behind the curtain. Like vm-bhyve it is purely shell based, so you can just look at the code and see what it actually does.

I wouldn't recommend using it today because it still has some rough edges and weird behaviour (and development seems to have stalled). I still have it on 2 machines where I also constantly fall over the annoyingly inverted syntax: chyves <vm> <commands> instead of e.g. vm/iohyve/iocage/iocell/AnyOtherVmOrJailTool <command> <jail/vm>.
 
SirDice said:
What more information do you need?

github.com

Virtual Switches

Shell based, minimal dependency bhyve manager. Contribute to churchers/vm-bhyve development by creating an account on GitHub.
github.com github.com
Click to expand...
I tried it with the vm switch create command but this didn't work. The vm that's attached to this switch doesn't have internet. Then I tried it with creating a bridge and a virtual interface and attached this to the bridge but this didn't work either. What would be the order in which to create this manually? Thanks!!
 
SirDice said:
What more information do you need?

github.com

Virtual Switches

Shell based, minimal dependency bhyve manager. Contribute to churchers/vm-bhyve development by creating an account on GitHub.
github.com github.com
Click to expand...
Ok so my physical interface is called em0. I have created a virtual interface em0.10. When I create a vm switch, lets say its name is test, this is the bridge then right? I create this test switch then and add interface em0.10 to this switch. Then I tell my virtual machine that it should attach to test. So then an interface tap0 gets created and attached to this vm-test bridge right? But then when I start my virtual machine it still doesn't have internet. Am I doing something wrong?
 
Is em0 connected to a 'trunked' port on the switch? Is the switch passing those VLANs? Is there a router on your network that actually routes the traffic between the VLANs?
 
SirDice said:
Is em0 connected to a 'trunked' port on the switch? Is the switch passing those VLANs? Is there a router on your network that actually routes the traffic between the VLANs?
Click to expand...
em0 should indeed be connected to a 'trunked' port but when I go into my router I see the vlan option is not enabled :P Perhaps this will fix it :P Was the way I did it correct though? SirDice :P
 
SirDice said:
Is em0 connected to a 'trunked' port on the switch? Is the switch passing those VLANs? Is there a router on your network that actually routes the traffic between the VLANs?
Click to expand...
With was the way I did it correct I meant is the order in which I created the bhyve switch etc.. so would you indeed first create a virtual interface by doing
Code: 
ifconfig create em0.10
and then create a bhyve switch by doing
Code: 
vm switch create test
and then add this switch an em0.10 interface together like so
Code: 
vm switch add test em0.10
and then change the network switch in your vm config to this test switch?
 
Dodge said:
so would you indeed first create a virtual interface by doing
Click to expand...
That's not a vlan(4) interface, it's named like a VLAN interface but not configured as such. ifconfig em0.10 vlan 10 vlandev em0

Code: 
     vlan vlan_tag
             Set the VLAN tag value to vlan_tag.  This value is a 12-bit VLAN
             Identifier (VID) which is used to create an 802.1Q or 802.1ad
             VLAN header for packets sent from the vlan(4) interface.  Note
             that vlan and vlandev must both be set at the same time.
...
     vlandev iface
             Associate the physical interface iface with a vlan(4) interface.
             Packets transmitted through the vlan(4) interface will be
             diverted to the specified physical interface iface with 802.1Q
             VLAN encapsulation.  Packets with 802.1Q encapsulation received
             by the parent interface with the correct VLAN Identifier will be
             diverted to the associated vlan(4) pseudo-interface.  The vlan(4)
             interface is assigned a copy of the parent interface's flags and
             the parent's Ethernet address.  The vlandev and vlan must both be
             set at the same time.  If the vlan(4) interface already has a
             physical interface associated with it, this command will fail.
             To change the association to another physical interface, the
             existing association must be cleared first.

             Note: if the hardware tagging capability is set on the parent
             interface, the vlan(4) pseudo interface's behavior changes: the
             vlan(4) interface recognizes that the parent interface supports
             insertion and extraction of VLAN tags on its own (usually in
             firmware) and that it should pass packets to and from the parent
             unaltered.

Just use vm switch create -n $VLANID -i em0 myvlanswitch for example.
Code: 
root@hosaka:~ # vm switch create -n 20 -i lagg0 testswitch
root@hosaka:~ # vm switch list
NAME        TYPE      IFACE          ADDRESS  PRIVATE  MTU   VLAN  PORTS
servers     standard  vm-servers     -        no       9000  11    lagg0
public      standard  vm-public      -        no       9000  10    lagg0
testswitch  standard  vm-testswitch  -        no       -     20    lagg0
 
SirDice said:
That's not a vlan(4) interface, it's named like a VLAN interface but not configured as such. ifconfig em0.10 vlan 10 vlandev em0

Code: 
     vlan vlan_tag
             Set the VLAN tag value to vlan_tag.  This value is a 12-bit VLAN
             Identifier (VID) which is used to create an 802.1Q or 802.1ad
             VLAN header for packets sent from the vlan(4) interface.  Note
             that vlan and vlandev must both be set at the same time.
...
     vlandev iface
             Associate the physical interface iface with a vlan(4) interface.
             Packets transmitted through the vlan(4) interface will be
             diverted to the specified physical interface iface with 802.1Q
             VLAN encapsulation.  Packets with 802.1Q encapsulation received
             by the parent interface with the correct VLAN Identifier will be
             diverted to the associated vlan(4) pseudo-interface.  The vlan(4)
             interface is assigned a copy of the parent interface's flags and
             the parent's Ethernet address.  The vlandev and vlan must both be
             set at the same time.  If the vlan(4) interface already has a
             physical interface associated with it, this command will fail.
             To change the association to another physical interface, the
             existing association must be cleared first.

             Note: if the hardware tagging capability is set on the parent
             interface, the vlan(4) pseudo interface's behavior changes: the
             vlan(4) interface recognizes that the parent interface supports
             insertion and extraction of VLAN tags on its own (usually in
             firmware) and that it should pass packets to and from the parent
             unaltered.

Just use vm switch create -n $VLANID -i em0 myvlanswitch for example.
Code: 
root@hosaka:~ # vm switch create -n 20 -i lagg0 testswitch
root@hosaka:~ # vm switch list
NAME        TYPE      IFACE          ADDRESS  PRIVATE  MTU   VLAN  PORTS
servers     standard  vm-servers     -        no       9000  11    lagg0
public      standard  vm-public      -        no       9000  10    lagg0
testswitch  standard  vm-testswitch  -        no       -     20    lagg0
Click to expand...
Thanks!!!!! Sorry I didn't fully understand till this :P haha ty so much :)
 
RTFM - read the friendly manpages.

No, really! They are great on FreeBSD (or the BSDs and the other remaining 'real' UNIXes like illumos in general), especially for the base system and -tools they are usually exceptionally well written and thorough and usually contain helpful examples.
Usually if you stumble upon something that doesn't have a manpage, it's a package/port that has been "imported" from linux where they seem to dread writing manpages or documentation "like the devil dreads the holy water" as a german proverb has it...

If you don't know what manpage might contain what you are looking for, use apropos(1) and/or the manpages mentioned in the "SEE ALSO" section of a manpage that is near the topic you are looking into.
 
Phishfry said:
It seems an important networking function like vlan(4) would indeed have a basic EXAMPLES section.
Unfortunately it does not.
ifconfig(8) is void of a VLAN example too.
Click to expand...

TBH I didn't check the ifconfig/vlan manpages for this - have been using it for years and somehow believed it is mentioned in one of the manpages.

At least in the description of the first vlan-specific parameter ('vlan') states:
The following parameters are specific to vlan(4) interfaces:

vlan vlan_tag
Set the VLAN tag value to vlan_tag. This value is a 12-bit VLAN
Identifier (VID) which is used to create an 802.1Q VLAN header
for packets sent from the vlan(4) interface. Note that vlan and
vlandev must both be set at the same time.
Click to expand...

But yes, I also think the required parameters could/should be mentioned and shown in an example. This might be worth a PR...
 
I always thought PR's for "Feature Request" was ridiculous but in this case I agree.

The top 3 sites (for me) from Google are all outside sources too.

I think SirDice example should make an appearance in one of those two manuals.
 
OK I will help with this one.
Here is a sample of what I think works. Please correct any faults.
Please add your vlan config to this post if unique.
Also suggest helpful examples.

<<Add Description like ifconfig manpage>>
Create the Virtual LAN device interface of em0.10 on parent device em0
ifconfig em0.10 vlan 10 vlandev em0

<<Add Description like ifconfig manpage>>
Remove the Virtual LAN device interface on em0.10
ifconfig em0.10 -vlan 10 vlandev em0


<<Add Description like ifconfig manpage>>
Create the VLAN device interface vlan.200 on parent interface lagg0 and set static ip
ifconfig lagg0.200 create vlan 200 vlandev lagg0 inet 192.168.1.3/24
 
Thank you very much. It is extremely difficult to find any notes on VM networking that are NOT referring you back to vm-bhvye. Maybe it's an excellent tool, I don't know... but hey, I want to learn how things work. And man pages are cryptic, like you must be an IT university graduate to use that stuff. It's almost like if you're "one of us", you'll understand it, and if you're not -- well, too bad for you LOL.
 
free-and-bsd said:
Thank you very much. It is extremely difficult to find any notes on VM networking that are NOT referring you back to vm-bhvye. Maybe it's an excellent tool, I don't know... but hey, I want to learn how things work. And man pages are cryptic, like you must be an IT university graduate to use that stuff. It's almost like if you're "one of us", you'll understand it, and if you're not -- well, too bad for you LOL.
Click to expand...
Sounds like complaining doesn't it.

I'll give an example. For example, I didn't find in any mans much about ROUTING in vlans. Yes, it mentions forwarding packets between vlans etc. But how exactly do I implement that in terms of networking?
Say I have an outbound igb0 interface which is parent to igb0.5.

Now it never mentions how vlan 5 will be able to use igb0 as default gateway. In terms of networking, your GATEWAY must be on the same network. Now by definition vlan 5 is a different network from that of igb0, its parent interface. Which means route add default will be rejected.

So how does it look practically? Man says nothing except that packets are being forwarded there and back. But how does it work and look for a mchine using such vlan as the only network available to it? So it's like some other docs must cover it, but they don't either...
 
free-and-bsd said:
For example, I didn't find in any mans much about ROUTING in vlans. Yes, it mentions forwarding packets between vlans etc. But how exactly do I implement that in terms of networking?
Click to expand...
If you're asking this question you don't fully understand what a VLAN is. There is no routing with VLANs. All a VLAN does is break up the network in different broadcast domains, in simpler words, network segments. Routing is done on a router. You typically have a 'core' router that has all the VLANs connected to it and the routing is "just" the regular routing you would do without VLANs. Think of a VLAN as a separate switch and connection because for all intents and purposes that's what they are.
 
T-Daemon said:
There is a guide Understanding VLAN Configuration on FreeBSD I bookmarked a while ago (haven't tried it yet in practice). It has nice sketched. Assuming what is explained there corresponds to the best application and has no errors, the sketches and manual configuration makes understanding vlans clearer.
Click to expand...
Read this one already)) It didn't say anything about routing either.

Ok, here's what will answer my question completely. Does it all mean that for igb0.5 vlan to get access to igb0 LAN I need a firewall rule? And, especially, do I need ip forwarding enabled on the host for that to work? The latter is the most important for me to understand.
 
You must log in or register to reply here.
Back
Top