Solved Running Jails inside Bhyve

GoNeFast_01

GoNeFast_01

There is some writing about running Bhyve within a Jail (testing the opposite).

Can you run jails within a Bhyve freebsd instances?

I assume yes so, so I spend a few hours messing with a Bhyve FreeBSD I have for testing purposes... Installed a few jails in it and while I can make them communicate within the Bhyve network.

I cannot get the Vnet jails within the Bhyve to route to the outside network (I cannot ping 8.8.8.8 for example). Bhyve can ping to 8.8.8.8 and can hit the router/gateway... VNET jails cannot hit the router/gateway nor the 9.9.9.9.

I can ping the epair0a, I can ping a VLAN in host(if I created 10.0.100.1), I can ping an alias in bhyve host if I assign it to the vtnet0 nic. I am guessing maybe there has to do something with the route table but I am not too sure.

Has anyone tried doing JAILS within BHYVE emulation? How did you do the routing or Vnet configuration? Can you post your config files?

Why wouldn't I do Jails inside of Bhyve? (Outside of the resources) Is it too slow?

Code: 
#mkdir -p /usr/jails/jails-data/JTest_1-data
zfs create -o mountpoint=/usr/jails/jails-data/JTest_1-data zroot_fbsd/ROOT/default/JTest_1
ifconfig epair create
jail -c name=JTest_1 host.hostname=JTest_1.domain.com path=/usr/jails/jails-data/JTest_1-data children.max=5 persist vnet vnet.interface=epair0b allow.mount=1 allow.mount.devfs=1 allow.raw_sockets=1 devfs_ruleset=4 sysvmsg=1 sysvsem=1 sysvshm=1
ifconfig bridge create
ifconfig bridge addm epair0a addm vtnet0
&& \
jail -r JTest_1 || jail -r 1
zfs destroy -R zroot_fbsd/ROOT/default/JTest_1

Do I have to nat the nic/epair with PF?
 
Your bridge isn't connected to anything besides the jail. There's no uplink interface, so the traffic would never leave the host.
 
SirDice said:
Your bridge isn't connected to anything besides the jail. There's no uplink interface, so the traffic would never leave the host.
Click to expand...
Could you provide a quick a example SirDice ? I did create a bridge on the host/bhyve and I assigned the vnet end of the jail epair0a (also added the vtnet0 of the bhyve).

I also tested by adding a second bridge inside the jail via bridge0 addm epair0b and still nothing....

I also tested by putting the epair in the same subnet as the vtnet/bhyve host and gateway, for some reason I still I cannot get the uplink jail leave the host/bhyve.
 
It's going to be a little complex, but treat the VM as a separate machine (for all intents and purposes that's what it is). When you configure a VNET jail you tie one end of the epair to the jail and the other to a bridge. That bridge has to be connected to an external interface or else it will not be able to communicate with the outside world. A bridge is a layer 2 connection. It currently has no other connection on layer 2.

So you have a machine (the jail), that's connected to a switch (the bridge) but that switch isn't connected to anything else. That's the situation you have right now.
 
This is last test I performed trying to get the jail to ping to 8.8.8.8... No Result ....

Ping between 10.1.100.1 and 10.1.100.2 works...

HOST/BHYVE
Code: 
vtnet0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80008<VLAN_MTU,LINKSTATE>
        ether 00:a0:98:8d:85:ae
        inet 192.168.2.240 netmask 0xffffff00 broadcast 192.168.2.255
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair0a: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:33:b8:b5:7f:0a
        inet 10.1.100.1 netmask 0xffffff00 broadcast 10.1.100.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8802<UP,BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:10:ff:c2
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 0 ifcost 0 port 0
        member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 2000
        member: vtnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 2000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>

JAIL:

Code: 
jexec JTest_1 ifconfig

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair0b: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:33:b8:b5:7f:0b
        inet 10.1.100.2 netmask 0xffffff00 broadcast 10.1.100.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Any thoughts SirDice

HOST/BHYVE:

Code: 
ping 8.8.8.8
64 bytes from 8.8.8.8: icmp_seq=0 ttl=118 time=6.078 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=5.065 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss

JAIL:
Code: 
jexec JTest_1 ping 8.8.8.8
ping: sendto: Network is unreachable
ping: sendto: Network is unreachable
ping: sendto: Network is unreachable
ping: sendto: Network is unreachable
^C
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
 
I'm betting it has to do with the return traffic. The pings are actually going out but there's nothing coming back. That's a common mistake. When a packet travels through a network any routers it encounters only deal with the destination address. But the response packets have their source and destination addresses reversed. So not only do you have to look at the routes for packets traveling from A to B, you also need to look at the route of the return traffic going from B to A. People new to networking always forget the return traffic.

tcpdump(1) is what you need to use, run it on various parts, like the jail's bridge. Then the bridge on the VM host. Then your external interfaces. Actually have a look at the packets themselves. Double check where they're going and where they end up. Also check if you see anything being returned or not (the response packets) and where things stop appearing.
 
Emrion said:
Your bridge is down. Try:
ifconfig bridge0 up
Click to expand...
Thank you small oversight (I tested with this UP and DOWN) ... This does not fix the issue no ping 8.8.8.8 from inside jail.

I always figured it was something dealing with route so here are the current BHYVE settings there....
HOST/BHYVE
Code: 
#netstat -nr
Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.2.254      UGS      vtnet0
10.1.100.0/24      link#3             U       epair0a
10.1.100.1         link#3             UHS         lo0
127.0.0.1          link#2             UHS         lo0
192.168.2.0/24     link#1             U        vtnet0
192.168.2.240      link#1             UHS         lo0

JAIL
Code: 
Internet:
Destination        Gateway            Flags     Netif Expire
10.1.100.0/24      link#2             U       epair0b
10.1.100.2         link#2             UHS         lo0
127.0.0.2          link#1             UH          lo0


TCPDUMP (1) ... I was using it just couldn't figure out where package where dropping due to next issue (so this is what I have done already testing wise).....

Code: 
tcpdump -c 1000 -i vtnet0 -e
tcpdump -c 1000 -i bridge0 -e
tcpdump -c 1000 -i epair0a -e
Above VERY NOISY no point in putting any output.... But its getting everything (ARP :Request who-has.... etc) all the way from gateway 192.168.2.254

SirDice here comes the next issue I am facing....
Question is HOW do I get tcpdump working inside Jail in BHYVE.... ?

/etc/devfs.rules
[devfsrules_jail=4]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path fuse unhide
add path zfs unhide
add path dev unhide
add path 'dev/*' unhide
add path 'bpf*' unhide
add path 'vtnet*' unhide
add path 'epair*' unhide

I tested with all of the above trying to get tcpdump working inside the jail CANNOT passed the tcpdump is there some setting on the bhyve end that I need to enable to be able to pass the bpf to jail?

JAIL:
Code: 
#jexec JTest_1 tcpdump -c 1000 -i epair0b -e
tcpdump: (there are no BPF devices)
 
Emrion said:
You thought it gonna work better with a bridge down? :rolleyes:
Click to expand...
Sure... Thanks you know... The hacker mindset + tcpdump you never know what stupidity can go through someone's mind. :-/

Anyways I got this to work... Crazy , but hey for anyone stupid like me here it is. *Below recipe could probably be done more efficient with tun/tap also.*

Code: 
#mkdir -p /usr/jails/jails-data/JTest_1-data
zfs create -o mountpoint=/usr/jails/jails-data/JTest_1-data zroot_fbsd/ROOT/default/JTest_1
ifconfig epair create
#epair0
ifconfig epair create
#epair1
jail -c name=JTest_1 host.hostname=JTest_1.domain.com path=/usr/jails/jails-data/JTest_1-data children.max=5 persist vnet vnet.interface=epair0b allow.mount=1 allow.mount.devfs=1 allow.raw_sockets=1 devfs_ruleset=4 sysvmsg=1 sysvsem=1 sysvshm=1
ifconfig bridge create
ifconfig bridge addm epair0a addm vtnet0 up
ifconfig epair0a inet 10.0.100.1/24 up
ifconfig epair1a inet 192.168.2.243/24 up
jexec JTest_1 ifconfig epair0b inet 10.0.100.2/24 up
jexec JTest_1 ifconfig epair1b inet 192.168.2.242/24 up
jexec JTest_1 ifconfig bridge create
jexec JTest_1 ifconfig bridge0 addm epair0b addm epair1b up
jexec JTest_1 route add default [YOUR DEFAULT_GATEWAY=192.168.2.254 my case] #*You need to go around the BHYVE net directly to router jail - > bhyve -> host (doubt its possible to do these hops).#
&& \
jail -r JTest_1 || jail -r 1
zfs destroy -R zroot_fbsd/ROOT/default/JTest_1

BHYVE/HOST:
Code: 
ping -c 5 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=118 time=4.420 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=5.168 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=118 time=5.369 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=118 time=5.090 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=118 time=4.631 ms

--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 4.420/4.936/5.369/0.353 ms

JAIL:
Code: 
jexec JTest_1 ping -c 5 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=118 time=6.446 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=5.029 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=118 time=5.299 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=118 time=5.414 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=118 time=5.122 ms

--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 5.029/5.462/6.446/0.510 ms

Complete.....

I am still curious on how to get tcpdump working inside a jail with BHYVE emulation thought anyone? (Review the devfs.rules I tested) virtio vnet passed via the bpf or is this something else?

Code: 
#jexec JTest_1 tcpdump -c 1000 -i epair0b -e
tcpdump: (there are no BPF devices)
 
Last edited:
While it would certainly be nice to have a look with tcpdump(1) in the jail itself it's not really necessary if you can observe the traffic on the bridge. If you can see the traffic on the bridge you don't need to look in the jail itself. Because the only way that traffic would be on the bridge is if the jail's network is working.
 
SirDice said:
While it would certainly be nice to have a look with tcpdump(1) in the jail itself it's not really necessary if you can observe the traffic on the bridge. If you can see the traffic on the bridge you don't need to look in the jail itself. Because the only way that traffic would be on the bridge is if the jail's network is working.
Click to expand...
I agreed with you but is there a way? Everyone say is a security risk (understand this) I would enable only for testing but can it be done BHYVE -> JAIL (tcpdump aka sniff packets from jail upwards BHYVE all the way to the host that is running bhyve) ? I tested with the devfs.rules above and no tcpdump.
 
GoNeFast_01 said:
I agreed with you but is there a way?
Click to expand...
There probably is but I don't know. I rarely use VNET jails and they behave a little different compared to 'regular' jails. With a regular jail I would suspect devfs.rules and the jail not being able to access the bpf(4) device. And perhaps raw_sockets. But it looks like you enabled those already.
 
You must log in or register to reply here.
Back
Top