bhyve nested bhyve virtual machines

I

ikevin8me

I'm now trying nested bhyve virtual machines:

Host OS: FreeBSD 13
Guest OS: FreeBSD 13
Yet Another Guest OS inside FreeBSD 13: OpenBSD 6.9

Upon trying to execute the "bhyve" command inside the Guest OS: FreeBSD 13, I get the following:
vmx_modinit: processor does not support VMX operation
module_register_init: MOD_LOAD (vmm, Oxffff .....) error 6
vm_create: Device not configured
Click to expand...

So, is there a way to make it work?
 
ikevin8me said:
I'm now trying nested bhyve virtual machines:

Host OS: FreeBSD 13
Guest OS: FreeBSD 13
Yet Another Guest OS inside FreeBSD 13: OpenBSD 6.9

Upon trying to execute the "bhyve" command inside the Guest OS: FreeBSD 13, I get the following:


So, is there a way to make it work?
Click to expand...
I think BHYVE does not support nested virtualization this way. The virtualization hardware in the CPU is already taken by the main hypervisor.
 
I tried to use qemu, but it does not seem to be so easy. Anydefinitive guide to install OpenBSD as guest?
 
Why do you want to nest?

You should keep in mind that there's no hardware supporting this. It can be done in software of course, but the performance will suffer. For more background, read for example: https://en.wikipedia.org/wiki/Second_Level_Address_Translation

A design decision for bhyve is to only work with this hardware support, so you can never nest bhyve inside a bhyve guest. Other hypervisors with the same design decision include Linux' kvm and OpenBSD's vmm.
 
ikevin8me said:
I'm now trying nested bhyve virtual machines:

Host OS: FreeBSD 13
Guest OS: FreeBSD 13
Yet Another Guest OS inside FreeBSD 13: OpenBSD 6.9

Upon trying to execute the "bhyve" command inside the Guest OS: FreeBSD 13, I get the following:


So, is there a way to make it work?
Click to expand...

No, this is not supported as bhyve would have to emulate VT-x/SVM [1], which would definitely add a lot of bloat and has terrible performance anyways - so nobody would want to do that.

A more sane approach to contain bhyve instances would be running bhyve instances inside of jails. This approach is already used in production e.g. by ClonOS and Project-FiFo (IIRC Free/TrueNAS abandoned that approach and is/was using KVM directly on the host instead).
Using vnet-jails this is basically the same approach as smartOS is doing with bhyve/KVM inside of zones.
However, runnig FreeBSD as a bhyve guest on top of FreeBSD usually isn't what you want to do as jails are much faster and more flexible. For other guests the jail + bhyve approach adds another layer of security and it can simplify management and migration, which is still a bit fiddly with bhyve on its own. When carrying all configuration inside the jail, moving a VM to another host is basically the same as moving a jail - which nowadays is mostly automated by some tools (i.e. iocage/iocell).


[1] https://www.mail-archive.com/freebsd-virtualization@freebsd.org/msg09004.html
 
I do see some security advantages to using vnet/jail + bhyve. However...

In order to run bhyve within a jail, the "vmm" needs to be passed into the jail ("add path vmm unhide", etc.). This exposes all hardware devices to the jail and bhyve. (I tried dmidecode and pciconf - and everything shows up) So, I think there is no security enhancement with regards to accessing hardware. What is your opinion?
 
Well, exposing one device to a jail should never automatically expose other devices as well, so something is fishy there…

But still, I don't see what good starting bhyve from inside a jail should do. The major part of the hypervisor is implemented in the kernel anyways, therefore isn't restricted by a jail.

I do agree that for running a FreeBSD "VM" on FreeBSD, most of the time a jail is the best solution. If you need completely separated networking, go for VNET. There are usecases (like e.g. running a newer version of FreeBSD, or needing exclusive access to some PCI device inside the VM) where full virtualization is necessary, but if it isn't, go with a (VNET) jail.

Repeating my question from above: Why do you want to nest?

If the answer is something with "security", you're hunting an illusion…
 
Zirias said:
Repeating my question from above: Why do you want to nest?
Click to expand...
I want to test how far bhyve can go, in terms of performance and hardware abstraction, etc., before deciding/finalizaing how to set up the infrastructure. Based on the feedback and my research, I think bhyve can practically go just 1 level (ie. HostOS->(bhyve)GuestOS.
 
ikevin8me said:
I want to test how far bhyve can go, in terms of performance and hardware abstraction, etc., before deciding/finalizaing how to set up the infrastructure. Based on the feedback and my research, I think bhyve can practically go just 1 level (ie. HostOS->(bhyve)GuestOS.
Click to expand...
Just to make this absolutely clear: A hypervisor that allows nesting will still suffer from bad performance, because hardware only supports a single layer, see also link above about SLAT/NPT. So it's always the best idea not to nest virtual machines. There might be specific usecases for nesting (e.g. you could have a virtual machine for development and still want to test your software in a nested VM there), but you should be aware of the implications ;)
 
ikevin8me said:
I want to test how far bhyve can go, in terms of performance and hardware abstraction, etc., before deciding/finalizaing how to set up the infrastructure. Based on the feedback and my research, I think bhyve can practically go just 1 level (ie. HostOS->(bhyve)GuestOS.
Click to expand...

yes, it can go exactly 1 layer - because more doesn't make sense.
Bhyve was designed as a fast and ultra-lightweight hypervisor by leveraging modern cpu-mechanisms for virtualization with as less software emulation as possible. If you want/need every old cruft up to FDD and ISA controllers emulated, go one of the fully-(b)loa[t|d]ed hypervisors like QEMU, VMware or Virtualbox. They drag software-emulation for every obsolete hardware since the ~mid-80s around with them, which massively increases the attack surface and overhead and allow impractical and useless features like nesting over multiple layers.
 
sko said:
However, runnig FreeBSD as a bhyve guest on top of FreeBSD usually isn't what you want to do as jails are much faster and more flexible.
Click to expand...

There is one reason I can think of (and currently do think of) changing a jail to bhyve: dummynet is not working in jails, there is only one instance. (In 12.2 at least, but I have no notice that this would have improved in 13.)
Only I am not yet sure if (and how) I would route the network stuff into and out of a bhyve - but that should be workable.
 
The answer to a question on why someone would need a nested virtualization is simple when you need to run Linux on FreeBSD during mobile applications development and then run Android emulator inside Linux to test your application.

There are many other situations where we are forced to use Linux or worse OS because vendor decided these are only supported platforms.

Nested virtualization is possible, its called VT-X / AMD-V, quick search shows that it is supported by VMWare / VirtualBox / Hyper-v:

www.itechtics.com

How To Enable Nested Virtualization (VT-x) On Hyper-V, VMWare, VirtualBox VMs In Windows

Nested virtualization enables you to perform virtualization tasks on top of a VM, like running a VM over a VM. However, this feature needs to be enabled manually per VM.
www.itechtics.com www.itechtics.com

This is highly demanded feature, it would be nice to have one in BHYVE as well :-)
 
cederom said:
Nested virtualization is possible, its called VT-X / AMD-V
Click to expand...
Sorry to tell you, but, no, these are just the marketing names for the very similar "virtualization extensions" offered by intel resp amd CPUs. They include SLAT (I mentioned 2 years ago) as well as a few other things (like a second "dimension" for privileges). Still there's only support for ONE extra address translation level, for ONE extra privilege level for the hypervisor, etc. Supporting these extensions inside a VM means software emulation for anything missing, again. It will perform worse, there's no other way.

cederom said:
This is highly demanded feature, it would be nice to have one in BHYVE as well :)
Click to expand...
I don't think it would be a good idea to "waste" developer resources to build complex software emulation into bhyve, it would also contradict the design goal to be both lightweight and performant.
 
Allright.. so I have tried on the same machine, VirtualBox on Windoze, it reports VT-X/AMD-V ready, Debian inside runs Android Studio and the Virtual Device Emulator (as opposed to bhyve)... but it did not even fully boot the Android in long time lol. You are probably right this is only a marketing scam that does not even work well in reality. Hardware is here but it seems to be software trick only. Looks like I just need a dedicated development machine. Thanks for the hint! :-)
 
cederom said:
You are probably right this is only a marketing scam that does not even work well in reality. Hardware is here but it seems to be software trick only.
Click to expand...
Not exactly. I'm pretty sure some parts of the virtualization extensions (VT-X or AMD-V) can be used as-is even inside a VM. It's just, some other parts that are pretty relevant for performance (to reach the "close to native" promise) must be emulated, because the hardware can only offer one single level (like the very relevant SLAT, "second level ..." – you would need a "third level" for nesting, which isn't there)

When implemented correctly, nested virtualization should work nevertheless. Just performing worse. And it needs a lot of complex emulation code for that.
 
So I bought second hand USB 3.0 PCI-E controller (Dell's NEC uPD720200 USB 3.0 Host Controller), passed it to bhyve, attached my device over USB 3.0 Unitek HUB, works like a charm, thank you :-)
 
Leaving out the "Inception" jokes for the moment... I was on a bhyve dev call last week and they mentioned exactly what sko already highlighted. I didn't ask for the details, but there does seem to be merit and benefits in running bhyve in a jail.

It would allow you to run it with a separate vnet network stack, in an independent VLAN and there's likely more.
 
You must log in or register to reply here.
Back
Top