Jails vs bhyve

priyadarshan

priyadarshan

Since FreeBSD 11 is here, I am tempted to simplify our server setup, and use Bhyve also for all jail needs.

I would not mind exchanging a small percentage of available CPU for more simplicity, but does it make sense to standardise on Bhyve vs Jails, in order to simplify setup and workflow?

In other words, does one loses anything by using only Bhyve, where he could instead use Jails?

If so, what would be the "break-even point" in a server environment that would favour using Bhyve instead of Jails?
 
It all depends on your hardware resources and needs. It would be a waste of resources to run FreeBSD inside Bhyve if using the same version as the host. I would only assume if user wants total anonymity then FreeBSD inside Bhyve will be better since admins won't have access to contents inside Bhyve. The only drawback is if the user forgets the login password and there's no way to get back into OS inside Bhyve then reinstalling is the only way back in. With Jail, user can ask the admin to reset the password. With Bhyve, users are usually on their own in maintaining updates and configurations since admins don't have direct access. With Jail, admin can help users with updates, configurations, etc.

User can pretty much do anything inside Jails except they won't be able to use network stacks or firewall. If you recompile the kernel with vimage then user will be able to configure their own firewall inside Jail. Most servers have front-end firewall, router or firewall in host, and it'll be a waste of resource to have another firewall inside Jail or Bhyve. I usually have web and mail ports open for my clients and I can reconfigure the firewall upon special requests.

Bhyve is better for different OS other than FreeBSD such as Linux, Windows, etc. It makes no sense to run same FreeBSD version in both Host and Bhyve since it'll be a waste of resources.
 
Behyve doesn't really protect you from a malicious admin, the root user on the host has full access to the kernel's virtual memory space trough the /dev/mem device and all of the memory pages used a bhyve virtual machine will be visible there.
 
The only drawback is if the user forgets the login password and there's no way to get back into OS inside Bhyve then reinstalling is the only way back in
Click to expand...

There's a few ways you can get into FreeBSD without the root password. If they haven't been silly enough to set the console as insecure you can just boot into single user mode, re-mount the root file system r/w, then change root's password. Alternatively you could just mount the guest's disk image directly on the host and clear the password. One of the nice things about bhyve using "raw" format for VM disk images is that they work just like a real disk if you attach it to an md device.

Of course it's a bit different if they've encrypted the disk and forgotten that password.

Bhyve doesn't really protect you from a malicious admin
Click to expand...
Do you mean on the bhyve host itself? There's no reason to expect that a virtual machine on any hypervisor is safe from someone who has access to the host.

Obviously bhyve has higher overhead that jails, probably a noticeable performance impact, and higher memory use due to having a separate kernel loaded for each machine. I do appreciate it can make management easier though. Running lots of services in jails does worry me when it comes to performing upgrades on the host, especially major versions. With virtual machines things are more much self-contained. Guests can easily be moved between FreeBSD versions (assuming they both support bhyve), and the host can be upgraded without worrying about breaking anything in the guests.
 
Thank you, that's what I felt, when I asked about simplifying setup and workflow.
Also, some peace of mind, to really be isolated from host, even FreeBSD-on-FreeBSD, as an extra layer of protection from upgrades and other unreliable activities.
As I said, I do not mind using a few more percentage points of available CPU.
Also, in general we always have quite a bit of extra disk space and extra ram available, and it would seem a nice way to put it to good use.
 
If you don't plan to update FreeBSD inside Bhyve and your users will do all the updates themselves then Bhyve is better. If your user expects you to do the updates then Jail is better. Doing updates in Bhyve in single user mode will require a lot of downtime for your user and more of your time as well.

I usually don't have problems updating Jails during minor and major upgrades but its important to keep everything updated including packages. I use poudriere to keep all packages updated since some packages require customized settings. Of course, I do test runs on my test server first to be sure nothing breaks before doing any major work on the production server.

I use rsync to migrate Jails to other server.
 
Remington said:
I usually don't have problems updating Jails during minor and major upgrades but its important to keep everything updated including packages.
Click to expand...

I am curious to learn what jail management software you are using. For me, updating the jails is a real pain. I am currently using ejzail but I have not managed to perform binary upgrades.
 
gkontos said:
I am curious to learn what jail management software you are using. For me, updating the jails is a real pain. I am currently using ejzail but I have not managed to perform binary upgrades.
Click to expand...

I use ezjail as well. I don't use binary upgrades. Everything is compiled and installed from source which makes updating Jails much easier.
 
Remington said:
I use ezjail as well. I don't use binary upgrades. Everything is compiled and installed from source which makes updating Jails much easier.
Click to expand...

Yes, that's the way I do it also. But if you have a lot of jails it would be nice to be able to perform binary updates. Why do you think that source updating is easier?
 
gkontos said:
Yes, that's the way I do it also. But if you have a lot of jails it would be nice to be able to perform binary updates. Why do you think that source updating is easier?
Click to expand...

Main reason why I compile from source is custom kernel. I've always compile from source and never had issues. I've seen binary upgrade had problems or broke something. At least I have compiled src and obj files I can deploy on my other servers, perform upgrade and all servers will have same patched version.

I guess its a habit that I've been doing for very long time.
 
Remington said:
Main reason why I compile from source is custom kernel. I've always compile from source and never had issues. I've seen binary upgrade had problems or broke something. At least I have compiled src and obj files I can deploy on my other servers, perform upgrade and all servers will have same patched version.

I guess its a habit that I've been doing for very long time.
Click to expand...

I started using binary updates / upgrades this year. Like you I always used the source way. If you look at the recent Security Advisories regarding openssl you might start wondering if it is worth the procedure again!

I only wish that FreeBSD would come with IPSEC as standard in the kernel. That is the only reason why I still use custom kernels.
 
The make installworld procedure is quite time costly and suffers from one major weakness, you don't have a record of what was installed and nothing that was installed can be tracked afterwards. This is what the PkgBase project is trying to address. Once we have that it will be possible to build a set of base packages using the system sources using whatever customizations you wish and upgrade a jail base system with just a simple pkg upgrade operation.

https://wiki.freebsd.org/PkgBase
 
I've started using libressl recently. It seems many distros are adopting libressl as default installation and I ran into some ports that requires libressl. That's the reason why I started using poudriere to create packages using libressl. Personally, I think FreeBSD should remove openssl from its default installation and allow users to choose libressl or openssl.
 
Remington said:
I've started using libressl recently. It seems many distros are adopting libressl as default installation and I ran into some ports that requires libressl. That's the reason why I started using poudriere to create packages using libressl. Personally, I think FreeBSD should remove openssl from its default installation and allow users to choose libressl or openssl.
Click to expand...

I wouldn't be surprised if this happens quite soon in FreeBSD 12 with LibreSSL set as default, OpenSSL is much more of a liability than asset in its current form.

I doubt that OpenSSL is removed completely from base any time soon. There are some very hard to solve technical issues with that because the base system would then have to include a "switchboard" system to divert to either a base system OpenSSL or to OpenSSL/LibreSSL installed from ports and that's far from trivial to implement.
 
bhyve - Virtualisation: Creates a blank-disk that can be used to run an entire-OS (bootloader, kernel, userland, etc.) by the virtualisation software.

Jails - OS-Level Virtualisation: Jails uses the same kernel and main resources as the host OS (FreeBSD). But creates a separate and isolated userland and network-space. So the Jail cannot see the filesystems of the host OS.
 
You must log in or register to reply here.
Back
Top