How to compare bastille-bsd, ezjail, cbsd, pot, iocage,ezjail

I can talk from ezjail and bastille , ezjail is solid and works well , but bastille see more the jails as "containers" and had more functions that ezjail, for example, the use of ZFS , ejecute commands on jails without enter the jail
I been a long time user of ezjail, but now I am using bastille
 
Used EZJail for a long time too. But the developer stopped a long time ago (there were some issues between the developers of Ezjail and QJail, not a pretty story). So it's getting a bit stale nowadays. Lacks a lot of modern features and still uses the deprecated jail_* variables. Switched to Bastille some time ago, and I'm very pleased with it (templates are a really cool feature).
 
If i'm correct bastille is for jailing , i.e. a name-space, ware-as cbsd is also for bhyve and virtualisation ? Not ?
It would be nice to know "Dan Langille" takes on this.

​

 
Alain De Vos said:
If i'm correct bastille is for jailing , i.e. a name-space, ware-as cbsd is also for bhyve and virtualisation ? Not ?
It would be nice to know "Dan Langille" takes on this.

​

Click to expand...
pot/bastillebsd/iocage/ezjail for jail only, whereas cbsd can also manage virtual machines (like vm-bhyve).

You can compare the open issues per project (for bastillebsd, for cbsd, for pot , for iocage): it shows what users are missing (feature request)
 
This is my very little opinion which is far far but really far from an expert one.
I am still discovering the jail world of FreeBSD, so far I tried iocage, bastille and lately pot.

Iocage is nice, the work is done automatically and quickly.
Having plugins is a good idea unfortunately there are not ton of them, I was oddly surprised that they were not much popular.
Sadly its development entered in a slow state, PR are kind of stuck, issues tickets are growing. I hope it won't die.
https://github.com/iocage/iocage/issues/1251

Bastille looks okay, I didn't play a lot with it for some reason and I didn't like the fact that it was always showing a warning message about pf while ipfw was used instead. It was like it was expected pf no matter what, then I moved on something else at that time and forget about bastille.
That being said I appreciated the videos the author did about it on YT, or the quick articles on the bastille blog because they help and this type of gesture are always welcome.

Pot is the last one I play with, I don't have a lot to say about it since I still discovering it, but so far so good.
Note that jails are deployed slowly because of the xz's archive. Syntax is okay as well as bastille.
From what I understand it's different kind from the two mentioned before, but since it is above my skills there are certainly good people here to talk about it better than I can.

Overall I just do simple things when using jails because I still struggle with network and more specifically firewall, add vnet into the equation and my brain freezes.
As an average user this is probably my bigger weak point.
So I try to cure myself I bought "Network For System Administrators" and "FreeBSD Mastery Jails" but damn that's a lot to process I won't lie!
I just wanted to have fun at first :-)


Alain De Vos said:
It would be nice to know "Dan Langille" takes on this.
Click to expand...
It looks like Dan is more into mkjail I think he prefers when things are simple even if he has to put his hands into it, I may be wrong about it this is only my own deduction.
github.com

GitHub - mkjail/mkjail

Contribute to mkjail/mkjail development by creating an account on GitHub.
github.com github.com
 
For jails, you can also go bare-bones.
Just write a manual /etc/jail.conf i use for poudriere in a jail
cat jail.conf
Code: 
# Common configs for all jails
path = "/jails/$name";
host.hostname = "$name";
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
persist;
ip4 = inherit;
ip6 = inherit;
mount.devfs;
mount.fdescfs;
allow.mount;
allow.mount.devfs;
allow.mount.fdescfs;
allow.mount.nullfs;
allow.mount.tmpfs;
allow.mount.procfs;
allow.mount.zfs;
enforce_statfs=1;
children.max=100;
allow.socket_af;
allow.raw_sockets;
allow.chflags;
allow.sysvipc;
a {
devfs_ruleset="20";
}

/etc/devfs.rules
Code: 
[a=20]
add include $devfsrules_hide_all  
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path 'mixer*' unhide
add path 'dsp*' unhide
add path 'zfs*' unhide
 
Purkuapas said:
pot/bastillebsd/iocage/ezjail for jail only, whereas cbsd can also manage virtual machines (like vm-bhyve).
Click to expand...
You're wrong. CBSD can also manage XEN and QEMU/NVMM ;-). But these are virtual machines, and therefore irrelevant in this topic. As far as containers, CBSD can create ARMvX/RISCV... jails on x86-64 arch using qemu-user-mode ( usefull for testers/porters ). Personally, it seems to me that due to the lack of support from commercial companies (or FreeBSD Foundation), all such "jail managers" reach the same stage and die. Things like "create", "clone", "start/stop", "delete", "import" and "export" containers are the most basic primitives, but we don't see anything beyond that (for example: scaling, service-discovery/healthing, DRS ( dynamic resource scheduling ), public registry ( templates like:

Code: 
pkg install <portname>
service enable <portname>
service start <portname>

^ personally, I don't find this useful enough for serious installations (you have to configure the service, right?))
Perhaps the 'pot' project tried to jump over this "great desert" by adopting Hashicorp Nomad. There is also a higher-level abstraction above CBSD in the form of an API that can perform DRS functions (for example: you can create environments on servers that are located in different data centers, with load distribution (on creation) and an interconnected overlay network via vxlan(4) ). But this, like Nomad, is a different product/service. Otherwise, all managers are the same, only the interface differs (and a set of bugs), to which different users give certain preferences.
 
It would be beneficial to all if one of more members could share their experiences (or pieces of experience) on the two contending clustering suite for *BSD. And if more than these two, please enlighten me.
1) Nomad/Pot: Here is an additional link: https://honeyguide.eu/posts/virtual-dc1/
2) Corosync/Pacemaker:
forums.freebsd.org

FreeBSD Cluster with Pacemaker and Corosync

I always missed ‘proper’ cluster software for FreeBSD systems. Recently I got to run several Pacemaker/Corosync based clusters on Linux systems. I thought how to make similar high availability solutions on FreeBSD and I was really shocked when I figured out that both Pacemaker and Corosync tools...
forums.freebsd.org forums.freebsd.org

How To Create a High Availability Setup with Corosync, Pacemaker, and Reserved IPs on Ubuntu 14.04 | DigitalOcean

This tutorial will demonstrate how you can use Corosync and Pacemaker with a Reserved IP to create a high availability (HA) server infrastructure on DigitalO…
www.digitalocean.com www.digitalocean.com
 
Alain De Vos said:
For jails, you can also go bare-bones.
Just write a manual /etc/jail.conf i use for poudriere in a jail
cat jail.conf
Code: 
# Common configs for all jails
path = "/jails/$name";
host.hostname = "$name";
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
persist;
ip4 = inherit;
ip6 = inherit;
mount.devfs;
mount.fdescfs;
allow.mount;
allow.mount.devfs;
allow.mount.fdescfs;
allow.mount.nullfs;
allow.mount.tmpfs;
allow.mount.procfs;
allow.mount.zfs;
enforce_statfs=1;
children.max=100;
allow.socket_af;
allow.raw_sockets;
allow.chflags;
allow.sysvipc;
a {
devfs_ruleset="20";
}

/etc/devfs.rules
Code: 
[a=20]
add include $devfsrules_hide_all 
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path 'mixer*' unhide
add path 'dsp*' unhide
add path 'zfs*' unhide
Click to expand...
Thank you, yep I know it 's what describes the handbook somehow and it reminds me this blog article:
Note that I am not against some hand work but for now I try various wrapper just to see how it fits me, I 'll probably go back to the hard way at some point just to see if I can do it in long run, still just for fun haha :)
 
Lamia said:
It would be beneficial to all if one of more members could share their experiences (or pieces of experience) on the two contending clustering suite for *BSD.
Click to expand...
This is a great idea. In two months, I invite you all to a party :beer: - we will celebrate the tenth anniversary of the CBSD project..., my ten-year-indie-project ?.
I plan to write an article about what has been done during this time and where the project is heading. Including an article on building clusters based on FreeBSD+CBSD (yes they exist :sssh:)

This thread is about jail containers, so a small demo about the possibilities of CBSD in terms of jail clustering: Clustering with CBSD: pool-binded HA/DRS FreeBSD jails ;

As for that link, I'll save it for the future "How to compare vm-bhyve, iohyve, chyves, cbsd" thread in this forum. ¯\_(ツ)_/¯
 
Here are my 2 cents from the brief stints I spent with each:
Manual /etc/jail.conf:
  • Simple and easy to implement due to being able to consult the man pages directly.
  • Great if you only maintain 1-3 jails, but can get ugly fast the more jails you have to maintain.
Bastille (my current choice):
  • Very easy setup and makes it super easy to spin up "containers".
  • ZFS support.
  • Supports templates, this is really nice if you want to setup multiple similar jails fast.
  • Pretty good documentation on their website with examples.
  • I have some puzzling throughput issue with it. I posted about it here. I'm not sure if this is specific to the way Bastille sets stuff up or just an issue with VNEt jails in general.
iocage (only used this through TrueNAS):
  • Requires ZFS.
  • Fairly easy setup. Most commands are similar in the way it works to Bastille.
  • Supports NAT/port forwards through IPFW, but I'm not sure if this is setup automatically by iocage itself or by TrueNAS (never dug into that).
  • Not very actively maintained.
cbsd: I didn't really use this cause I thought it was so complicated, it was beyond my puny brain understanding. Seems to have the best feature set among them all, including support for various types of hypervisors.
 
Whattteva said:
cbsd: I didn't really use this cause I thought it was so complicated, it was beyond my puny brain understanding. Seems to have the best feature set among them all, including support for various types of hypervisors.
Click to expand...
Best feature indeed. I like! But sometimes, you want to get under the hood or do the plumbing yourself - e.g. change your car oil filter. By doing so, you begin to have more understanding of how things work. That is why PURE JAIL with none of the above framework has the best best feature. And many more people, including myself, use or prefer it.
 
I've tried Bastille last year when it was getting hyped and it lacks a lot of features. Also looking the issues page on github the maintainer doesn't seem to very active and the reported bugs take a bit of time to be fixed.

I've been using CBSD for about 6-7 years now and it's feature rich(everything you want and more) and when there's an issue Ole(very friendly and easy to communicate with) usually fixes it in hours. Thanks a mill, Ole!!!
 
I've been using CBSD for about 2-3+ years now and it's what I have been using for all my VM (it has everything you want and more as others said) I am not much of a jailer (I have a few jails for particular pkgs).... I'll second+ "when there's an issue Ole responds fairly quickly by opening an issue in the git repo 1-3 days max". I also like CBSD framework flow (I've had to edit some things manually in SQlite DB before) or copy/import manually ZFS from 1 HDD to another fairly easy to understand once you know where all the files are, what they do, and understand CBSD commands/steps to doing things which are extensive. It has the occasional small bug but there are plenty of methods to go around bugs you do find and it's pretty stable/updated frequently. I am running over 150+ VMs with CBSD and its rock solid. I will be using it as a cluster/node manager in the future (not much time to go deep into this feature set yet) . There are also sister options/addons like Reggae which I like.
 
I've been using CBSD for like 8 years now and I am very content with it.
The main points which I enjoy about it:
  • One tool to manage jails and VMs (and this is not just bhyve, also QEMU, XEN, ...)
  • Never had a single major issue with it and 95% of the minor issues I encountered were PEBKAC
  • Very easy to migrate jails & VMs between hosts
  • Allows to build clusters
  • It's essentially just a bunch of shell scripts with an SQLite database so if shit hits the fan you're not left with some obscure binary
  • A lot of advanced features I haven't even looked at yet
  • Ole is just awesome (great & fast support as well as quick in implementing feature requests)
The only downside to CBSD I can currently think of is the "lack" of "proper" documentation.
 
Last edited:
Relative to CBSD, people are divided into two categories:

1) 'CBSD so awesome', 'Best feature indeed', 'it's feature rich(everything you want and more)', 'it has everything you want and more';
2) 'Its too complicated'. (Then they go to the GitHub page of another project and register several 'Feature request');

But no jokes - I have been using CBSD for about 8 years. And during this time I saw quite a lot of waves of migrations to various new cool projects (qjail hype, iocage-sh hype, iocate-python hype. Now it's Bastille - they have a sufficiently powerful advertising department). Stability is the most distinctive side of the CBSD.

One of the disadvantages of CBSD is documentation. But its absence is felt only when you have rather complex and specific questions regarding to custom setup ( the flexibility of the project makes it very easy to integrate with different other projects as Consul, IPAM, and various kinds of ConfigManagement tools );
As for the basic things - this is the only project that you can start using without documentation at all - just type 'bsdconfig' - CBSD has its own submenu where you can create a virtual environment. In addition, many commands are self-documented and have a large number of examples.

In addition, the CBSD is used by cool guys ( Dirk-Willem van Gulik still using FreeBSD ;- ) ☝️

Therefore, I congratulate Ole on the decade of the project and wish further development.
 
I'm still using sysutils/iocell (the "original", shell-based iocage) mainly for its simplicity and observability (it's just some shell scripts!). iocage blew up on me several times after they switched to python, and with python just barfing out its guts on the screen instead of giving proper errors I was fed up with it pretty fast (i.e. after it nuked all jails on the second host) and switched back to the 'just working'™ shell variant.
Yes, it's not "actively maintained" in that it receives updates every few days - but its feature complete (hence: why constantly change/update?) and at least for what I need/use it for it is more than sufficient and is easily scriptable/extensible. And the most important part (as already said): it 'just works'™
 
You must log in or register to reply here.
Back
Top