Solved Jails vnet - FreeBSD Mastery - multiple interfaces

[tommiie]

Starting on p. 165, to create a jail with multiple interfaces, you need to do:

1. create a cloned interface (lo1),
2. specify multiple interfaces for the jail in a comma-separated list,
3. call the jib script with the bridge names,
4. be happy.

Step 1: /etc/rc.conf:

ifconfig_em0_name="lab"
cloned_interfaces="lo1"

Step 2: /etc/jail.conf:

gw {
   vnet;
   vnet.interface = e0b_$name, e1b_$name;
   exec.prestart += "/usr/local/scripts/jib addm $name lab lo1";
   exec.poststop += "/usr/local/scripts/jib destroy $name";
   allow.raw_sockets;
}

Step 3: service jail start gw which calls the script jib:

# service jail start gw
Starting jails: cannot start jail  "gw":
ifconfig: BRDGADD lo1: Invalid argument
jail: gw: /usr/local/scripts/jib addm gw lab lo1: failed
.
#

 

[tommiie]

So according to this link (from 2016) it is not supposed to work with loopback interfaces. Perhaps it does work in 13.0-CURRENT (as that is the release that Lucas uses) but not in 12.0-RELEASE which I use.

Furthermore, the jib script creates a bridge and attaches the physical interface to it. You must supply an interface (hence creating a new loopback interface as we are out of real interfaces) for the script to work and create the bridge.

Alas, I do not really understand the über shell scripting skills involved so I started reading man pages and tinkering with ifconfig commands and once I got something working, I wrote this Python script to use instead of the jib script.

Feedback is welcome as are improvements on the script.

 

[D-FENS]

Why at all rely on those intransparent tools. The jib script itself is meant as an example, but there's nothing you couldn't do yourself?
What you actually need is just a way to create a bunch of interfaces, add them to a bridge and vnet them into the jail at startup. You don't need vnet.interface, you don't need jib, etc.
I tried to comprehend all this and I could not wrap my head around it. vnet.interface from my experience supports only one interface (although the book suggests otherwise but I have never succeeded in making it work).

This is what I do:
- use vnet;
- leave vnet.interface empty.
- create scripts for exec.prestart, exec.created, exec.poststart, exec.prestop, exec.poststop.

In the scripts I do the following (approximately, make sure to create interfaces only when needed, otherwise reuse existing ones):

# jail.conf
exec.created = "..../exec.created.sh $name $path"

# exec.created.sh
jail="$1"
jailRoot="$2"

bridge="$(ifconfig bridge create)"
ifconfig $bridge 10.0.0.1/24 up

epairA="$(ifconfig epair create)"
epairB="${epairA%?}b"

ifconfig $epairA up
ifconfig $epairB up

ifconfig $bridge addm $epairA up

ifconfig $epairB vnet $jail up
jexec $jail ifconfig $epairB 10.0.0.2/24 up

jexec $jail route add default 10.0.0.1

Of course, you could throw in some nice names in the mix (that's what I do) etc. Don't bother with the jail.conf, just do whatever you need to do inside the hook scripts.
This works beautifully.

On stopping things are similar:

# exec.prestop.sh
# Before stopping the jail we need to cleanup its VNET interfaces. Otherwise there is a bug in VNET and on occasions the whole kernel panics and the host crashes! If we cleanup though everything is fine.
ifconfig $epairA -vnet $jail || true
ifconfig $epairA destroy || true
ifconfig $epairB destroy || true

Other things is managing the mounts. I actually use an fstab file for the jails, but again you could mount and unmount in the hook scripts. Unmounting does not work very cleanly in my jails, so I go umount in poststop to make sure everything unmounts cleanly.

Put some sensible names on the network interfaces and rename them after creation, so you can always identify a jail's interfaces easily.

 

[tommiie]

Your bash script is essentially the same as my python script. The vnet.interface parameter works beautifully but indeed this could be solved as well by the script. But why use the script when this parameter is so much cleaner?

 

[D-FENS]

I don't think it's cleaner. If it is so clean, why do you need to use the jib script at all? Could jail not simply create a VNET interface with this name on the fly as needed?
The script I showed above implements the same functions like the jib script, but it's better. You keep the control and everything is transparent. Any kind of network setup could be implemented.

 

[D-FENS]

Your bash script is essentially the same as my python script.

Sure. However, sh is installed by default and Python is a dependency. And quite a fat one.
My example could run anywhere, no need for anything except the system base.
Python is not necessary for such a simple task.

Also, why the need to implement a python wrapper for jail? Just use iocage. It's implemented in Python.

 

[tommiie]

I don't think it's cleaner. If it is so clean, why do you need to use the jib script at all?

vnet.interfaces

attaches existing interfaces to the jail in a clean way. Perhaps in a future version of BSD it can be made to also create the interfaces should they not exist. But how will it know that you want an epair instead of something else? And where to connect the other end then? You don't like it, fine. Don't use it.

sh is installed by default and Python is a dependency. And quite a fat one.

Perhaps for you it is a problem that "Python is a dependency". For me it is not. With that logic you can't install any software (nginx, BIND...) because they all are dependencies. It being a "fat" dependency is also a non-argument for me. I consider my computer to be fast enough to run Python without any issues. On top of that, I know how to program in Python and my bash programming skills are nearly non-existent. It was an easy call to make: use the tool you know best.

Just use iocage. It's implemented in Python.

I just started learning about jails last week from Lucas' bookk. His examples did not work so I had to investigate how to make it work. He covers iocage in his book and I will look into that later. First I want to understand how jails, bridges, and epairs work and how they interconnect. Then I can start learning about some abstraction layer, be it iocage, ezjail, or something else.

 

[tommiie]

Also, why the need to implement a python wrapper for jail? Just use iocage. It's implemented in Python.

You answer your own question nicely in a different thread ;-) I added the emphasis myself.

I don't use those automation tools (although they might do a great job, I prefer keeping the control over the network interfaces). If you understand how an epair works, then you can go ahead an use the scripts like jib or jng because that's exactly what they do. But for learning, better do it manually and see how things work.

 

[D-FENS]

ifconfig <interface> vnet <jail> attaches the interface to the jail just as cleanly as "vnet.interface". Unless you have a reason to believe that ifconfig is buggy?
At the end of the day, when you start the jail service it does ifconfig itself. So a wrapper script of something cannot be cleaner than the thing itself.

Regarding Python: iocage is a Python app that facilitates jail management. So you are more or less trying to reimplement iocage.

If you want to learn - sure, be my guest. I am not telling you to NOT implement a Python jail wrapper. Just do it because you want or need to. But iocage is already there and it works.
For me - it does not do everything I need, so I cannot use it. But if you are keen on doing more stuff on Python, you could just fork iocage and modify it.

 

[tommiie]

"vnet.interfaces" is clean in a way that it is one line in the central file containing all jail configuration data (i.e. /etc/jail.conf). I never stated or hinted towards ifconfig being buggy. I just consider one line in the central configuration file to be cleaner than to write custom scripts.

I'm not trying to reimplement iocage. I just wrote the same script you wrote except that I used Python instead of shell scripting. Are you trying to reimplement iocage in shell scripting?

When I read your posts I feel like you're attacking me personally. I hope that is not your intend. My original post was: Hey I bought this book and I'm trying to follow along with the examples provided by the author but they do not work. Help me fix it. Don't attack me for making different choices along the way than you made when you started using jails.

 

[D-FENS]

Are you trying to reimplement iocage in shell scripting?

Yes I have.

 

[m4r1n]

[...]
I tried to comprehend all this and I could not wrap my head around it. vnet.interface from my experience supports only one interface (although the book suggests otherwise but I have never succeeded in making it work).
[...]

It is possible to assign several vnet interfaces to a single jail by using the += operator (tested on 12-STABLE):

vnet.interface += "epair0b";
vnet.interface += "epair1b";