Solved Issue with hierarchical jails

[cmoerz]

I'm attempting to launch a jail in a jail on FreeBSD 13.2-p1. In principle, such a hierarchical jail should be working - as far as I understand, poudriere is making use of that. So I suppose I must have f*ed up my config somehow, but I can't seem to find the issue after poking at it for several hours.

Here's the jail.conf for the "parent" host:

firewall {
  vnet;
  vnet.interface  = "epair0b";
     children.max = 20;
     allow.mount;
     allow.mount.devfs;
     allow.mount.procfs;
     allow.mount.linprocfs;
     allow.mount.zfs;
     allow.mount.nullfs;
     allow.mount.tmpfs;
     allow.raw_sockets;
     allow.socket_af;
     allow.sysvipc;
     allow.chflags;
     enforce_statfs=1;

allow.socket_af;
allow.sysvipc;
  children.max = 32;

  enforce_statfs = 1;

  path = "/lab/runtime/firewall";
  host.hostname = "firewall";
  mount.devfs;

  mount.fstab "/lab/runtime/firewall.fstab";

  exec.prestart = "/sbin/ifconfig epair0 create up";
  exec.prestart += "/sbin/ifconfig epair0a name firewall0 up";
  exec.prestart += "/sbin/ifconfig switch0 addm firewall0 up";

  # Commands to run in jail after it is created
  exec.start  = "/sbin/ifconfig lo0 127.0.0.1 up";
  exec.start  += "/sbin/ifconfig epair0b 192.168.11.135 netmask 255.255.255.224 up";
  exec.start  += "/sbin/route add default 192.168.11.158";
  exec.start  += "/bin/sh /etc/rc";

  exec.stop  = "/bin/sh /etc/rc.shutdown";
  exec.poststop  = "ifconfig switch0 deletem firewall0";
  exec.poststop  += "ifconfig firewall0 destroy";

  #securelevel=1;

  persist;
}

The fstab file consists of multiple nullfs mounts that pull everything together. The child jail is also mounted directly into the jails filesystem space.

I noticed, that the sysctls for children.max are zero within the jail - not sure whether that is indicate of anything:

root@firewall:/ # sysctl -a |grep children
security.jail.param.children.max: 0
security.jail.param.children.cur: 0

Finally, trying to start a sub jail "dns" (configured in main jail's /etc/jail.conf) within that firewall jail gives:

root@firewall:/ # service jail start dns
Starting jails: cannot start jail  "dns":

Feels like the children.max parameter in the main jail is ignored, but for the life of me, I can't figure out why. If anyone has any troubleshooting suggestions, I'd very much appreciate any inputs.

Thanks.

 

[cmoerz]

Ok, one step further. I might be facing a configuration issue.

Following command actually now works within the main jail:
root@firewall:/ # jail -c name=foo host.hostname=foo allow.raw_sockets persist

Adding children.max to the jail call breaks it, though I could live with that.

Now to figure out what's broken with this dns jail:

root@firewall:/ # cat /etc/jail.conf.d/dns.conf
dns {
    vnet;
    vnet.interface = "epair1b";
    devfs_ruleset = "5";
    path = "/system/lab/dns";
    host.hostname = "dns";

    exec.prestart = "/sbin/ifconfig epair1 create up";
    exec.prestart += "/sbin/ifconfig epair1a name dns0 up";
    exec.prestart += "/sbin/ifconfig epair1b 192.168.10.66 netmask 255.255.255.224 up";
    exec.prestart += "/sbin/route add default 192.168.10.65";
    exec.start += "/sbin/sh /etc/rc";

    exec.stop = "/sbin/sh /etc/rc.shutdown";
    exec.poststop = "/sbin/ifconfig epair1b destroy";
    persist;
}

 

[cmoerz]

Yup, most certainly something's fishy with dns.conf:

+ eval 'rc_flags=${jail_dns_flags:-}'
+ rc_flags=''
+ eval 'command=${jail_dns_program:-/usr/sbin/jail}'
+ command=/usr/sbin/jail
+ command_args='-i -f /etc/jail.conf.d/dns.conf -c dns'
+ mktemp -t jail
+ _tmp=/tmp/jail.GpOodptH
+ /usr/sbin/jail -i -f /etc/jail.conf.d/dns.conf -c dns
+ echo ' cannot start jail ' '"dns": '
 cannot start jail  "dns":

 

[cmoerz]

Ah, crap. /sbin/sh. Yup - that's a facepalm.

 

[Alex Seitsinger]

allow.sysvipc;

enforce_statfs=1;

Allowing sysvipc might be a mistake. I would suggest not enabling this.

Both of these are included twice.

allow.socket_af;

You may not need to include this if you need to ping things.

vnet.interface = "epair0b";

Using this option may prevent the jail from starting successfull and lead to a kernel panic and reboot. You should manually add the epair after the jail starts using exec.created, instead.

 

[cmoerz]

Out of curiosity: I used vnet.interface cause I found that in various documentation artifacts. Is this a bug that's causing those issues you mention?

I've been using this for a while with other jails without any issues. How are you even handing over an interface into the jail manually? I assume something with ifconfig?

 

[Alex Seitsinger]

Is this a bug that's causing those issues you mention?

Yes, I believe so. Something to do with only one of the epair interfaces lingering after the jail disappears.

How are you even handing over an interface into the jail manually? I assume something with ifconfig?

Once the epair interfaces are created from exec.created they can be added to the jail with /sbin/ifconfig <name> vnet <jid>.