Solved Nested jail won't run automatically (but starts manually)

alfikmik

alfikmik

system: FreeBSD 13.0-RELEASE

HOST:
Code: 
cat /etc/jail.conf
# jail.conf
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;

allow.noset_hostname;
allow.raw_sockets;
allow.sysvipc;
allow.dying;

path = "/vol/jls/${name}";
host.hostname = "${name}.bsd";

parent_jail {
    jid = 50;
    mount.devfs;
    devfs_ruleset=5;
    vnet;
    vnet.interface="v${jid}";
    exec.consolelog = "/var/log/jail_${name}.log";
    exec.prestart += "ifconfig epair0 create up";
    exec.prestart += "ifconfig epair0a inet 10.20.20.254 netmask 0xffffff00";
    exec.prestart += "ifconfig epair0a name j${jid}";
    exec.created   = "ifconfig epair0b name v${jid}";
    exec.poststop += "sleep 4";
    exec.poststop += "ifconfig v${jid} -vnet $name";
    exec.poststop += "ifconfig j${jid} destroy";
        
    allow.mount;
    allow.mount.nullfs;
    allow.mount.devfs;
    allow.mount.tmpfs;
    enforce_statfs=1;
    
    allow.dying;
    allow.chflags;
    allow.socket_af;
    children.max=20;
}

CHILD (rc.conf):
Code: 
ifconfig_v50="inet 10.20.20.1 netmask 0xffffff00"
defaultrouter="10.20.20.254"
dumpdev="NO"
clear_tmp_enable="YES"
update_motd="no"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
sendmail_enable="NONE"
jail_enable="YES"
cron_enable="NO"
named_enable="NO"

CHILD (jail.conf):
Code: 
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
allow.noset_hostname;
allow.raw_sockets;
allow.sysvipc;
allow.dying;
path = "/vol/jls/${name}";
host.hostname = "v50-${name}.bsd";

nested_jail {
    jid = 51;
    interface = v50;
    ip4.addr = "v50|10.20.20.10/24";
    mount.devfs;
    devfs_ruleset=5;
    exec.consolelog = "/var/log/jail_${name}.log";
}

when i run from host service jail start parent_jail parent_jail starts but no nested_jail, when i log into parent_jail ( jexec 50 login -f root) and do service jail start nestad_jail is created

with some investignation i found that service -e doesn't show "jail":
Code: 
# service -e
/etc/rc.d/cleanvar
/etc/rc.d/ip6addrctl
/etc/rc.d/netif
/etc/rc.d/virecover
/etc/rc.d/newsyslog
/etc/rc.d/os-release
/etc/rc.d/syslogd

it appears that some services depends on security.jail.jailed wchich in jailed enviroment is equal to 1

any ideas? what am i doing wrong?
 
Your rc.conf should have the names of jails you want to start on boot, at least for the parent jail. Such might he required for the nested jail to.
 
Lamia said:
Your rc.conf should have the names of jails you want to start on boot, at least for the parent jail. Such might he required for the nested jail to.
Click to expand...

Nope, this is wrong way

Code: 
jail_enable="YES"
jail_conf="/etc/jail.conf"
jail_parallel_start="NO"
jail_list="nestedjail"

that doesn't helped, for all configurations that i do before, empty jail_list means run all jails


as i mentioned before service -e in host shows jail as enabled service, but in nested jail doesn't, and it seems that have to be correlated with security.jail.jailed: 1

Code: 
grep -C 5 "jailed" /usr/sbin/service
        exit $?
fi

if [ -n "$RESTART" ]; then
        skip="-s nostart"
        if [ `/sbin/sysctl -n security.jail.jailed` -eq 1 ]; then
        skip="$skip -s nojail"
        fi
        [ -n "$local_startup" ] && find_local_scripts_new
        files=`rcorder ${skip} ${local_rc} 2>/dev/null`

--
fi

if [ -n "$ENABLED" -o -n "$RCORDER" ]; then
        # Copied from /etc/rc
        skip="-s nostart"
        if [ `/sbin/sysctl -n security.jail.jailed` -eq 1 ]; then
                skip="$skip -s nojail"
        fi
        [ -n "$local_startup" ] && find_local_scripts_new
        files=`rcorder ${skip} /etc/rc.d/* ${local_rc} 2>/dev/null`
fi

Code: 
grep -C 5 "jailed" /etc/rc
# scripts to perform "boot-time configuration" including enabling and
# disabling rc.d scripts which appear later in the boot order.
trap "_rc_conf_loaded=false; load_rc_config" ALRM

skip="-s nostart"
if [ `/sbin/sysctl -n security.jail.jailed` -eq 1 ]; then
        skip="$skip -s nojail"
        if [ `/sbin/sysctl -n security.jail.vnet` -ne 1 ]; then
                skip="$skip -s nojailvnet"
        fi
fi

Code: 
grep -C 5 "jailed" /etc/rc.shutdown

# Determine the shutdown order of the /etc/rc.d scripts,
# and perform the operation
#
rcorder_opts="-k shutdown"
if [ `/sbin/sysctl -n security.jail.jailed` -eq 1 ]; then
        rcorder_opts="$rcorder_opts -s nojail"
        if [ `/sbin/sysctl -n security.jail.vnet` -ne 1 ]; then
                rcorder_opts="$rcorder_opts -s nojailvnet"
        fi
fi
 
For now working solution is:

1. disable nested jail in rc.conf (in parent jail - jail_enable="NO")

2. in host add to /etc/jail.conf:

Code: 
exec.poststart="jexec parentjail service jail onestart";
exec.prestop="jexec parentjail service jail onestop";

but I think there have to be a better way
 
Unfortunately, it looks like this is "by design". /etc/rc.d/jail contains the following metadata block:
Code: 
# PROVIDE: jail
# REQUIRE: LOGIN FILESYSTEMS
# BEFORE: securelevel
# KEYWORD: nojail shutdown

The keyword nojail means rc skips this script when running inside a jail.
 
Zirias said:
Unfortunately, it looks like this is "by design". /etc/rc.d/jail contains the following metadata block:
Click to expand...

So I'm curious why it is not well documented, and Hierarchical Jails section in jail(8) doesn't mention about any limitations or special way to run nested jails
 
alfikmik said:
So i'm curious why it is not well documented, and Hierarchical Jails section in jail(8) doesn't mention about any limitations or special way to run nested jails
Click to expand...
This manpage doesn't touch the init script at all. The nojail keyword is documented in rc(8) (but I admit, you kind of have to know that to find it...).

I guess it makes some sense as nested jails are typically started ad-hoc (one usecase is e.g. a jailed poudriere). But in your case, you could also decide to just remove this keyword. etcupdate(8) should make merging that change on updates a piece of cake.
 
this is very interesting:

https://github.com/freebsd/freebsd-src/blame/main/libexec/rc/rc.d/jail

1637588120244.png
 
Ah, so someone changed their minds ;) Well then, just remove the keyword, and with 14, it will be gone anyways.
 
I remember Dan Langille posting something on twitter regarding jails in jails. Can't find it anymore though. He was running into the same issue, those "sub" jails don't want to start automatically.
 
ok so :) changes is more:

Diff: 
diff /etc/rc.d/jail jail
9c9
< # KEYWORD: nojail shutdown
---
> # KEYWORD: shutdown
122a123
>       eval _jconfdir=\"/etc/jail.conf.d/${_j}.conf\"
127a129,131
>                       return 0
>               elif [ -r "$_jconfdir" ]; then
>                       _conf="$_jconfdir"

but in my case after remove nojail from /etc/rc.d/jail everything works like a charm
 
Alain De Vos said:
Jails within jails is speciality.
Danny Langville & Michael W Lucas could elaborate ?
Click to expand...

Yup, MWL do the job, I have all his books including "FreeBSD Mastery: Jails" where he wrote some about hierarchical jails, also freebsd docs have some info, but ... problem that is solved here isn't described anywhere.

also problem with hanging jails (often with system crash) in some cases when we use VNET (source here), is not solved and not described in books or docs i mean hack with sleep:

Code: 
exec.poststop += "sleep 4";
exec.poststop += "ifconfig v${jid} -vnet $name";
exec.poststop += "ifconfig j${jid} destroy";
 
You must log in or register to reply here.
Back
Top