jails issues starting multiple vnet jails

R

rsx11m

hey all,

i am having an issue on a dedicated host running 13.2 where only one vnet jail is starting at boot. the contents of each individual jail config are generally identical, and these are the first two jails that are being started, in the order they are being started (i have confirmed this happens with both parallel and sequential start):

Code: 
beastie {
    # STARTUP/LOGGING
    exec.start = "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";
    exec.consolelog = "/var/log/jail_console_${name}.log";

    # PERMISSIONS
    allow.raw_sockets;
    exec.clean;
    mount.devfs;
    devfs_ruleset = 5;

    # HOSTNAME/PATH
    host.hostname = "${name}";
    path = "/usr/local/jails/containers/${name}";

    # NETWORK
    vnet;
    vnet.interface = "e0b_${name}";
    exec.prestart += "jib addm ${name} ix1";
    exec.poststop += "jib destroy ${name}";
}

Code: 
cutler {
    # STARTUP/LOGGING
    exec.start = "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";
    exec.consolelog = "/var/log/jail_console_${name}.log";

    # PERMISSIONS
    allow.raw_sockets;
    exec.clean;
    mount.devfs;
    devfs_ruleset = 5;

    # HOSTNAME/PATH
    host.hostname = "${name}";
    path = "/usr/local/jails/containers/${name}";

    # NETWORK
    vnet;
    vnet.interface = "e0b_${name}";
    exec.prestart += "jib addm ${name} ix1";
    exec.poststop += "jib destroy ${name}";
}

when the host boots, only the beastie jail will start, i have been unable to get any others booting along with it. service jail restart will visually hang, but all jails will start, oddly enough:

Code: 
hush@warden:~ % sudo service jail restart
Password:
Stopping jails: beastie.
Starting jails: r5rmain seymour www^C
hush@warden:~ % jls
   JID  IP Address      Hostname                      Path
     2                  beastie                       /usr/local/jails/containers/beastie
     3                  seymour                       /usr/local/jails/containers/seymour
     4                  cutler                        /usr/local/jails/containers/cutler
     5                  www                           /usr/local/jails/containers/www
     6                  hlm                           /usr/local/jails/containers/hlm
     7                  hakdog                        /usr/local/jails/containers/hakdog
     8                  r5rmain                       /usr/local/jails/containers/r5rmain

is something in the above configs obviously wrong? am i using jib incorrectly?

apologies if this comes off a little braindumpy, been scratching my head over this for a few hours now... if there is anything more i can provide or any clarifications i can make please do let me know :)
 
Where you able to figure it out?

I had the same issue on a FreeBSD VM using a VirtIO Ethernet device. (vtnet0)

Might be a timing issue where the networking interface is not fully initialized before the jails try to attach to it.
Or the bridge is busy and not able to bring up more vnet interfaces.

I solved it by adding a exec.prestart += "sleep 5"; in the jails config, not the most elegant solution but it got the job done.


Here is an example:
Code: 
# /etc/jail.conf

    exec.start = "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";
    exec.consolelog = "/var/log/jail_console_${name}.log";

    exec.clean;
    mount.devfs;

    host.hostname = "${name}";
    path = "/usr/local/jails/containers/${name}";

    interface = vtnet0;

jail1 {
    vnet;
    vnet.interface="e0b_$name";
    exec.prestart += "sleep 5";
    exec.prestart+="jib addm $name vtnet0";
    exec.poststop+="jib destroy $name";
}

jail2 {
    vnet;
    vnet.interface="e0b_$name";
    exec.prestart += "sleep 7";
    exec.prestart+="jib addm $name vtnet0";
    exec.poststop+="jib destroy $name";
}

jail3 {
    vnet;
    vnet.interface = "e0b_$name";
    exec.prestart += "sleep 9";
    exec.prestart += "jib addm $name vtnet0";
    exec.poststop += "jib destroy $name";
}
 
Does the logs or output say anything interesting? Maybe the error can be found in the log or output, so we can find out what the solution is.
 
By default, the error messages related to this are only visible in the console.
Running the command sudo cat /var/log/* | grep BRDGADD returns nothing.

I had to enable console.info in /etc/syslog.conf:
console.info /var/log/console.log
Also, ensure that the log file exists:
# touch /var/log/console.log

After making these changes, the error messages will appear both in the console and in the /var/log/console.log file.

Code: 
freebsd kernel: Starting cron.
freebsd kernel: Starting jails:
freebsd kernel: ifconfig: BRDGADD vtnet0: Device busy
freebsd kernel: jail: jail3: jib addm jail3 vtnet0: failed
freebsd kernel: vtnet0bridge
freebsd kernel: ifconfig: BRDGADD vtnet0: Device busy
freebsd kernel: jail: jail1: jib addm jail1 vtnet0: failed
freebsd kernel: e0a_jail2
freebsd kernel: e0b_jail2
freebsd kernel: jail2: created
 
Good point, thanks:)
(It was just a quick fix/test to see if it would solve the issue.)

Also, nice work with AppJail & LittleJet btw.


The following adjustments with lockf -k using the file /var/run/jib.lock works, tested multiple reboots.

Code: 
...

jail1 {
    vnet;
    vnet.interface="e0b_$name";
    exec.prestart += "lockf -k /var/run/jib.lock jib addm $name vtnet0";
    exec.poststop += "lockf -k /var/run/jib.lock jib destroy $name";
}

jail2 {
    vnet;
    vnet.interface="e0b_$name";
    exec.prestart += "lockf -k /var/run/jib.lock jib addm $name vtnet0";
    exec.poststop += "lockf -k /var/run/jib.lock jib destroy $name";
}
...
 
You must log in or register to reply here.
Back
Top