jails Mounting nullfs in jails causes empty directories for parent mount

danbanta

danbanta

Good morning, evening, or afternoon.

I currently treat my jails as stateless. I do this by mounting host zfs(8) datasets into the jail(8) using nullfs(5) to store state. Sometimes this leaves the data directory on the host empty although the data is present. Why do I see this behavior? What am I doing wrong? Or what do I not understand completely? Thank you. :)

/etc/jail.conf
Code: 
###############################################################################
# Global Jail Configuration
###############################################################################
host.hostname       = "${name}.example.info";
path                = "/jails/jails/${name}/root";
devfs_ruleset       = "4";
exec.clean;
exec.system_user    = "root";
exec.jail_user      = "root";
exec.consolelog     = "/var/log/jail_${name}_console.log";
enforce_statfs      = "1";
allow.mount;
mount.devfs;
mount.fdescfs;
allow.raw_sockets;

# Jail Interfaces
vnet;

## Start
vnet.interface  = "ng0_${name}";         # vnet interface(s)
exec.prestart   += "jng bridge ${name} igb0";      # bridge interface(s)
exec.start      += "/bin/sh /etc/rc";

## Stop
exec.stop       = "/bin/sh /etc/rc.shutdown";
exec.poststop   += "jng shutdown ${name}";        # destroy interface(s)

## Timeouts
exec.timeout = "60"; # Total time (seconds) of shutdown process of a jail
stop.timeout = "30"; # Time (seconds) before forcibly terminating the jail

# Mount fstab for jails
mount.fstab = "/etc/jails/${name}.fstab";

###############################################################################
# Jails
###############################################################################
# syslog-ng server
logs {
    # jid = "10";
}

# Database Server (PgSQL)
db {
    # jid = "21";
    depend="logs";
    sysvmsg=new;
    sysvsem=new;
    sysvshm=new;
}


/etc/jails/db.fstab
Code: 
/jails/releases/12.2-RELEASE/root/bin           /jails/jails/db/root/bin           nullfs ro 0 0
/jails/releases/12.2-RELEASE/root/boot          /jails/jails/db/root/boot          nullfs ro 0 0
/jails/releases/12.2-RELEASE/root/lib           /jails/jails/db/root/lib           nullfs ro 0 0
/jails/releases/12.2-RELEASE/root/libexec       /jails/jails/db/root/libexec       nullfs ro 0 0
/jails/releases/12.2-RELEASE/root/rescue        /jails/jails/db/root/rescue        nullfs ro 0 0
/jails/releases/12.2-RELEASE/root/sbin          /jails/jails/db/root/sbin          nullfs ro 0 0
/jails/releases/12.2-RELEASE/root/usr/bin       /jails/jails/db/root/usr/bin       nullfs ro 0 0
/jails/releases/12.2-RELEASE/root/usr/include   /jails/jails/db/root/usr/include   nullfs ro 0 0
/jails/releases/12.2-RELEASE/root/usr/lib       /jails/jails/db/root/usr/lib       nullfs ro 0 0
/jails/releases/12.2-RELEASE/root/usr/libexec   /jails/jails/db/root/usr/libexec   nullfs ro 0 0
/jails/releases/12.2-RELEASE/root/usr/sbin      /jails/jails/db/root/usr/sbin      nullfs ro 0 0
/jails/releases/12.2-RELEASE/root/usr/share     /jails/jails/db/root/usr/share     nullfs ro 0 0
/jails/releases/12.2-RELEASE/root/usr/libdata   /jails/jails/db/root/usr/libdata   nullfs ro 0 0
/jails/releases/12.2-RELEASE/root/usr/lib32     /jails/jails/db/root/usr/lib32     nullfs ro 0 0
/jails/releases/12.2-RELEASE/root/usr/src       /jails/jails/db/root/usr/src       nullfs ro 0 0
/jails/releases/12.2-RELEASE/root/usr/tests     /jails/jails/db/root/usr/tests     nullfs ro 0 0
/db /jails/jails/db/root/db    nullfs  rw,late  0   0

tree -L 2 /db executed on host.
Code: 
/db
└── prod-pg12

tree -L 2 /jails/jails/db/root/db/ executed on host.
Code: 
/jails/jails/db/root/db/
└── prod-pg12
    ├── base
    ├── ca.cer
    ├── dhparams.pem
    ├── global
    ├── pg_commit_ts
    ├── pg_dynshmem
    ├── pg_hba.conf
    ├── pg_ident.conf
    ├── pg_logical
    ├── pg_multixact
    ├── pg_notify
    ├── pg_replslot
    ├── pg_serial
    ├── pg_snapshots
    ├── pg_stat
    ├── pg_stat_tmp
    ├── pg_subtrans
    ├── pg_tblspc
    ├── pg_twophase
    ├── PG_VERSION
    ├── pg_wal
    ├── pg_xact
    ├── postgresql.auto.conf
    ├── postgresql.conf
    ├── postmaster.opts
    ├── postmaster.pid
    ├── server.cer
    └── server.key

tree -L 2 /db/ executed in the jail, db.
Code: 
/db/
`-- prod-pg12
    |-- PG_VERSION
    |-- base
    |-- ca.cer
    |-- dhparams.pem
    |-- global
    |-- pg_commit_ts
    |-- pg_dynshmem
    |-- pg_hba.conf
    |-- pg_ident.conf
    |-- pg_logical
    |-- pg_multixact
    |-- pg_notify
    |-- pg_replslot
    |-- pg_serial
    |-- pg_snapshots
    |-- pg_stat
    |-- pg_stat_tmp
    |-- pg_subtrans
    |-- pg_tblspc
    |-- pg_twophase
    |-- pg_wal
    |-- pg_xact
    |-- postgresql.auto.conf
    |-- postgresql.conf
    |-- postmaster.opts
    |-- postmaster.pid
    |-- server.cer
    `-- server.key
 
You're not doing anything wrong, this behavior is actually by design, see also the jailed property in zfs(8).
After a dataset is attached to a jail and the jailed property is set, a
jailed file system cannot be mounted outside the jail, since the jail
administrator might have set the mount point to an unacceptable value.
Click to expand...
 
Thank you for your reply, ShelLuser.

?I am attaching the dataset to the jail using nullfs. Also, I am not setting the jailed property on the dataset. I still think I am missing something.
 
It might be an order problem. Where the jail and the postgresql database is started before the /db filesystem gets a chance to mount. What happens if you remove that late option?
 
You must log in or register to reply here.
Back
Top