jails files and directories invisible from jail

[keldonin]

Hello,

I'm encountering an issue that I can't explain, so looking for SME advice here

I run a linux (debian) inside a jail. I'm using iocage for jail management. deboostrap was used to deploy the guest system. Everything seems to work well except that many files from /etc directory are not visible/accessible from the jail, when using linux ls. If I however use BSD rescue ls (grabbed from host), I can list all files and directories.

# iocage console cell3
Last login: Wed Jun  9 13:21:17 UTC 2021 on pts/1
Linux cell3 3.17.0 FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f: Fri Apr x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@cell3:~# ls -l /etc
total 43K
-rw-r--r-- 1 root root     115 Jun  9 11:30 hosts
-rw-r--r-- 1 root root   29546 Jun  9 13:36 ld.so.cache
-rw-r--r-- 1 root root    1337 Jun  9 11:42 passwd
-rw-r--r-- 1 root root    1337 Jun  9 11:42 passwd-
-rwxr-xr-x 1 root root      63 Jun  9 11:57 rc
drwxr-xr-x 2 root root       4 Jun  9 11:42 rc2.d
drwxr-xr-x 2 root root       4 Jun  9 11:42 rc3.d
drwxr-xr-x 2 root root       4 Jun  9 11:42 rc4.d
drwxr-xr-x 2 root root       4 Jun  9 11:42 rc5.d
-rw-r----- 1 root shadow   809 Jun  9 11:42 shadow
-rw-r----- 1 root shadow   809 Jun  9 11:42 shadow-
drwxr-xr-x 2 root root       9 Jun  9 11:47 ssh
-r--r----- 1 root root       0 Jun  9 11:30 sudoers
drwxr-xr-x 3 root root       3 Jun  9 11:42 ufw
root@cell3:~# rescue ls -lo /etc
total 493
drwxr-xr-x  3 0  root    uarch     3 Jun  8 20:36 .java
-rw-------  1 0  root    uarch     0 Jun  8 13:29 .pwd.lock
drwxr-xr-x  7 0  root    uarch    11 Jun  8 20:36 X11
-rw-r--r--  1 0  root    uarch  2981 Jun  8 13:29 adduser.conf
-rw-r--r--  1 0  root    uarch   185 Jun  8 20:16 aliases
drwxr-xr-x  2 0  root    uarch   137 Jun  9 11:07 alternatives
drwxr-xr-x  3 0  root    uarch     4 Jun  8 20:08 apparmor.d
drwxr-xr-x  7 0  root    uarch     9 Jun  9 11:27 apt
-rw-r--r--  1 0  root    uarch  1994 Apr 18  2019 bash.bashrc
drwxr-xr-x  2 0  root    uarch     3 Jun  8 20:30 bash_completion.d
-rw-r--r--  1 0  root    uarch   367 Mar  2  2018 bindresvport.blacklist
drwxr-xr-x  2 0  root    uarch     2 Mar 18 19:59 binfmt.d
drwxr-xr-x  3 0  root    uarch     3 Jun  8 13:25 ca-certificates
-rw-r--r--  1 0  root    uarch  5989 Jun  8 13:29 ca-certificates.conf
drwxr-xr-x  2 0  root    uarch     3 Jun  8 13:29 calendar
drwxr-xr-x  2 0  root    uarch     3 Jun  8 13:29 cron.d
drwxr-xr-x  2 0  root    uarch    11 Jun  8 20:16 cron.daily
drwxr-xr-x  2 0  root    uarch     3 Jun  8 13:29 cron.hourly
drwxr-xr-x  2 0  root    uarch     3 Jun  8 13:29 cron.monthly
drwxr-xr-x  2 0  root    uarch     4 Jun  8 20:08 cron.weekly
-rw-r--r--  1 0  root    uarch  1042 Oct 11  2019 crontab
drwxr-xr-x  4 0  root    uarch     4 Jun  8 20:14 dbus-1
-rw-r--r--  1 0  root    uarch  2969 Feb 26  2019 debconf.conf
-rw-r--r--  1 0  root    uarch     5 Mar 19 23:44 debian_version
drwxr-xr-x  2 0  root    uarch    13 Jun  9 11:47 default
-rw-r--r--  1 0  root    uarch   604 Jun 26  2016 deluser.conf
drwxr-xr-x  4 0  root    uarch     6 Jun  8 13:29 dhcp
drwxr-xr-x  4 0  root    uarch     5 Jun  8 13:29 dpkg
drwxr-xr-x  3 0  root    uarch     3 Jun  8 20:14 emacs
-rw-r--r--  1 0  root    uarch   312 Mar 18 08:10 email-addresses
-rw-r--r--  1 0  root    uarch     0 Jun  8 13:29 environment
drwxr-xr-x  2 0  root    uarch     3 Jun  8 20:36 environment.d
drwxr-xr-x  3 0  root    uarch     6 Jun  8 20:16 exim4
drwxr-xr-x  4 0  root    uarch     5 Jun  8 20:36 fonts
-rw-r--r--  1 0  root    uarch    37 Jun  8 12:24 fstab
-rw-r--r--  1 0  root    uarch  2584 Aug  1  2018 gai.conf
drwxr-xr-x  2 0  root    uarch     4 Jun  8 20:08 groff
-rw-r--r--  1 0  root    uarch   654 Jun  8 20:30 group
-rw-r--r--  1 0  root    uarch   643 Jun  8 20:16 group-
-rw-r-----  1 0  shadow  uarch   546 Jun  8 20:30 gshadow
-rw-r-----  1 0  shadow  uarch   538 Jun  8 20:16 gshadow-
drwxr-xr-x  3 0  root    uarch     3 Jun  8 20:15 gss
drwxr-xr-x  2 0  root    uarch     3 Jun  8 20:36 gtk-2.0
drwxr-xr-x  2 0  root    uarch     3 Jun  8 20:36 gtk-3.0
-rw-r--r--  1 0  root    uarch     9 Aug  7  2006 host.conf
-rw-r--r--  1 0  root    uarch   109 Jun  8 13:29 hosts
-rw-r--r--  1 0  root    uarch   411 Jun  8 20:16 hosts.allow
-rw-r--r--  1 0  root    uarch   711 Jun  8 20:16 hosts.deny
drwxr-xr-x  2 0  root    uarch    14 Jun  9 11:47 init.d
-rw-r--r--  1 0  root    uarch  1748 May  5  2018 inputrc
drwxr-xr-x  4 0  root    uarch    13 Jun  8 13:29 iproute2
-rw-r--r--  1 0  root    uarch    27 Mar 19 23:44 issue
-rw-r--r--  1 0  root    uarch    20 Mar 19 23:44 issue.net
drwxr-xr-x  5 0  root    uarch    13 Jun  8 20:36 java-11-openjdk
drwxr-xr-x  4 0  root    uarch     4 Jun  8 13:28 kernel
-rw-r--r--  1 0  root    uarch 26719 Jun  8 20:36 ld.so.cache
-rw-r--r--  1 0  root    uarch    34 Mar  2  2018 ld.so.conf
drwxr-xr-x  2 0  root    uarch     5 Jun  9 11:33 ld.so.conf.d
drwxr-xr-x  2 0  root    uarch     3 Jun  8 20:16 ldap
-rw-r--r--  1 0  root    uarch   191 Apr 25  2019 libaudit.conf
-r--r--r--  1 0  root    uarch   118 Jun  9 14:21 localtime
drwxr-xr-x  3 0  root    uarch     3 Jun  8 13:28 logcheck
-rw-r--r--  1 0  root    uarch 10477 Jul 27  2018 login.defs
-rw-r--r--  1 0  root    uarch   435 Aug 22  2018 logrotate.conf
drwxr-xr-x  2 0  root    uarch    11 Jun  8 20:16 logrotate.d
-r--r--r--  1 0  root    uarch    33 Jun  8 13:29 machine-id
-rw-r--r--  1 0  root    uarch   111 Jan 25 21:40 magic
-rw-r--r--  1 0  root    uarch   111 Jan 25 21:40 magic.mime
-rw-r--r--  1 0  root    uarch  3397 Jun  8 20:36 mailcap
-rw-r--r--  1 0  root    uarch   449 Feb  9  2019 mailcap.order
-rw-r--r--  1 0  root    uarch     6 Jun  8 20:16 mailname
-rw-r--r--  1 0  root    uarch  5174 Feb 10  2019 manpath.config
drwxr-xr-x  2 0  root    uarch    11 Jun  8 20:18 mc
-rw-r--r--  1 0  root    uarch 24512 Feb  9  2019 mime.types
-rw-r--r--  1 0  root    uarch   812 Jan 10  2020 mke2fs.conf
drwxr-xr-x  2 0  root    uarch     2 Feb  9  2019 modprobe.d
-rw-r--r--  1 0  root    uarch   195 Jun  8 13:29 modules
drwxr-xr-x  2 0  root    uarch     3 Jun  8 13:28 modules-load.d
-rw-r--r--  1 0  root    uarch   286 Mar 19 23:44 motd
drwxr-xr-x  4 0  root    uarch     7 Jun  8 20:16 mysql
-rw-r--r--  1 0  root    uarch  9278 Jun 12  2019 nanorc
drwxr-xr-x  7 0  root    uarch     8 Jun  8 13:29 network
-rw-r--r--  1 0  root    uarch    60 Jun  8 13:29 networks
-rw-r--r--  1 0  root    uarch   494 Feb 10  2019 nsswitch.conf
drwxr-xr-x  2 0  root    uarch     2 Jun  8 13:29 opt
lrwxr-xr-x  1 0  root    uarch    21 Mar 19 23:44 os-release -> ../usr/lib/os-release
-rw-r--r--  1 0  root    uarch   552 Feb 14  2019 pam.conf
drwxr-xr-x  2 0  root    uarch    22 Jun  9 11:47 pam.d
-rw-r--r--  1 0  root    uarch  1291 Jun  8 20:16 passwd
-rw-r--r--  1 0  root    uarch  1233 Jun  8 20:16 passwd-
drwxr-xr-x  4 0  root    uarch     4 Jun  8 20:03 perl
drwxr-xr-x  3 0  root    uarch     3 Jun  8 20:15 ppp
-rw-r--r--  1 0  root    uarch   767 Mar  4  2016 profile
drwxr-xr-x  2 0  root    uarch     2 Mar 19 23:44 profile.d
-rw-r--r--  1 0  root    uarch  2932 Feb 10  2019 protocols
drwxr-xr-x  2 0  root    uarch     3 Jun  8 20:16 python2.7
-rwxr-xr-x  1 0  root    uarch    62 Jun  9 11:59 rc
drwxr-xr-x  2 0  root    uarch     7 Jun  8 20:16 rc0.d
drwxr-xr-x  2 0  root    uarch     4 Jun  8 20:16 rc1.d
drwxr-xr-x  2 0  root    uarch     6 Jun  8 20:16 rc2.d
drwxr-xr-x  2 0  root    uarch     6 Jun  8 20:16 rc3.d
drwxr-xr-x  2 0  root    uarch     6 Jun  8 20:16 rc4.d
drwxr-xr-x  2 0  root    uarch     6 Jun  8 20:16 rc5.d
drwxr-xr-x  2 0  root    uarch     7 Jun  8 20:16 rc6.d
drwxr-xr-x  2 0  root    uarch     8 Jun  8 20:36 rcS.d
-rw-r--r--  1 0  root    uarch   101 Jun  9 14:21 resolv.conf
lrwxr-xr-x  1 0  root    uarch    13 Apr 23  2019 rmt -> /usr/sbin/rmt
-rw-r--r--  1 0  root    uarch   887 Feb 10  2019 rpc
-rw-r--r--  1 0  root    uarch  1988 Feb 26  2019 rsyslog.conf
drwxr-xr-x  2 0  root    uarch     2 Feb 26  2019 rsyslog.d
-rw-r--r--  1 0  root    uarch  4141 Jul 27  2018 securetty
drwxr-xr-x  4 0  root    uarch    13 Jun  8 13:29 security
drwxr-xr-x  2 0  root    uarch     3 Jun  8 13:29 selinux
drwxr-xr-x  2 0  root    uarch     3 Jun  8 20:36 sensors.d
-rw-r--r--  1 0  root    uarch 10593 Dec 19  2018 sensors3.conf
-rw-r--r--  1 0  root    uarch 18774 Feb 10  2019 services
-rw-r-----  1 0  shadow  uarch   783 Jun  8 20:16 shadow
-rw-r-----  1 0  shadow  uarch   783 Jun  8 20:16 shadow-
-rw-r--r--  1 0  root    uarch    73 Jun  8 13:29 shells
drwxr-xr-x  2 0  root    uarch     5 Jun  8 13:29 skel
drwxr-xr-x  2 0  root    uarch     4 Jun  9 11:42 ssh
drwxr-xr-x  4 0  root    uarch     5 Jun  8 13:29 ssl
-rw-r--r--  1 0  root    uarch     0 Jun  8 13:29 subgid
-rw-r--r--  1 0  root    uarch     0 Jun  8 13:29 subuid
-r--r-----  1 0  root    uarch   669 Jan 20 12:26 sudoers
drwxr-xr-x  2 0  root    uarch     3 Jun  9 11:28 sudoers.d
-rw-r--r--  1 0  root    uarch  2351 May 31  2018 sysctl.conf
drwxr-xr-x  2 0  root    uarch     5 Jun  8 13:29 sysctl.d
drwxr-xr-x  5 0  root    uarch    13 Jun  8 13:29 systemd
drwxr-xr-x  2 0  root    uarch     3 Jun  8 13:29 terminfo
drwxr-xr-x  2 0  root    uarch     2 Mar 18 19:59 tmpfiles.d
-rw-r--r--  1 0  root    uarch  1260 Dec 14  2018 ucf.conf
drwxr-xr-x  4 0  root    uarch     5 Jun  8 13:29 udev
drwxr-xr-x  2 0  root    uarch     3 Jun  8 13:29 update-motd.d
drwxr-xr-x  2 0  root    uarch     4 Jun  8 13:29 vim
-rw-r--r--  1 0  root    uarch   642 Mar  1  2019 xattr.conf
drwxr-xr-x  4 0  root    uarch     6 Jun  8 20:18 xdg

Of course, the issue goes beyond the simple execution of ls. It does affect any application dealing with filesystem, through the linux ABI compatibility layer.

The file system is ZFS.

Here is what mount yields from the jail:

root@cell3:~# mount
jail/iocage/jails/cell3/root on / type zfs (rw)
devfs on /dev type devfs (rw)
tmpfs on /dev/shm type tmpfs (rw)
fdescfs on /dev/fd type fdescfs (rw)
proc on /proc type proc (rw)
/sys on /sys type sysfs (rw)

Here is what mount says from the host, for that jail:

$ mount  | grep cell3
jail/iocage/jails/cell3 on /jail/iocage/jails/cell3 (zfs, local, nfsv4acls)
jail/iocage/jails/cell3/root on /jail/iocage/jails/cell3/root (zfs, local, nfsv4acls)
devfs on /jail/iocage/jails/cell3/root/dev (devfs)
tmpfs on /jail/iocage/jails/cell3/root/dev/shm (tmpfs, local)
fdescfs on /jail/iocage/jails/cell3/root/dev/fd (fdescfs)
linprocfs on /jail/iocage/jails/cell3/root/proc (linprocfs, local)
linsysfs on /jail/iocage/jails/cell3/root/sys (linsysfs, local)

• what have I missed?
• are there specific limitations in the way how the linux compatibility layer access files?
• are there tunables to adjust behaviour?
• should I implement another filesystem type instead (UFS?)?

Note: the host runs on 13.0-RELEASE.


Any help greatly appreciated.

 

[SirDice]

You're looking at two distinctly different /etc/ directories. Just look at the timestamps of the files they have in common, they don't match up either. And neither does the group ownership. So the obvious conclusion is that it shows different directories.

 

[DaBrooklyner]

• what have I missed?

A good way to think about FreeBSD jails is as a one-way mirror. From the jail host you can see activity in the jail and from the jail, you can only see the jail. As a consequence, your paths are going to be quite different from either vantage point. Pick one and stick with it, I usually choose the jail host because it makes my head hurt less.

• are there specific limitations in the way how the Linux compatibility layer access files?
  
  I use bhyve for anything that requires Linux, but the compatibility layer should work the same in the host and in the jail. In bhyve I use a very light distro, because all I want it to have is a user land and maybe run one application.
  
  
• are there tunables to adjust behaviour?
  
  Probably. There are a lot of tunables for jails using sysctl, but you can usually accomplish the same with iocage itself.
  
  
• should I implement another filesystem type instead (UFS?)?
  
  Iocage requires ZFS to operate, but with standard jails you can use either. Will switching to UFS solve your problem, nope. Could you run a zvol with UFS, maybe.
  
  There is a lot you can do with jails and with iocage, best to think about what problem you are trying to solve.
  
  
• Do you just want a Linux user land, you probably have one already.
  
  
• Are you looking to run an application that requires Linux.
  
  If it runs on the jail host It SHOULD run in the jail. Not saying it will run well without tuning. There are edge cases where it will not even run at all.
  
  Other edge cases involve firewalls and networking and all sorts of fun stuff that can keep you up at night. FreeBSD Mastery: Jails is probably what you want to look at because it covers both iocage and standard jails and shows how to solve a problem in each. It also covers a lot of the edge cases. It has been out for a bit, so look to the current docs from the foundation to update the book to your version of the OS.
  
  I have been running jails in production for years and one thing I have found is that FreeBSD jails are very stable and that there is always a reason something works or does not work. That reason may not be readily apparent, but with study it comes. The iocage utility is an abstraction, it definitely has its uses and I do run it on my jail host, but the native jail.conf will teach you more about how jails actually work.

 

[keldonin]

You're looking at two distinctly different /etc/ directories. Just look at the timestamps of the files they have in common, they don't match up either. And neither does the group ownership. So the obvious conclusion is that it shows different directories.

Good catch, the /etc directory inode numbers are different indeed.

I finally found the culprit: when listing [jailroot]/etc with Linux's ls, what you reach is actually [jailroot]/compat/linux/etc, while FreeBSD's ls looks at [jailroot]/etc. Since the base Debian system was deployed using debootstrap executed from the FreeBSD host, files were deployed inside of [jailroot]/etc, something the Linux userland processes do not see, being routed by linux.ko to [jailroot]/compat/linux/etc instead.

Turned off the jail, merged [jailroot]/etc to [jailroot]/compat/linux/etc, removed [jailroot]/etc directory and symlinked [jailroot]/compat/linux/etc to [jailroot]/etc. Works like a charm now!

Other edge cases involve firewalls and networking and all sorts of fun stuff that can keep you up at night. FreeBSD Mastery: Jails is probably what you want to look at because it covers both iocage and standard jails and shows how to solve a problem in each. It also covers a lot of the edge cases. It has been out for a bit, so look to the current docs from the foundation to update the book to your version of the OS.

I read that book, great reading indeed. However the section on linux jails is rather limited. Thanks for all the guidance!

 

[SirDice]

I finally found the culprit: when listing [jailroot]/etc with Linux's ls, what you reach is actually [jailroot]/compat/linux/etc, while FreeBSD's ls looks at [jailroot]/etc.

There's a jail option specific for Linux jails nowadays. Forgot what it's called though, it's a relatively new setting.