Solved jail: mount zfs as unprivileged user -- Insufficient privileges

skhal

skhal

Hello,

I am running successfully FreeBSD jails with full userland-stack and enjail'ed ZFS datasets. The technology works great:

- /etc/rc and /etc/rc.shutdown initialize mutli-user environment
- root user has full control over enjail'ed datasets in the jail: create, mount, destroy, allow, etc.

The problem is to allow non-privileged users in the jail to manage child datasets via ZFS privileges (zfs allow and friends).

I allowed a user/group to create/mount/destroy datasets in a jail, but an attempt to actually create a dataset fails at mounting with Insufficient Privileges.

Workflow

Allow test user ztest:ztest inside the jail to manage child datasets:
Code: 
jail:mercury # zfs list | tail -1
zroot/net.skhal/dev/mercury/zroot    88K  14.6G    88K  /

jail:mercury # zfs allow zroot/net.skhal/dev/mercury/zroot
---- Permissions on zroot/net.skhal/dev/mercury/zroot ----------------
Permission sets:
    @zjail canmount,create,destroy,mount,mountpoint
Local+Descendent permissions:
    group ztest @zjail

Now, create a child-dataset as ztest:ztest user:
Code: 
jail:ztest@mercury % id -Gn
ztest

jail:ztest@mercury % zfs create -o mountpoint=/tmp/ztest zroot/net.skhal/dev/mercury/zroot/ztest
cannot mount 'zroot/net.skhal/dev/mercury/zroot/ztest': Insufficient privileges
filesystem successfully created, but not mounted

Howerver, running the command as root in the jail works:
Code: 
jail:mercury # zfs create -o mountpoint=/tmp/rtest zroot/net.skhal/dev/mercury/zroot/rtest

jail:mercury # zfs mount
zroot/net.skhal/dev/mercury/ROOT/default  /
zroot/net.skhal/dev/mercury/zroot/rtest  /tmp/rtest

Host Setup
Code: 
% uname -a
FreeBSD freebsd.skhal.net 11.1-RELEASE FreeBSD 11.1-RELEASE #0 r321309: Fri Jul 21 02:08:28 UTC 2017     root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

% grep '\(allow\|enforce\)' /etc/jail.conf
allow.mount;
allow.mount.zfs;
allow.noset_hostname;
allow.raw_sockets;
enforce_statfs = 1;

% { grep -m 1 '^[[:alnum:]]'; cat; } < /etc/sysctl.conf
vfs.usermount=1
security.jail.allow_raw_sockets=1
security.jail.mount_allowed=1
security.jail.enforce_statfs=1
 
The easiest solution would be to give the user root privileges. Another possible option (not desirable) could be to set the setuid bit on /sbin/mount (execute as owner, in this case root) or you could rely on software such as security/sudo.
 
I am trying to avoid sudo-privileges at all costs b/c the goal is to use a service/daemon I am working on, that will run under some user (ztest:ztest in the example above) and be able to create child-datasets starting from some point in ZFS.

The idea is to let ZFS manage user privileges and don't worry about the service break other datasets (or even apply quotas in the future and alike).
 
Thanks for quick reply. I am wondering, is it a limitation of jailed ZFS, e.g. only root can manage datasets inside the jail?

The idea of privileging users with ZFS permissions model works great on the host-machine though.
 
skhal said:
I am wondering, is it a limitation of jailed ZFS, e.g. only root can manage datasets inside the jail?
Click to expand...
Basically the jail will apply plenty of restrictions to the system. However... re-reading your first message I noticed that you set certain jail settings in /etc/sysctl.conf instead of /etc/jail.conf, why is that? That could be a clue as to why this isn't working:

Code: 
root@unicron:/home/peter # grep sock /etc/jail.conf
        allow.raw_sockets;
root@unicron:/home/peter # sysctl security.jail.allow_raw_sockets
security.jail.allow_raw_sockets: 0
root@unicron:/home/peter # jexec psi ping -c1 10.0.1.100
PING 10.0.1.100 (10.0.1.100): 56 data bytes
64 bytes from 10.0.1.100: icmp_seq=0 ttl=255 time=0.345 ms

--- 10.0.1.100 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.345/0.345/0.345/0.000 ms
See what I mean? I don't mess with the security.jail.allow_raw_sockets setting because there is no need to. I think you're confusing what that setting actually does. It's not something you set on the host, but it's actually used by the jail:

Code: 
root@unicron:/home/peter # jexec psi sysctl security.jail.allow_raw_sockets
security.jail.allow_raw_sockets: 1
So re-reading I think that's where the cause of your problem might be.

Check /etc/jail.conf (see also jail(8)) and be sure to set parameters such as allow.mount and allow.mount.zfs. Also be sure to keep an eye out for enforce_statfs.

(edit)

Meh, I need more coffee. Noticed that you have those options in jail.conf but because of the weird way you pasted it there's no way for me to tell if you placed the options in the right spot. If this doesn't work then start by making sure that you configured jail.conf properly (your first post does not show proof of this).

Also get rid of the stuff in /etc/sysctl.conf because there's no need for that.

(edit2)

Code: 
             allow.mount.zfs
                     privileged users inside the jail will be able to mount
                     and unmount the ZFS file system.  This permission is
                     effective only together with allow.mount and only when
                     enforce_statfs is set to a value lower than 2.  See
                     zfs(8) for information on how to configure the ZFS
                     filesystem to operate from within a jail.
So solely basing myself on this snippet I'm tempted to go with my first response: this is most likely limited to privileged users only.
 
Good catch. In fact, the original version of /etc/sysctl.conf only had vfs.usermount=1 .

But then I discovered the issue: was not able to mount datasets in the jail -- and started to desperately try different weird options after couple days of struggling with it, including these extra settings in the /etc/sysctl.conf.

My understanding (please correct, if I am wrong) is that /etc/sysctl.conf controls the host, while same settings in /etc/jail.conf only apply to the host when the jail starts. Therefore two configs should not contradict each other. I thought that if there is something wrong with /etc/jail.conf, then global setting in /etc/sysctl.conf should ensure that settings are set on the host and ZFS inside the jail should mount the dataset for the allowed user.
 
The security.jail.allow_* settings have been deprecated in favor of per-jail settings. Some older tutorials may still have them and they still work but those settings turn the features on for all jails.
 
In order to localize the issue, I created a virtual machine in VirtualBox of FreeBSD 11.1-RELEASE in VirtualBox with a single jail, taking into account previous comments.

The host setup is:
Code: 
root@host:~ # sysrc -a | grep '\(jail\|zfs\)'
jail_enable: YES
zfs_enable: YES

root@host # { grep '^[[:alnum:]]'; cat ; } < /etc/sysctl.conf
vfs.usermount=1

The jail configuration is:
Code: 
root@host # cat -n /etc/jail.conf
     1    allow.mount;
     2    allow.mount.zfs;
     3    allow.mount.devfs;
     4    allow.noset_hostname;
     5    allow.raw_sockets;
     6
     7    mount.devfs;
     8    enforce_statfs = 1;
     9
    10    host.hostname = "$name.dev.skhal.net";
    11    path = "/usr/net.skhal/dev/$name";
    12
    13    persist;
    14    exec.clean;
    15
    16    exec.start = "/bin/sh /etc/rc";
    17    exec.poststart = "/sbin/zfs jail $name zroot/net.skhal/dev/$name/zroot";
    18    exec.stop = "/bin/sh /etc/rc.shutdown";
    19
    20    mercury {
    21      ip4.addr  = "em0|10.0.2.121/24";
    22    }

Inside the jail, the setup is minimal:
Code: 
root@mercury:/ # sysrc -a
clear_tmp_enable: YES
syslogd_flags: -ss
zfs_enable: YES

root@mercury:/ # getent passwd ztest
ztest:PASSWORD:1001:1001:User &:/home/ztest:/bin/sh

root@mercury:/ # getent group ztest
ztest:*:1001

Jail is setup on ZFS and replicates FreeBSD datasets hierarchy, e.g. ROOT/default holds the base system, while all customizations are put into zroot (so that I can en-jail the dataset but not the jail base system):
Code: 
root@mercury:/ # zfs mount
zroot/net.skhal/dev/mercury/ROOT/default  /

root@mercury:/ # zfs list
NAME                                USED  AVAIL  REFER  MOUNTPOINT
zroot                               955M   757M    88K  /zroot
zroot/net.skhal                     296M   757M    88K  /usr/net.skhal
zroot/net.skhal/dev                 296M   757M    88K  /usr/net.skhal/dev
zroot/net.skhal/dev/mercury         296M   757M    88K  none
zroot/net.skhal/dev/mercury/zroot    88K   757M    88K  /

In order to eliminate a possible bug/issue of integrating ZFS/mount and user privileges, I allow everyone to manage datasets and mountpoints under zroot (the security is the least concern at this point):
Code: 
root@mercury:/ # zfs allow zroot/net.skhal/dev/mercury/zroot
---- Permissions on zroot/net.skhal/dev/mercury/zroot ----------------
Local+Descendent permissions:
    everyone create,destroy,mount,mountpoint

Now, the experiment goes: create a child dataset as a non-root user in the jail:
Code: 
root@mercury:/ # su ztest

ztest@mercury % ls -l /tmp
total 0

ztest@mercury % zfs create -o mountpoint=/tmp/ztest zroot/net.skhal/dev/mercury/zroot/ztest
cannot mount 'zroot/net.skhal/dev/mercury/zroot/ztest': Insufficient privileges
filesystem successfully created, but not mounted

Interesting enough ZFS has actually created the dataset, it even has created a mounting folder (which is supposed to be owned by the user, in order to mount the dataset):
Code: 
ztest@mercury % zfs list -o name,canmount,mounted,mountpoint -r zroot/net.skhal/dev/mercury/zroot
NAME                                     CANMOUNT  MOUNTED  MOUNTPOINT
zroot/net.skhal/dev/mercury/zroot             off       no  /
zroot/net.skhal/dev/mercury/zroot/ztest        on       no  /tmp/ztest

ztest@mercury % ls -l /tmp
total 1
drwxr-xr-x  2 ztest  wheel  2 Apr 21 21:36 ztest

The mounting part has failed, e.g. ZFS privileges part worked perfectly fine, but mounting didn't pass.

If I try to mount the dataset as root in the jail, it works perfectly fine:
Code: 
root@mercury:/ # zfs mount -a

root@mercury:/ # zfs list -o name,canmount,mounted,mountpoint -r zroot/net.skhal/dev/mercury/zroot
NAME                                     CANMOUNT  MOUNTED  MOUNTPOINT
zroot/net.skhal/dev/mercury/zroot             off       no  /
zroot/net.skhal/dev/mercury/zroot/ztest        on      yes  /tmp/ztest

So, it seems to me, that something is probably wrong or missing in the jail setup.

FYI, FreeBSD Mastery books "ZFS" and "Advanced ZFS" say that a non-root user can mount a dataset as long as the user is granted mounting privileges through zfs allow and owns the mount point on the filesystem: both of these are met so far (I use /tmp/ztest, which is owned by the user). I can re-run the exercise on the host machine as non-root user with success.

Is there additional setup I need to make for a non-root user in the jail to allow ZFS mounts?
 
In order to complete the story, I'll post my solution.

The idea was to run a jail on ZFS and manage datasets inside the jail using zfs allow privileges, granted to unprivileged users (daemons). The privileges would be basic ones, such as dataset creation, mount, snapshot, etc., and advanced - NFS share. I didn't want to use sudo/doas in order to increase security.

Apparently, there are two issues in given setup:
  1. mounting datasets inside the jail is problematic without sudo/doas -- I could probably tolerate this and change my application if the problem would only be in mounting datasets
  2. NFS sharing can only be done from the host machine (jails can not control/run an NFS server)
So, I decided to manage ZFS datasets outside of the jail and access them inside the jail as regular/mounted folders:
  1. run a small daemon on the host that would create/mount ZFS datasets -- daemon user must be granted privileges on the host using zfs allow
  2. the daemon would also mount these datasets into the jail(s) and NFS share datasets as needed
  3. the application inside the jail communicates to the daemon through one of RPC methods (say gRPC) to request dataset creation, mount, etc.
I'd like to thank everyone, who replied.
 
You must log in or register to reply here.
Back
Top