[ENHANCEMENT] support zfs jailed datasets

[dch]

Is your feature request related to a problem? Please describe.

Map (jail & mount) a generic zfs dataset into a bastille jail

I would like a largely disposable jail, where the data lives in a jailed zfs dataset,
that is separate to the bastille jail tree. This enables rebuilding jails as needed,
for fast updates, and only dropping in a small number of config files into the jail,
leaving the large dataset outside the bastille tree, which simplifies backups and
sysadmin work.

Given a single bastille jail (possibly built from a template), I would like to have:

• a zfs dataset (& possibly child datasets)
• that lives outside the bastille jail tree
• that is auto-mounted into the jail on startup, including any of its child datasets
• and un-mounted (unjailed) on shutdown
• and can be attached to a new bastille jail when it is (re)built
• optionally allows the jail user to issue appropriate delegated zfs commands

this enables using jails ephemerally, with the following structure:

• zroot/jailed/:jail/var/www/nginx -> inside the jail /var/www/nginx
• zroot/bastille/jails/:jail/root -> the main jail root filesystem

Describe the solution you'd like

user must prepare the following dataset:

# zfs create -o jailed=on -o canmount=off -o mountpoint=/var/www zroot/jailed/alcatraz
# zfs create zroot/jailed/alcatraz/nginx
# zfs list -o canmount,mounted,jailed,mountpoint,name,usedbydataset -r embiggen/jailed
CANMOUNT  MOUNTED  JAILED  MOUNTPOINT                       NAME                                                      USEDDS
off       no       off     none                             embiggen/jailed                                              96K
off       no       on      /var/www                         embiggen/jailed/alcatraz                                     96K
on        no       on      /var/www/nginx                   embiggen/jailed/alcatraz/nginx                               96K

when jail is created, bastille needs to:

• pull in a zfs_dataset or similar property from config, naming the root dataset zroot/jailed/alcatraz from above
• set bothallow.mount.zfs=1 enforce_statfs=1
• add zfs_enable=YES to jail's /etc/rc.conf or similar
• append something (perhaps to /usr/local/jail.conf ) to do the mounting, if the above zfs_enable won't do it already

This example, taken from iocell, shows the outcome:

root@f01 /u/h/dch# iocell list
JID  UUID                                  BOOT  STATE  TAG        TYPE       IP4          RELEASE
1    90a81ad1-64b9-11eb-b7d0-0cc47a16edea  on    up     eden       clonejail  100.64.0.10  12.2-RELEASE
2    92049a12-64b9-11eb-b7d0-0cc47a16edea  on    up     couchdb    clonejail  100.64.0.3   12.2-RELEASE
3    9367693e-64b9-11eb-b7d0-0cc47a16edea  on    up     mu         clonejail  100.64.0.8   12.2-RELEASE
4    94da7aa1-64b9-11eb-b7d0-0cc47a16edea  on    up     www        clonejail  100.64.0.6   12.2-RELEASE
5    9650ef84-64b9-11eb-b7d0-0cc47a16edea  on    up     seaweedfs  clonejail  100.64.0.2   12.2-RELEASE
6    97df0758-64b9-11eb-b7d0-0cc47a16edea  on    up     serenity   clonejail  100.64.0.9   12.2-RELEASE

root@f01 /u/h/dch# zfs list -r zroot/jailed
NAME                             USED  AVAIL  REFER  MOUNTPOINT
zroot/jailed                    6.74G   105G    88K  none
zroot/jailed/couchdb3            736M   105G   155M  /var/db/couchdb3
zroot/jailed/couchdb3/views     79.3M   105G  12.2M  /var/db/couchdb3/views
zroot/jailed/fdb                  88K   105G    88K  /var/db/fdb
zroot/jailed/mu                   88K   105G    88K  /var/mu
zroot/jailed/seaweedfs           744K   105G   264K  /var/db/seaweedfs
zroot/jailed/seaweedfs/dataset    96K   105G    96K  /var/db/seaweedfs/dataset
zroot/jailed/www                6.02G   105G  5.44G  /var/www
...

Describe alternatives you've considered

nullfs mounts. easy, but we want 2 things:

• leverage zfs so that the jail can do its own snapshots etc
• hide the dataset from the jail host, zfs jail .. does this for us

Additional context

Happy to get involved implementing this with some indication of where the various options/code would be best added.

[dch]

any interest in this?

[cedwards]

This looks interesting. Happy to discuss more; thank you for offering to implement.

Due to the changes required for rc.conf and jail.conf I'd recommend building into create.sh, possible triggered by a -Z option (for ZFS jailed). We have similar options in create.sh for -V (VNET) or -T ("thick"), etc.

I don't have time to implement myself but will gladly review a pull request if you come up with something.

[markjdb]

I have a use for this feature as well. It's mostly possible to get this working with config commands, at least when it's a single dataset (haven't thought much about the case of child datasets).

So far I did this:

• unhide /dev/zfs in the jail's devfs ruleset
• set allow.mount.zfs and enforce_statfs=1
• set exec.created such that it invokes zfs jail <name> <dataset>
  • this is annoying to set via bastille config since I can't use jail.conf variables without the shell expanding them
• add an fstab entry to mount the volume automatically when the jail is started
  • I can't use bastille mount for this since it only permits nullfs mounts for some reason.

This is similar to the steps listed above. Setting zfs_enable might be better than editing the jail fstab, I'm not sure.

Invoking zfs jail seems like it should become a jail(8) feature, but I guess we don't want to depend on new base system features.

Regarding the implementation, adding -Z <dataset> to the create command could be a way to go, but I wonder if it'd be better to let it be applied to an existing jail. (Also so you can "detach" a dataset without destroying the jail.) bastille config is strictly for setting jail parameters, but maybe it'd be good to support setting higher-level, Bastille-specific configuration parameters in some way.

[g-teley]

This is exactly what I was looking for

[patmaddox]

fwiw for anyone interested in delegating ZFS datasets to bastille jails, here are my additions to the bastille-generated jail.conf. This is for poudriere so there may be some extra stuff, but the basic line is exec.created += 'zfs jail pkg zroot/jdata/pkg'; and then tweaking the other params until it works.

pkg {
  # additions / changes
  allow.mount;
  allow.mount.devfs;
  allow.mount.zfs;
  allow.mount.nullfs;
  allow.mount.tmpfs;
  allow.mount.procfs;
  children.max = 10;
  devfs_ruleset = 4;
  enforce_statfs = 1;
  securelevel = 0; # defaults to 2, can use 2 with USE_TMPFS in poudriere
  exec.created += 'zfs jail pkg zroot/jdata/pkg';
  
  # bastille defaults
  exec.clean;
  exec.consolelog = /var/log/bastille/pkg_console.log;
  exec.start = '/bin/sh /etc/rc';
  exec.stop = '/bin/sh /etc/rc.shutdown';
  host.hostname = pkg;
  mount.devfs;
  mount.fstab = /usr/local/bastille/jails/pkg/fstab;
  path = /usr/local/bastille/jails/pkg/root;

  interface = bastille0;
  ip4.addr = 172.16.1.4;
  ip6 = disable;
}

[bdrewery]

Should also have this but it doesn't seem to matter much.

exec.poststop += 'zfs unjail pkg zroot/jdata/pkg';

[cedwards]

I've added this to the roadmap for an upcoming release. Thank you for your patience. 😄

[tschettervictor]

Why not just mount the directories into the jail using "bastille mount"?

[vegged]

Why not just mount the directories into the jail using "bastille mount"?

This does not provide the same amount of user control as a delegated zfs dataset.

When zfs set jailed=on $dataset & zfs jail $jailname $dataset are invoked, $dataset is bound to $jailname and can be fulled managed within the jail. This allows the creation of child datasets with their own unique properties by jail users with no access to the host whatsoever, for instance!

[tschettervictor]

What about creating a template with the CONFIG hook applying all the above properties to your jail?

CONFIG exec.created += 'zfs jail pkg zroot/jdata/pkg'
CONFIG allow.mount

etc...

You could also have ARGS on top like JAILNAME and replace args in the strings like

ARG JAILNAME

CONFIG exec.created += 'zfs jail ${JAILNAME} zroot/jdata/${JAILNAME}';

Then it can be applied to each jail using the template command.

[vegged]

Should also have this but it doesn't seem to matter much.

exec.poststop += 'zfs unjail pkg zroot/jdata/pkg';

zfs unjail has to be called while the jail is still running, @bdrewery, so for anyone stumbling upon this discussion looking for guidance, this should be placed in jail.conf, in addition to the exec.created config directive mentioned by @patmaddox :

exec.prestop += '/sbin/zfs unjail pkg zroot/data/pkg';

I'll give the template approach a try, @tschettervictor!