Linux Jails - Experimental Script

[Nicholas Flamy]

I'm using this to run a postgresql server as well as a ffmpeg build script that executes every other month. It's great.

 

[cap]

I think it's great that jailmaker exists. However, I don't understand how a bind mount works with the ACLs in TrueNAS Scale.
I have created a jail and installed Jellyfin in it. I have mounted a dataset "Test" read only via bind: 'systemd_nspawn_user_args=--bind-ro=/mnt/default/Test'. This also works when I am in the terminal (bash) in the jail.
But in Jellyfin I cannot access the contents of the dataset. However, I get access to Jellyfin if I change the permissions for the test dataset in TrueNAS Scale and give the permissions to 'everyone@' in the ACLs.

How do I set the ACLs so that only Jellyfin (in addition to other authorized users and groups) and not everyone gets access to the dataset?

edit:

Code:

root@jellyfin:~# id jellyfin
uid=102(jellyfin) gid=108(jellyfin) groups=108(jellyfin),44(video),105(render)

 

[sretalla]

You can either create a user in TrueNAS called jellyfin with UID 108 and then assign that user as owner or just assign UID 108 with chown (but then not be able to access the files from TrueNAS).

 

[cap]

You can either create a user in TrueNAS called jellyfin with UID 108 and then assign that user as owner

Bult-In User "Consul" already has UID 108 :/

Edit:
I will probably go the Docker route (also because I am interested in Paperless-NGX and Immish). But I might run into the same problem there...

 

[sretalla]

Or you could set your app to use a different UID... to one that isn't taken already in TrueNAS.

 

[cap]

and then assign that user as owner

It should be enough if the corresponding user is given read permissions (possibly also write permissions) and does not become the owner straight away.

I have changed the uid and gid of jellyfin. But now I can no longer establish a connection in the browser. I must have done something wrong. I will go the Docker route now, also because I need Docker for other things.

 

[Jip-Hop]

I don't think you need ACLs for this basic scenario. I have removed all ACLs from my datasets and just use regular Unix filesystem permissions. Have zero trouble using these files inside jailmaker jails. I can even share these datasets simultaneously via SMB with ACL disabled.

 

[anodos]

FYI in DragonFish we are beginning to make the most of boot pool readonly and so if you need to make modifications to it, you may want to prompt users to run /usr/local/libexec/disable-rootfs-protection as root if attempts to make changes fails with EROFS.

 

[Jip-Hop]

Thanks. As long as systemd-container package is pre-installed I don't need to make modifications to the rootfs for jailmaker to work. Not modify the rootfs was always the goal but then there was at some point a release of SCALE where this package was missing so I had to resort to enable apt and install thus modify the rootfs. But good to know there's a way to disable rootfs protection in case it's required to keep jailmaker working in the future.

 

[cap]

I think this is possible, but I have no experience with it: https://manpages.debian.org/bullsey...emd-nspawn.1.en.html#User_Namespacing_Options.

Any news on this one?

 

[Jip-Hop]

Jailmaker doesn't set this up for you but I suppose you could add those flags to the systemd_nspawn_additional_args in the config file. Just try it out if you're interested :)

 

[NugentS]

For the record I have Portainer running on a Jip-Hop Jailmaker Scale Jail.
24 Containers at the moment. This runs extremely well and is far far better than running in a VM or via the built in apps menu.
I also have 8 containers running as TrueNAS Apps which I haven't migrated over yet (will probably only migrate 6, leaving 2 as TN Apps)

I will admit that I am probably not running in a manner that might meet best practises when it comes to security.

 

[cap]

I would like to use TrueNAS Scale as it is intended. However, I have noticed that with TrueNAS Apps enabled, the server consumes noticeably more power. The jails that you build with Jailmaker, on the other hand, are not noticeable at all. This is just a first impression.
Kubernetes may be good. But I don't need these features at home. I know that TrueNAS is aimed at professional users. Nevertheless, it would be nice to have such options without a "hack".

 

[cap]

You can either create a user in TrueNAS called jellyfin with UID 108 and then assign that user as owner or just assign UID 108 with chown (but then not be able to access the files from TrueNAS).

It should be enough if the corresponding user is given read permissions (possibly also write permissions) and does not become the owner straight away.

Seems to work with Docker. I created the user jellyfin with UID 3050 in TrueNAS Scale and then gave him read access to the dataset 'Test" via ACL.

Code:

docker run -d \
--name jellyfin \
--user 3050:3050 \
--net=host \
--volume /srv/jellyfin/config:/config \
--volume /srv/jellyfin/cache:/cache \
--mount type=bind,source=/mnt/default/Test,target=/media \
--restart=unless-stopped \
jellyfin/jellyfin

 

[Davvo]

Kubernet wastes CPU power as a feature, maybe that's where the difference in power consumption you noticed comes from.

 

[cap]

Jailmaker doesn't set this up for you but I suppose you could add those flags to the systemd_nspawn_additional_args in the config file. Just try it out if you're interested :)

Despite having little knowledge of Linux, I experimented a little: --private-users=pick --private-users-chown

Example:
Jailmaker default in container myjail:

Code:

# cat /proc/self/uid_map
         0          0 4294967295

Addition of the following flags (# jlmkr edit myjail): systemd_nspawn_user_args--private-users=pick --private-users-chown

Code:

startup=1
docker_compatible=1
gpu_passthrough_intel=1
gpu_passthrough_nvidia=0
systemd_nspawn_user_args=--bind-ro=/mnt/default/media/center --private-users=pick --private-users-chown
# You generally will not need to change the options below
systemd_run_default_args=--property=KillMode=mixed --property=Type=notify --property=RestartForceExitStatus=133 --property=SuccessExitStatus=133 --property=Delegate=yes --property=TasksMax=infinity --co>
systemd_nspawn_default_args=--keep-unit --quiet --boot

=> in container myjail:

Code:

# cat /proc/self/uid_map
         0 1357119488      65536

However, you can then no longer access the bind-mount (ootb). (**)

Code:

# cd /mnt/default/media/center
bash: cd: /mnt/default/media/center: Operation not permitted

Suddenly starting the jail takes longer with the following message:

Code:

# jlmkr shell myjail


Connected to machine myjail. Press ^] three times within 1s to exit session.


bash: /root/.bash_profile: Permission denied

Despite the above adjustments, this was previously possible without any delay.
I can't see what I've done differently.

Edit:
(**)
To be able to assign permissions in TrueNAS Scale via ACL, it is better to assign the UID directly and not to assign it via the "pick" option, e.g. --private-users=10000000 => systemd_nspawn_user_args=--bind-ro=/mnt/default/media/center --private-users=10000000

 

[Jip-Hop]

Great initial exploration of rootless jails, thanks for sharing. Looks like it's mostly working! What are the permissions and owner/group of the /root/.bash_profile file? And which UID is the user inside the jail?

 

[cap]

What are the permissions and owner/group of the /root/.bash_profile file? And which UID is the user inside the jail?

Inside Container:

Code:

# cat /proc/self/uid_map
         0   10000000      65536

Code:

# id
uid=0(root) gid=0(root) groups=0(root)

Code:

# ls -l
total 88
lrwxrwxrwx   1 nobody nogroup   7 Dec 27 06:25 bin -> usr/bin
drwxr-xr-x   2 nobody nogroup   2 Dec  9 22:08 boot
drwxr-xr-x   7 root   root    440 Jan  5 00:50 dev
drwxr-xr-x  45 nobody nogroup  99 Jan  4 21:31 etc
drwxr-xr-x   2 nobody nogroup   2 Dec  9 22:08 home
lrwxrwxrwx   1 nobody nogroup   7 Dec 27 06:25 lib -> usr/lib
lrwxrwxrwx   1 nobody nogroup   9 Dec 27 06:25 lib32 -> usr/lib32
lrwxrwxrwx   1 nobody nogroup   9 Dec 27 06:25 lib64 -> usr/lib64
lrwxrwxrwx   1 nobody nogroup  10 Dec 27 06:25 libx32 -> usr/libx32
drwxr-xr-x   2 nobody nogroup   2 Dec 27 06:25 media
drwxr-xr-x   3 nobody nogroup   3 Jan  4 22:39 mnt
drwxr-xr-x   2 nobody nogroup   2 Dec 27 06:25 opt
dr-xr-xr-x 474 nobody nogroup   0 Jan  5 00:50 proc
drwx------   3 nobody nogroup   6 Jan  4 21:31 root
drwxr-xr-x   9 root   root    220 Jan  5 00:50 run
lrwxrwxrwx   1 nobody nogroup   8 Dec 27 06:25 sbin -> usr/sbin
drwxr-xr-x   2 nobody nogroup   2 Dec 27 06:25 srv
dr-xr-xr-x  13 nobody nogroup   0 Dec 31 16:59 sys
drwxrwxrwt   3 root   root     60 Jan  5 11:14 tmp
drwxr-xr-x  14 nobody nogroup  14 Dec 27 06:25 usr
drwxr-xr-x  11 nobody nogroup  13 Dec 27 06:25 var

Code:

# ls -a -l /root
ls: cannot open directory '/root': Permission denied

From outside the container (= root@truenas}:

Code:

# ls -l             
total 131
lrwxrwxrwx  1 329580544 329580544  7 Dec 27 06:25 bin -> usr/bin
drwxr-xr-x  2 329580544 329580544  2 Dec  9 22:08 boot
drwxr-xr-x  3 329580544 329580544  3 Jan  4 21:16 dev
drwxr-xr-x 45 329580544 329580544 99 Jan  4 21:31 etc
drwxr-xr-x  2 329580544 329580544  2 Dec  9 22:08 home
lrwxrwxrwx  1 329580544 329580544  7 Dec 27 06:25 lib -> usr/lib
lrwxrwxrwx  1 329580544 329580544  9 Dec 27 06:25 lib32 -> usr/lib32
lrwxrwxrwx  1 329580544 329580544  9 Dec 27 06:25 lib64 -> usr/lib64
lrwxrwxrwx  1 329580544 329580544 10 Dec 27 06:25 libx32 -> usr/libx32
drwxr-xr-x  2 329580544 329580544  2 Dec 27 06:25 media
drwxr-xr-x  3 329580544 329580544  3 Jan  4 22:39 mnt
drwxr-xr-x  2 329580544 329580544  2 Dec 27 06:25 opt
drwxr-xr-x  2 329580544 329580544  2 Dec  9 22:08 proc
drwx------  3 root      329580544  6 Jan  4 21:31 root
drwxr-xr-x  2 329580544 329580544  2 Dec 27 06:25 run
lrwxrwxrwx  1 329580544 329580544  8 Dec 27 06:25 sbin -> usr/sbin
drwxr-xr-x  2 329580544 329580544  2 Dec 27 06:25 srv
drwxr-xr-x  2 329580544 329580544  2 Dec  9 22:08 sys
drwxrwxrwt  2 329580544 329580544  2 Dec 27 06:25 tmp
drwxr-xr-x 14 329580544 329580544 14 Dec 27 06:25 usr
drwxr-xr-x 11 329580544 329580544 13 Dec 27 06:25 var

jails config:

Code:

startup=1
docker_compatible=1
gpu_passthrough_intel=0
gpu_passthrough_nvidia=0
systemd_nspawn_user_args=--bind=/mnt/default/media/center --private-users=10000000
# You generally will not need to change the options below
systemd_run_default_args=--property=KillMode=mixed --property=Type=notify --property=RestartForceExitStatus=133 --property=SuccessExitStatus=133 --property=Delegate=yes --property=TasksMax=infinity --co>
systemd_nspawn_default_args=--keep-unit --quiet --boot

Edit:
"2.If the parameter is omitted, or true, user namespacing is turned on. The UID/GID range to use is determined automatically from the file ownership of the root directory of the container's directory tree. To use this option, make sure to prepare the directory tree in advance, and ensure that all files and directories in it are owned by UIDs/GIDs in the range you'd like to use. Also, make sure that used file ACLs exclusively reference UIDs/GIDs in the appropriate range. If this mode is used the number of UIDs/GIDs assigned to the container for use is 65536, and the UID/GID of the root directory must be a multiple of 65536."

systemd-nspawn(1) — systemd-container — Debian bullseye — Debian Manpages

----

New Container "Test" with flag: systemd_nspawn_user_args=--private-users=pick --private-users-chown --bind-ro=/mnt/default/media/center

Now the container starts directly again, just as it did in the first container at the beginning.

Code:

root@truenas[...ware/jailmaker/jails/paperless/rootfs]# jlmkr shell test
Connected to machine test. Press ^] three times within 1s to exit session.
root@test:~# 

Code:

root@test:/# cat /proc/self/uid_map
         0 1492320256      65536

root@test:/# id
uid=0(root) gid=0(root) groups=0(root)

root@test:/# ls -l
total 88
lrwxrwxrwx   1 root   root      7 Dec 27 06:25 bin -> usr/bin
drwxr-xr-x   2 root   root      2 Dec  9 22:08 boot
drwxr-xr-x   8 root   root    460 Jan  5 11:25 dev
drwxr-xr-x  45 root   root     99 Jan  5 11:20 etc
drwxr-xr-x   2 root   root      2 Dec  9 22:08 home
lrwxrwxrwx   1 root   root      7 Dec 27 06:25 lib -> usr/lib
lrwxrwxrwx   1 root   root      9 Dec 27 06:25 lib32 -> usr/lib32
lrwxrwxrwx   1 root   root      9 Dec 27 06:25 lib64 -> usr/lib64
lrwxrwxrwx   1 root   root     10 Dec 27 06:25 libx32 -> usr/libx32
drwxr-xr-x   2 root   root      2 Dec 27 06:25 media
drwxr-xr-x   3 root   root      3 Jan  5 11:25 mnt
drwxr-xr-x   2 root   root      2 Dec 27 06:25 opt
dr-xr-xr-x 477 nobody nogroup   0 Jan  5 11:25 proc
drwx------   3 root   root      6 Jan  5 11:22 root
drwxr-xr-x  12 root   root    320 Jan  5 11:25 run
lrwxrwxrwx   1 root   root      8 Dec 27 06:25 sbin -> usr/sbin
drwxr-xr-x   2 root   root      2 Dec 27 06:25 srv
dr-xr-xr-x  13 nobody nogroup   0 Dec 31 16:59 sys
drwxrwxrwt   8 root   root    160 Jan  5 11:25 tmp
drwxr-xr-x  14 root   root     14 Dec 27 06:25 usr
drwxr-xr-x  11 root   root     13 Dec 27 06:25 var

root@test:~# ls -a -l /root
total 39
drwx------  3 root root   6 Jan  5 11:22 .
drwxr-xr-x 17 root root  23 Dec 27 06:27 ..
-rw-------  1 root root 248 Jan  5 11:32 .bash_history
-rw-r--r--  1 root root 571 Apr 10  2021 .bashrc
-rw-r--r--  1 root root 161 Jul  9  2019 .profile
drwx------  2 root root   2 Dec 27 06:25 .ssh

From outside the container (= root@truenas}:

Code:

root@truenas[.../software/jailmaker/jails/test/rootfs]# ls -l
total 131
lrwxrwxrwx  1 1492320256 1492320256  7 Dec 27 06:25 bin -> usr/bin
drwxr-xr-x  2 1492320256 1492320256  2 Dec  9 22:08 boot
drwxr-xr-x  3 1492320256 1492320256  3 Jan  5 11:20 dev
drwxr-xr-x 45 1492320256 1492320256 99 Jan  5 11:20 etc
drwxr-xr-x  2 1492320256 1492320256  2 Dec  9 22:08 home
lrwxrwxrwx  1 1492320256 1492320256  7 Dec 27 06:25 lib -> usr/lib
lrwxrwxrwx  1 1492320256 1492320256  9 Dec 27 06:25 lib32 -> usr/lib32
lrwxrwxrwx  1 1492320256 1492320256  9 Dec 27 06:25 lib64 -> usr/lib64
lrwxrwxrwx  1 1492320256 1492320256 10 Dec 27 06:25 libx32 -> usr/libx32
drwxr-xr-x  2 1492320256 1492320256  2 Dec 27 06:25 media
drwxr-xr-x  3 1492320256 1492320256  3 Jan  5 11:25 mnt
drwxr-xr-x  2 1492320256 1492320256  2 Dec 27 06:25 opt
drwxr-xr-x  2 1492320256 1492320256  2 Dec  9 22:08 proc
drwx------  3 1492320256 1492320256  6 Jan  5 11:22 root
drwxr-xr-x  2 1492320256 1492320256  2 Dec 27 06:25 run
lrwxrwxrwx  1 1492320256 1492320256  8 Dec 27 06:25 sbin -> usr/sbin
drwxr-xr-x  2 1492320256 1492320256  2 Dec 27 06:25 srv
drwxr-xr-x  2 1492320256 1492320256  2 Dec  9 22:08 sys
drwxrwxrwt  2 1492320256 1492320256  2 Dec 27 06:25 tmp
drwxr-xr-x 14 1492320256 1492320256 14 Dec 27 06:25 usr
drwxr-xr-x 11 1492320256 1492320256 13 Dec 27 06:25 var

jails config:

Code:

startup=1
docker_compatible=1
gpu_passthrough_intel=1
gpu_passthrough_nvidia=0
systemd_nspawn_user_args=--private-users=pick --private-users-chown --bind-ro=/mnt/default/media/center
# You generally will not need to change the options below
systemd_run_default_args=--property=KillMode=mixed --property=Type=notify --property=RestartForceExitStatus=133 --property=SuccessExitStatus=133 --property=Delegate=yes --property=TasksMax=infinity --co>
systemd_nspawn_default_args=--keep-unit --quiet --boot

PROBLEM with the pick option:
UID like 1492320256 is not within the range required by TrueNAS Scale (Add User): "Should be between 0 and 90000000". You have to assign the user to be able to control the access rights (ACLs) to datasets via the UI of TueNAS Scale.

 

[anodos]

User namespace mapping for internal ZFS ACLs has not been fully implemented yet and so I would not be surprised if it does not work 100%. We reserve higher numbers of IDs for directory services (AD / LDAP) and so you won't be able to use those for local users via our APIs.

 

[cap]

User namespace mapping for internal ZFS ACLs has not been fully implemented yet and so I would not be surprised if it does not work 100%. We reserve higher numbers of IDs for directory services (AD / LDAP) and so you won't be able to use those for local users via our APIs.

Thanks. I just suspected that:
"Typically "map" is the best choice, since it transparently maps UIDs/GIDs in memory as needed without modifying the image, and without requiring an expensive recursive adjustment operation. However, it is not available for all file systems, currently."

systemd-nspawn(1) — Arch manual pages

man.archlinux.org