Linux Jails - Experimental Script

[murzik]

yes

 

[Jip-Hop]

@Jip-Hop this is something your script will need to account for.

Yes I'm working on it.

Anyone running into this please try jlmkr.py from the develop branch if you're using the beta release of SCALE 24.04.

 

[Jip-Hop]

I just released Jailmaker v1.1.0 with improvements, new features and support for the upcoming release of TrueNAS SCALE 24.04 (Dragonfish). There should be no breaking changes. You can simply replace jlmkr.py to upgrade to the latest version.

Changes:​

• Added non-interactive create command #85
• Added templates for new jails #84
• Added restart function #81
• Added automatic loading of nvidia kernel module da2c903
• Added --bind-ro=/sys/module by default (for new jails) to make lsmod happy 303f79a
• Added --inaccessible=/sys/module/apparmor by default (for new jails) to trick the jail into thinking the apparmor kernel module is not loaded 4655f17
• Added exit status codes for each command 00e98ac
• Added start/stop hooks f9730d3
• Added initial_setup command/script option in config file 8571caa
• Added jlmkr shell aliases as alternative to the jlmkr symlink d7b3001
• Remove redundant system-call-filter 53689df
• Improved jlmkr list output 21eef74
• Improved jlmkr stop will now wait until jail is stopped 7be1c7c

 

[Jip-Hop]

Please use v1.1.1 as in 1.1.0 the GPU passthrough feature is broken.

 

[cap]

Yes I'm working on it.

Anyone running into this please try jlmkr.py from the develop branch if you're using the beta release of SCALE 24.04.

Jails are not stored on the root filesystem.
Is it sufficient if the developer mode is only switched on for the installation of Jailmaker or does it have to be switched on permanently?

 

[Jip-Hop]

Just use the latest release of jailmaker and don't worry about the readonly rootfs.

 

[grahamperrin]

… a PoC of how to run Incus (LXD, LXC, KVM) inside a jail. …

Superseded by <https://github.com/Jip-Hop/jailmaker/blob/main/templates/incus/README.md>?

(I can't find incus_lxd_lxc_kvm.md on main and – sorry – I'm speed-reading here.)

 

[Jip-Hop]

Superseded by <https://github.com/Jip-Hop/jailmaker/blob/main/templates/incus/README.md>?

(I can't find incus_lxd_lxc_kvm.md on main and – sorry – I'm speed-reading here.)

Yes superseded. Still experimental though.

 

[SamOCk]

I'm suspecting I've missed the bleeding obvious but here goes, my system does not seem to be able to download lxc images. My current suspicion is that I've mis-configured DNS or Network Settings in some manner, but the double '/' in the URL is a little strange to me.
I've put the output from my console below:

Code:

Creating jail debDocker from config template ./templates/docker-jail/config.

TIP: Run `jlmkr create` without any arguments for interactive config.
Or use CLI args to override the default options.
For more info, run: `jlmkr create --help`

Downloading the image index
Downloading the rootfs
ERROR: Failed to download https://images.linuxcontainers.org//images/debian/bookworm/amd64/default/20240311_05:24//rootfs.tar.xz
Aborting...

Thanks

 

[Jip-Hop]

The download link works for me on my phone: https://images.linuxcontainers.org/...m/amd64/default/20240311_05:24//rootfs.tar.xz

The double slashes come from the lxc download template code: https://github.com/Jip-Hop/lxc/blob/jlmkr/templates/lxc-download.in but they should not be causing issues.

Can you curl the download link from the TrueNAS host?

 

[dirtyfreebooter]

jailmaker is great. thanks for your hard work. finally a simple docker solution that can access ZFS shares. no more 10% k3s-server cpu usage all the time. install docker/portainer, nvidia/intel gpus just work. would be nice to one day see a jails UI, but this works great and after i setup my one jail and setup docker, i haven't messed with it since. thank you @Jip-Hop

 

[benda]

Noob Question.
Does jailmaker work with TrueNAS Scale 23.xx , or do i have to use the latest beta 24.x?

Found the answer. https://github.com/Jip-Hop/jailmaker/blob/main/docs/compatibility.md#truenas-compatibility

 

[dirtyfreebooter]

Noob Question.
Does jailmaker work with TrueNAS Scale 23.xx , or do i have to use the latest beta 24.x?

yes it works with 23

 

[delgas]

I've been using FreenNAS since 9.10 . I looked at SCALE when it first released and was not impressed.

This is one of the most amazing developments I've ever seen! I do a lot of work with very large databases, with much of the software needing to run under linux. Using Linux VMs with access to datasets over NFS in TrueNAS CORE was becoming very tiresome.

This is amazing! In about an hour I was up and running and doing serious testing on my old backup NAS. I'm very seriously considering migrating to SCALE so I can access my datasets natively from Linux and vastly improve performance because of this Linux jail capability you've rolled up.

Thank you! Incredible work.

 

[delgas]

What's the best way to limit the amount of memory a jail can use?

When working with large databases they will use every byte of RAM they can get their hands on as buff/cache , which eventually reduces my ARC cache to near zero.

I read through the thread and didn't see it mentioned. I'm off to RTFM on nspawn to see if I can find it there but wanted to reach out and see if anyone had an idea on limiting RAM for a jail.

thanks

--d

 

[Stux]

This looks useful

Dockge

Self-hosted - Docker compose.yaml - Stack-oriented Manager

dockge.kuma.pet

This Docker Compose UI is amazing! //Dockge Tutorial

In this video, I'll be introducing you to Dockge, a free and open-source web UI for managing Docker Compose projects in your Homelab. Developed by the creato...

www.youtube.com

I never did like portainer


EDIT: I liked Dockge so much, I made a tutorial for setting up Docker, Dockge and Jellyfin with Sandboxes and Jailmaker. I also going into static IPs and maclvan vs bridge networking

TrueNAS Scale - Setting up Sandboxes with Jailmaker

How to setup Sandboxes with Jailmaker for Docker, Dockge, Jellyfin and any other docker compose stacks on TrueNAS Scale Dragonfish (24.04)Sandboxes allow ins...

www.youtube.com

 

[delgas]

Following up a bit, in case someone else might find this useful. I was able to get docker and portainer up and running in a 'jail' . In order to create some useful containers within portainer, i needed to add the following SYSTEMD_NSPAWN_USERS_ARGS line to the config file for the 'jail' (via jlmkr edit <jailname>:

--capability=CAP_BLOCK_SUSPEND,CAP_MAC_ADMIN,CAP_IPC_LOCK,CAP_MAC_OVERRIDE,CAP_SYSLOG,CAP_SYS_MODULE,CAP_SYS_PACCT,CAP_SYS_RAWIO,CAP_SYS_TIME,CAP_WAKE_ALARM

I will follow up if I find any additional parameters that are required for creating other containers.

 

[dirtyfreebooter]

Following up a bit, in case someone else might find this useful. I was able to get docker and portainer up and running in a 'jail' . In order to create some useful containers within portainer, i needed to add the following SYSTEMD_NSPAWN_USERS_ARGS line to the config file for the 'jail' (via jlmkr edit <jailname>:

--capability=CAP_BLOCK_SUSPEND,CAP_MAC_ADMIN,CAP_IPC_LOCK,CAP_MAC_OVERRIDE,CAP_SYSLOG,CAP_SYS_MODULE,CAP_SYS_PACCT,CAP_SYS_RAWIO,CAP_SYS_TIME,CAP_WAKE_ALARM

I will follow up if I find any additional parameters that are required for creating other containers.

by default, jailmaker passes

Code:

--capability=all

to each jail. i wonder if that is broken.

Code:

$ ps ax | grep nspawn
   4026 ?        Ss     0:00 /usr/bin/systemd-nspawn --bind-ro=/sys/module --boot \
    --inaccessible=/sys/module/apparmor --quiet --keep-unit --machine=auberon \
    --directory=rootfs --capability=all --property=DeviceAllow=char-drm rw --bind=/dev/dri \
    --network-bridge=br0 --resolv-conf=bind-host ...

 

[benda]

Following up a bit, in case someone else might find this useful. I was able to get docker and portainer up and running in a 'jail' . In order to create some useful containers within portainer, i needed to add the following SYSTEMD_NSPAWN_USERS_ARGS line to the config file for the 'jail' (via jlmkr edit <jailname>:

--capability=CAP_BLOCK_SUSPEND,CAP_MAC_ADMIN,CAP_IPC_LOCK,CAP_MAC_OVERRIDE,CAP_SYSLOG,CAP_SYS_MODULE,CAP_SYS_PACCT,CAP_SYS_RAWIO,CAP_SYS_TIME,CAP_WAKE_ALARM

I will follow up if I find any additional parameters that are required for creating other containers.

I used the jailmaker docker template. https://github.com/Jip-Hop/jailmaker/tree/main/templates/docker

With it i had no problem to install portainer.

 

[dirtyfreebooter]

yea, that was my experience, i just grabbed the script, installed debian 12, get-docker.sh, portainer and never messed with the jail again. works great. intel gpu passthrough, etc.