[HowTo] FreeNAS 11.1 + RancherOS + Docker + RancherUI + PLEX

[-fun-]

Ok, so this finally worked. For my calibre-server container I did the following (a running Rancher VM as a prerequisite).

I have a dataset /mnt/flospool/jaildata/calibre/ with share type UNIX. As I want to expose two directories ´books` and ´config` to the container I added two directories ´books` and ´config` in the dataset. (One major pitfall was to try to create datasets ´books` and ´config`. This just didn't work! It would have allowed to separately snapshot config and books.)

In FreeNAS I created user/group rancher/rancher with UID=1100 and GID=1100. These IDs match the IDs of the rancher user in the RancherOS VM. This user and group is not specific to the calibre server.

Then I changed user and group ownership in the dataset recursively to rancher/rancher. Also I granted read/write/execute to user/group/other (777).

I setup the NFS sharing service with ´Enable NFSv4` and ´NFSv3 ownership model for NFSv4` checked.

For the dataset I defined an NFS share with ´All Directories´ checked and no Mapall or Maproot entries. ´All Directories` allows to map the subdirectories ´books´ and ´config` to be shared separately. Maproot and Mapall entries are not required because the container will access the share with correct UID and GID.

In Rancher I added the Rancher NFS infrastructure stack. (I use the default environment in Rancher.) I configured a MOUNT_DIR of /mnt/flospool/jaildata/, an NFS_SERVER of 172.27.2.16 (IP address of my FreeNAS box and thus the NFS server host) as well as the ON_REMOVE=retain option. Neither of these settings are used later on however, see below.

Next I defined two storage volumes ´calibre-books` and ´calibre-config`. Here driverOpts must be used. calibre-books is defined as follows:

• export=/mnt/flospool/jaildata/calibre/books (note the full path including the ´books` directory defined earlier)
• host=172.27.2.16 (Important: The ´export` driveOpt just does not work without ´host´ defined! ´host` is not inherited from the NFS_SERVER as one would expect. This took me hours ...)
• onRemove=retain (I prefer to keep the data in the directory should the container be removed.)

The volume for calibre-config is defined accordingly.

After this I added the container. I'm using the technosoft2000/calibre-web container (https://hub.docker.com/r/technosoft2000/calibre-web/).

Definitions include:

• Volumes calibre-books:/books and calibre-config:/calibre-web/config
• Environment variables USE_CONFIG_DIR=true, PGID=1100, PUID=1100
• Port mapping 8083=8083

The PGID and PUID match the rancher user IDs, see above.

Now I started the container. In the browser the new calibre server is available with the IP address of the Rancher VM and the port exposed in the container, in my case: http://172.27.2.35:8083.

Some notes:

• After first launch the container had created some files in the ´books` directory with the right group but an unknown user ID. This is probably due to a problem in the container. I corrected this (set permissions recursively to rancher/rancher ...) and the calibre server works fine since then.
• A mapping of UID and GID in the container should not be required if read/write/execute is granted to ´other` anyway. Or permissions read/write/execute would not have to be granted to ´other` if UID and GID are mapped to the rancher user and group anyway. Depending on security requirements the second might be the better choice but not all containers allow mapping of UID and GID.
• I'm not entirely happy with Rancher: It seems errors in any configuration just cannot be corrected. Throw away and start from scratch.

Many of the above details are derived from input from KrisBee, credits go there, thank you!

As a disclaimer: This is my first working solution. There will be more and probably better solutions for this setup.

 

[KrisBee]

@-fun- Glad to see you've made progress, but it can't be correct to have to use world readable perms.

Belatedly I have found a few rough notes I made and should have included the host in the driver options pairs in my post above (edited now in case anyone else reads it).

I'm need to find time to have another look at all this.

 

[indivision]

Thank you for the guide.

I'm trying to set this up on a static IP.

In order to reserve an IP address in my router I need to assign a MAC address. But, the MAC address for rancher seems to be different every time it boots up. So, it loses the ability to be assigned to the reserved IP.

Any ideas on how to fix that within this set up?

 

[-fun-]

Yes, check first post in this thread.

In Chapter 14.5.4 in the Guide for FreeNAS 11.1 after first access to the VM you can assign a fixed IP address before issuing the command docker run -d --restart=unless-stopped -p 8080:8080 rancher/server.

I did this:

sudo su -
ros config set rancher.network.interfaces.eth0.address 172.27.2.35/24
ros config set rancher.network.interfaces.eth0.gateway 172.27.2.1
ros config set rancher.network.interfaces.eth0.mtu 1500
ros config set rancher.network.interfaces.eth0.dhcp false

Alternatively you would assign a MAC address to the VMs NIC. In the DHCP-server you should be able to create a static mapping for this MAC.

 

[indivision]

Thank you.

I had seen the method in the first post. I didn't think that the router would allow asking for an IP in the range I have set up for reserved IPs without assigning it to a MAC. BUT, I tried it and it appears to work.

If it acts up later I will use the MAC address approach you point to. That is really what I was looking for. Thanks again.

 

[indivision]

Does anyone know what the process is to correct a bad "mount.yaml" merge? I realized that I put the wrong IP address in on the first try. Now, I'm not sure how to correct what was merged into the config.

Also, the directories that you mount the shares to, do those need to be created locally? Or, will this setup make those directories if they do not exist?

Finally, can I mount a subdirectory of a share? For instance "//100.200.300.400/myshare/mydirectoryinshare". Or, does the mounted share have to be the root of the share?

 

[-fun-]

Can't help with the first question. (The missing description of how to remove a merged yaml file kept me from using this at all. I did not even find this in the rancher documentation.)

We're talking about docker containers here. If you setup a docker container this should specify and provide the mount point in the container. So no, no creation of a mount point necessary.

Mounting a subdirectory of a share works fine, see my post above. The subdirectories should not be datasets of their own however.

 

[indivision]

Thank you for the info. Tough for me to tell what is going wrong then. The external links don't show up with "df -h" for me.

Can you elaborate on how you did your set up? It is already a lengthy description. But, I'm hoping to get a command-line guide if possible?

 

[-fun-]

Most of my setup was actually done in the GUI, so I really can't provide a command line guide. From your post above it seems you have the Rancher VM up and running? Do you want to setup the same container (https://hub.docker.com/r/technosoft2000/calibre-web/) as I did? In a fresh Rancher VM that requires 5 steps:

• Setting up local authorization
• Adding a host
• Adding and configuring the NFS service
• Adding volumes
• Setting up the container

How far did you get, which one do you need help for?

 

[indivision]

Ah. Ok. I misunderstood. I didn't realize you were referring to GUI steps.

I do have the Rancher VM up and running. But, I'm not looking to set up the calibre container. I'm focused on just getting the base Rancher system completely setup (with mounted areas on my server that the containers will be able to reach). Then I will "shop" for containers to complement or replace the plugins I'm currently using.

In your list, I think I am stuck at the "Adding volumes" step. I followed the OP merge approach because it sounds like it is needed for the VM to reconnect those volumes at startup. The way you set it up, do you have to re-attach volumes after every re-boot?

 

[-fun-]

Ok, I see.

The difference in the OPs approach and mine is that the OP mounts (CIFS) shares in the Rancher VM. Access is possible from the VM to these shares independent from any docker container. Docker containers can then be configured to use these shares in a second step.

The way I configured this is different: I did not mount a share in rancher. Instead the nfs service provides a way to directly map a container volume to an NFS share. These shares are then made available to the container once the container is started. While the container is running I can see and access the shares from the Rancher VM, these are mapped into directories in /var/lib/rancher/volumes/rancher-nfs/. Once the container is running I can check these using df -h for example.

Volumes defined in Infrastructure / storage, already mounted as visible in the right column:

The volumes definition in the container

While the container is running I can see the shares in rancher. I can't tell why the directories are listed twice though when using df -h:

My calibre container does not start automatically after reboot of the Rancher VM. This would be preferable of course. But starting the container does map the required volumes automatically. So no manual re-attach of shares required.

 

[indivision]

Got it. Thank you for the clarification. Interesting.

I think I would prefer a CIFS arrangement since multiple containers might access the same locations and I already have CIF shares set up. Have you tried setting it up that way previously?

 

[-fun-]

No, I started with NFS straight away. But this was just due to my preference. My setup is not any more valid than the OP's especially when considering existing CIFS shares.

 

[indivision]

I got it working. Turns out that I had written "mount:" instead of "mounts:"

/facepalm

I would still love to learn how to easily remove typo keys like that. From what I've read so far it sounds like there is no delete. Hope that isn't true...

 

[cheetahshrk]

I couldn't get CIFS to mount for a hour until I disabled IPv6 in sysctl for RancherOS. Why is it a problem? No idea other than the fact on a normal Linux system CIFS/mount is usually just as pants on retarded when it comes to networking and would attempt IPv6 even with no IPv6 interfaces bound....sigh

Simple sysctl.yml

Code:

rancher:
  sysctl:
	net.ipv6.conf.all.disable_ipv6: 1
	net.ipv6.conf.default.disable_ipv6: 1
	net.ipv6.conf.lo.disable_ipv6: 1

and then
sudo ros config merge sysctl.yml

Otherwise just constant mount -115 errors with refusal to mount automatically. Manual mount would always work. I noticed a slew of IPv6 messages before and after the dmesg entry for mount failure so I thought to try.

 

[simp]

So I tried to simplify the whole process a bit and make it easily repeatable. This might be useful for others as well.

I packaged the plex container so it could be added to Rancher catalogue. Nothing too dificult, you can see it here: https://github.com/simpss/rancher-plex

If you guys find it useful I might think about pushing it to the community catalogue, but for now it'll stay separate.

usage:

Go to Admin --> Settings --> Catalog --> Add Catalog

name: plex
url: https://github.com/simpss/rancher-plex.git
branch: master

It should look like this:




After refreshing the page you should have an extra catalogue in the top menu:



and a single item:



click on "view details" and it'll take you to the deployment section.

continued in next post....

 

[simp]

Pick a version. The versioning scheme is the same as for the image itself. https://hub.docker.com/r/plexinc/pms-docker/tags/



I included every option that I needed, which was pretty much all of them. They all have sensible defaults, but you'll probably want to go over them and change em up.
More details on what each configuration option does can be seen from here: https://github.com/plexinc/pms-docker#parameters

You'll definately want to change "allowed networks" and "advertise ip".



for data access, I set up suitable NFS shares and only allowed access from the rancherOS VM and then mounted the NFS share to the rancherOS VM with:

Code:

ros config set mounts '[["HOST-IP:/mnt/tank/vm/plex","/mnt/plex","nfs4", ""], ["HOST-ip:/mnt/tank/films","/mnt/films","nfs4", ""]]'

 

[Ochirasu]

If anyone spent forever fixing permission issues like I have, and is about kill themself

If you saw ERROR logs like:
Permission denied: "/config/Library/Application Support/Plex Media Server/Logs"
attempt to write a readonly database
disk I/O error

Add these Environment Variables:
PLEX_UID = 1100
PLEX_GID = 1100

Explanation:
When rancher is creating these folders, it uses the 'rancher' owner and group which is 1100 and 1100 respectively. Problem is then Plex cannot access the files that got created anymore. Setting Plex to use the same UID and GID will let it access the files and fix the problem.

 

[Ochirasu]

If you mess something up, you cannot edit any of these settings. You will have to CLONE your container and change what you need to. I haven't had much success with cloning so I used to destroy and create from scratch.

One thing I found out is that if you create a stack first, and then create the container within the stack, you can do an "upgrade" to the container which will allow you to modify the settings. When you do an upgrade, it will actually create 2 versions of the container, and then you can finalize the upgrade when you know everything is working and it will delete the previous one. This saved me a lot of time when I had to keep changing settings trying to get it to work.

 

[coredalae]

For anyone that is unexperienced with docker... (like me, took me a week to get this sorted out >.<)

When you enable the NFS service by creating
nfs.yml in /var/lib/rancher/conf/cloud-config.d/nfs.yml

Code:

#/var/lib/rancher/conf/cloud-config.d/nfs.yml
write_files:
- path: /etc/rc.local
	permissions: "0755"
	content: |
	 #!/bin/bash
	 [ ! -e /usr/bin/docker ] && ln -s /usr/bin/docker.dist /usr/bin/docker

rancher:
services:
	nfs:
	 image: d3fk/nfs-client
	 labels:
		io.rancher.os.after: console, preload-user-images, wait-for-network
		io.rancher.os.scope: system
	 net: host
	 privileged: true
	 restart: always
	 volumes:
		- /usr/bin/iptables:/sbin/iptables:ro

you can mount nfs share's directly in your docker containers. Skipping the host OS. (making your config more portable, for when you decide you want to run stuff on a raspberry pi 3, but still use the same nfs shares.. or whatever)

by using the

Code:

 --mount 

command.
e.g. I boot my sonarr using:

Code:

docker run -d \
		  --restart=unless-stopped \
		 --name sonarr \
		 --mount 'type=volume,src=media,volume-driver=local,dst=/mnt/media,volume-opt=type=nfs,volume-opt=device=:/mnt/volume1/media/server,"volume-opt=o=addr=192.168.1.98,vers=4,soft,timeo=180,rsize=1048576,wsize=1048576,retrans=2"' \
		 --mount 'type=volume,src=workdisk,volume-driver=local,dst=/config,volume-opt=type=nfs,volume-opt=device=:/mnt/workdisk/docker/sonarr,"volume-opt=o=addr=192.168.1.98,vers=4,soft,timeo=180,rsize=1048576,wsize=1048576,retrans=2"' \
		 -p 8989:8989 \
		 -e APP=sonarr \
		 -e PUID=1000 \
		 -e PGID=1000 \
		 -e UMASK=022 \
		 -e VERSION=stable \
		 -e BACKUP=yes \
		 -v /etc/localtime:/etc/localtime:ro \
		 hotio/suitarr

syntax in this case should be something like:

Code:

--mount 'type=volume,src=src_name,volume-driver=local,dst=/mnt,volume-opt=type=nfs,volume-opt=device=:/nfs-share,"volume-opt=o=nfs-server,vers=4,hard,timeo=600,rsize=1048576,wsize=1048576,retrans=2"'

make use of it as you like (: