SOLVED 11.2-U2.1 Docker Woes

[SavageAUS]

I am running an nginx reverse proxy in docker to share my apps (some docker, some jails, some standalone/VMs) out over 443.

Port conflicts on docker are easily handled in the docker run command (or in the Rancher Port settings for the service), you just pick a new non-conflicting port to use on the host and map the container internal port (can be 80 on every container if the app wants it) and map it to the selected port on the host. Optionally, you don’t even need to do that if you link the nginx container to the other containers and it can access the container by name and use the original port number without needing to publish it to the host at all.

I’m not really too keen to just write down everything I’ve done, but I will give you pointers to helpful resources and give tips on what to look at or how to get to the next stage as you go.

First is to install a container for nginx with 80 and/or 443 published and then port-forward 80 and/or 443 on your router to that container host.

Then you need to work on having an nginx config file that suits what you need (which will depend heavily on how you want to proceed.

I have been pulling my hair out trying to get this working.
Are you able to give me a working example of a reverse proxy please?
I've deleted all info in the "default"file (/etc/nginx/sites-available/default) and that is where i am trying to set it up.
This is an example of what I've been trying.

Code:

server {
  listen 80;

  server_name organizr.local;

  location / {
      proxy_set_header Host $host;
      proxy_pass http://192.168.1.99:80/;
      proxy_set_header X-Real-IP $remote_addr;
  }
}

I'm trying to re-route the ip address of organizr which uses port 80 to use organizr.local.
I'm likely going about this all wrong.

This is my setup
Rancher with portainer - 192.168.1.99:9000
Containers i will/want to use:
Emby (8096)
Jackett (9117)
Organizr (80)
Plex (32400, 3005, 8324, 32469, 1900, 32410, 32412, 32413, 32414)
Radarr (7878)
Sonarr (8989)
Tautulli (8181)
Transmission (9091)
Pi-Hole (53, 67, 80, 443)
Ombi (3579)

At the moment i am testing nginx on a ubuntu server (192.168.1.98) until i can work out how to place my config for nginx in a docker container. As you can see, so far there is only 1 port conflict i think.

 

[sretalla]

It looks like you're pretty close, but if that's your entire config file, you're missing the http section:

Code:

http {
  server {
    listen 80;

    server_name organizr.local;

    location / {
        proxy_set_header Host $host;
        proxy_pass http://192.168.1.99:80/;
        proxy_set_header X-Real-IP $remote_addr;
    }
  }
}

 

[SavageAUS]

Yes that was my entire conf. Thanks I’ll give that a try and let you know. So that should work now?

 

[sretalla]

until i can work out how to place my config for nginx in a docker container.

My volume config for my nginx container has these:

nginx:/etc/nginx/:ro
logs:/usr/share/nginx/logs/:rw
letsencrypt_certs:/usr/nginx/letsencrypt/:ro


The config goes in the first one.

So that should work now?

I don't know what other parameters organizr might want to play nice with the reverse proxy, but it should at least be closer to working.

 

[sretalla]

I just looked back at your docker ps output and see you have no ports published other than the rancher server itself (8080 for the GUI).

You will need to modify your container run statement for ombi or organizr or whatever container you plan to run to publish the ports and then use nginx to point at the public port you select for each container since nginx is currently on a separate host.

If you run nginx in docker also, that isn't necessary as long as you link the containers to the nginx one and call them by name.

All of that is much easier using the rancher GUI, but whatever works for you.

 

[SavageAUS]

I do plan on running nginx in a container once 1/ I get reverse proxy working 2/ I figure out how to edit/upload my config for proxy to the container.
This morning before work I trashed everything and installed fresh rancher/rancher, no portainer this time. So after work I’ll redo my vm for nginx and spin up a few containers and see how I go.
P.S I appreciate the help.

Is this your config mapped to the container?
nginx:/etc/nginx/:ro

 

[sretalla]

Is this your config mapped to the container?
nginx:/etc/nginx/:ro

Yes.

You create a volume on rancher-nfs called nginx, then put your config there.

You can either edit it from another docker container like cloud9 (change the ro to rw for that) or just from freenas with ee at the CLI.

As a general rule, you never want to copy anything into the container, only mount it, so the container is 100% disposable and can be replaced/upgraded with your data/config intact living somewhere else (preferably FreeNAS via rancher-nfs so you can snapshot it).

 

[SavageAUS]

So if i have a folder mapped to my docker host from my freenas box via nfs (/mnt/apps/nginx) and place my config (nginx.conf) in the previous nginx folder, where in the container would i mount it? (/config) (/etc/nginx)
I still dont think my config is right with using

Code:

http {
  server {
    listen 80;

    server_name organizr.local;

    location / {
        proxy_set_header Host $host;
        proxy_pass http://192.168.1.99:80/;
        proxy_set_header X-Real-IP $remote_addr;
    }
  }
}

As organizr.local isn't loading the organizr homepage

 

[sretalla]

Are you able to connect to it directly at http://192.168.1.99:80/ ? Last time I saw your docker ps, you didn't have the port published.

Can the docker host and your workstation resolve organizr.local?

where in the container would i mount it? (/config) (/etc/nginx)

As per the volume spec:
nginx:/etc/nginx/:ro (means mount the rancher-nfs volume nginx ... you could replace that part with /mnt/apps/nginx if that's where it is on the docker host rather than using rancher-nfs : at the location in the container /etc/nginx : in read-only)

 

[SavageAUS]

I think maybe what’s holding up the process is the tag I have to use to get Organizr to load (php-fpm)
Without that tag Organizr will not load and log shows chown errors and something about something already running on 127.0.0.0.
No nothing can resolve organizr.local yet.
Yes generally I can access Organizr at 192.168.1.99:80 but only when using the above tag.

 

[sretalla]

OK, so if you can get to it directly, then you should be able to reverse proxy it.

What happens when you use the IP of the nginx server in the config (you could add it after a space rather than replacing organizr.local) and then in the browser?

 

[SavageAUS]

Sorry for the delay but I’ve been testing my apps on Docker and so far I’m unimpressed. Sonarr and Radarr are extremely slow (to the point of being unresponsive) and transmission doesn’t even load. All using the defaults as posted on Docker hub with linuxserver/ images. Ive even tried giving the Docker host 4 cpus and 8Gb of ram (instead of the previous 2 cpus and 4Gb) and that didn’t help. I don’t know if it’s due to the rancher implementation in FreeNAS or because Docker accesses it’s files from nfs shares (on the same machine) so at this stage my thought process is to revert back to jails and just run pihole in a Ubuntu vm. Also in all my testing which may be relevant is that all container ports besides rancher are not actually listed in Docker ps but they are exposed and work with ip_addr: port (space included to null smiley)

Also to add the IP address I’m using for rancher is outside of my dhcp scope (same as my FreeNAS ip, not the same ip but set outside of the scope), it’s set with a static ip. Could being outside of the dhcp scope be causing the port publishing issues?

 

[sretalla]

Don't give up on it just yet.

Perhaps you can try installing a container to test things from within the docker environment itself in order to understand if it's a networking thing or the containers performing poorly. I would suggest consol/centos-xfce-vnc
So from that container, you can use VNC and then Chrome or Firefox to browse to the other container services (you will need to look at the container specific IPs rather than the docker host itself, as your consol container will not be able to see the public IP as having any other ports than it publishes itself).

You can see those IPs (typically in the 172 range) in portainer on the containers page.

Give it a try and see if the performance is more acceptable that way, then we can work on the details of why your network isn't working to get access from outside docker.

We can also consider how the storage is being presented... is it via rancher-nfs as we had discussed previously or is it something else?

 

[SavageAUS]

Ok, ive spun up a new rancher host, installed rancherUI, started a sonarr container and performance is way more acceptable.
No i am not using rancher-nfs i am using a very long command with all my mount points with "sudo ros config set mounts".
To be honest i'm not sure how to use rancher-nfs or how else to mount multiple paths to the docker host.
The performance in radarr is also increased accessing it via its docker ip 172 through the vnc client.

This is the command i just used to create my container.

Code:

docker run -d \
  --name=radarr \
  -e TZ=Australia/Adelaide \
  -p 7878:7878 \
  -v /mnt/apps/radarr:/config \
  -v /mnt/media/movies:/media/movies \
  -v /mnt/downoads/complete:/media/downloads \
  --restart unless-stopped \
  linuxserver/radarr

Previously i was using the rancher gui to create my containers, using the command has changed some things including networking as now my ports are exposed

Code:

[rancher@rancher ~]$ docker ps
CONTAINER ID        IMAGE                             COMMAND                  CREATED             STATUS              PORTS                                                                        NAMES
f311b6b6f3cb        linuxserver/transmission          "/init"                  4 minutes ago       Up 4 minutes        0.0.0.0:9091->9091/tcp, 0.0.0.0:51413->51413/tcp, 0.0.0.0:51413->51413/udp   transmission
03fc0d7a8f0b        linuxserver/radarr                "/init"                  16 minutes ago      Up 15 minutes       0.0.0.0:7878->7878/tcp                                                       radarr
94a407d868cb        linuxserver/sonarr                "/init"                  30 minutes ago      Up 30 minutes       0.0.0.0:8989->8989/tcp                                                       r-sonarr-2fdb9915
febe1cacd8f6        consol/centos-xfce-vnc            "/dockerstartup/vnc_…"   5 hours ago         Up 5 hours          0.0.0.0:5901->5901/tcp, 0.0.0.0:6901->6901/tcp                               mystifying_golick
2a2408c56a00        rancher/dns:v0.17.4               "/rancher-entrypoint…"   5 hours ago         Up 5 hours                                                                                       r-network-services-metadata-dns-1-7ff949f8
feb618c3c868        rancher/net:v0.13.17              "/rancher-entrypoint…"   5 hours ago         Up 5 hours                                                                                       r-ipsec-ipsec-router-1-7f58a400
01604168e1bd        rancher/net:v0.13.17              "/rancher-entrypoint…"   5 hours ago         Up 5 hours                                                                                       r-ipsec-ipsec-connectivity-check-1-81dc9f95
7ee35e32b9f3        rancher/healthcheck:v0.3.8        "/.r/r /rancher-entr…"   5 hours ago         Up 5 hours                                                                                       r-healthcheck-healthcheck-1-f847c508
648a8c7eaf01        rancher/net:holder                "/.r/r /rancher-entr…"   5 hours ago         Up 5 hours                                                                                       r-ipsec-ipsec-1-a2069c58
f63d8534beb6        rancher/metadata:v0.10.4          "/rancher-entrypoint…"   5 hours ago         Up 5 hours                                                                                       r-network-services-metadata-1-1451ba11
f75867c46b37        rancher/scheduler:v0.8.6          "/.r/r /rancher-entr…"   5 hours ago         Up 5 hours                                                                                       r-scheduler-scheduler-1-04ca078f
64779054b6aa        rancher/net:v0.13.17              "/rancher-entrypoint…"   5 hours ago         Up 5 hours                                                                                       r-ipsec-cni-driver-1-e9948f0d
56ef93ca3e69        rancher/network-manager:v0.7.22   "/rancher-entrypoint…"   5 hours ago         Up 5 hours                                                                                       r-network-services-network-manager-1-5057b227
00572c92f443        rancher/agent:v1.2.11             "/run.sh run"            5 hours ago         Up 5 hours                                                                                       rancher-agent
3db5512b32d7        rancher/server                    "/usr/bin/entry /usr…"   5 hours ago         Up 5 hours          3306/tcp, 0.0.0.0:8080->8080/tcp                                             rancher
[rancher@rancher ~]$

Networking for the containers now say "default" where as previously i was setting them to "host".


EDIT:
Ok so the performance issue must be the way im mounting my nfs shares as when i create a container with no volumes its super zippy.
So i need to look into how to create volumes with rancher-nfs.

 

[SavageAUS]

To create a volume in rancher I’m lost. What does the key pair mean?
Is that where I put the nfs server ip and share?

 

[sretalla]

So, you can have a look at my post in the other thread : https://www.ixsystems.com/community/threads/need-some-help-with-setting-up-nfs-in-rancheros.71590/

What I show there is how to add a volume on another NFS share than the one you define when you set up rancher-nfs initially (where you already specify the "default NFS share")

If you specify nothing other than a name (no key-value pairs), then the volume (sub-directory) is created on the default NFS share.

If you specify the key-value pairs like I indicated in that other post, it creates the volume (sub-dir) on the specified NFS.

I hope that is clear enough.

Did you get it to work on the default NFS first?

 

[SavageAUS]

So, you can have a look at my post in the other thread : https://www.ixsystems.com/community/threads/need-some-help-with-setting-up-nfs-in-rancheros.71590/

What I show there is how to add a volume on another NFS share than the one you define when you set up rancher-nfs initially (where you already specify the "default NFS share")

If you specify nothing other than a name (no key-value pairs), then the volume (sub-directory) is created on the default NFS share.

If you specify the key-value pairs like I indicated in that other post, it creates the volume (sub-dir) on the specified NFS.

I hope that is clear enough.

Did you get it to work on the default NFS first?

My default nfs share in the “driver” is “/”without the “” as I have a few different pools on my FreeNAS box that I need to expose to containers. /mnt/Tank, /mnt/DownloadsSSD, /mnt/JailsSSD.
Would I set the default to /mnt?
Yes the link to your other thread made creating volumes make sense but for me it didn’t work.

Code:

Error (Failed to mount volume sonarr. Driver: rancher-nfs. Status code: 500. Status: 500 internal server error. Error Message: Failed mount -o ,nfsvers=4 192.168.1.2://sonarr /var/lib/rancher/volumes/rancher-nfs/sonarr)

Volume was of course sonarr that I created and it should have pointed to /mnt/JailsSSD/Apps/Sonarr:/config

 

[sretalla]

The basePath of the NFS shares needs to be specified in the NFS share on the FreeNAS side also, so you will probably not be able to use one NFS share with "/mnt/" as the basePath. You'll need to be more specific (I think NFS helps with this too, telling you that you need to keep your NFS shared paths in the same dataset for each NFS share).

Look at the NFS settings (not the share, but under the service) and ensure you have specified NFSv4 there (and restart the service if not).

 

[sretalla]

Also, to be clear, setting the basepath to / will be attempting to create root-level subfolders as your volumes (on the FreeNAS boot drive!). I suspect that will fail due to permissions and the setting on the NFS share itself (hopefully).

 

[SavageAUS]

NFSv4 is enabled/checked on both FreeNAS and rancher-nfs. I just tried setting the driver default to /mnt/JailsSSD/Apps and then creating a volume of /Sonarr and it mounts but my sonar config isn’t applied. So it’s a step on the right direction. Yes the share exists of /mnt/JailsSSD/Apps. Which basePath are you referring too? The one for the nfs driver or the exportBase? <— that’s the one I tried with /Sonarr in the volume creation.