How to set up an nginx reverse proxy with SSL termination in a jail

How to set up an nginx reverse proxy with SSL termination in a jail

samuel-emrys

Contributor
Joined
Dec 14, 2018
Messages
136
samuel-emrys submitted a new resource:

How to set up an nginx reverse proxy with SSL termination in a jail - Self host multiple services on the internet with HTTPS

Instructions are available here: https://www.samueldowling.com/2020/01/18/nginx-reverse-proxy-freenas-ssl-tls/

Recently I went through the process of standing up a Bitwarden server. As this is the second service that I plan to make available externally to my LAN, I had to set up a reverse proxy. I couldn't find any instructions that dealt with exactly what I did, so I thought this would be a good opportunity to write something that might be useful to others.

The guide...

Read more about this resource...
 
Joined
Jan 27, 2020
Messages
577
@samuel-emrys very good write-up!
Do you have any directions on how to set-up a jitsi-ubuntuVM in that configuration. jitsi needs TCP and UDP to work.
I have jitsi along with nginx inside a ubuntu VM installed and try to proxy - through a nginx revers proxy jail - my external domain meet.mydomain.com to the ubuntu VM (192.168.x.x).
Any ideas?
 

samuel-emrys

Contributor
Joined
Dec 14, 2018
Messages
136
@samuel-emrys very good write-up!
Do you have any directions on how to set-up a jitsi-ubuntuVM in that configuration. jitsi needs TCP and UDP to work.
I have jitsi along with nginx inside a ubuntu VM installed and try to proxy - through a nginx revers proxy jail - my external domain meet.mydomain.com to the ubuntu VM (192.168.x.x).
Any ideas?
@mistermanko unfortunately I have no experience with Jitsi or it's requirements, and troubleshooting this any further without diagnosing the issues you're actually running in to first is probably going to be impossible. Having said this, some quick research returned this as the template nginx configuration for Jitsi:

My read of this is that it should be as straight forward as following my instructions, and obviously if Jitsi handles its own SSL then use https://<jitsi ip> in the proxy_pass statement instead of using just http. Are you having a specific issue?
 
Joined
Jan 27, 2020
Messages
577
@mistermanko unfortunately I have no experience with Jitsi or it's requirements, and troubleshooting this any further without diagnosing the issues you're actually running in to first is probably going to be impossible. Having said this, some quick research returned this as the template nginx configuration for Jitsi:

My read of this is that it should be as straight forward as following my instructions, and obviously if Jitsi handles its own SSL then use https://<jitsi IP> in the proxy_pass statement instead of using just http. Are you having a specific issue?
The challenge is to get a coturn server to work with the proxy along with nextcloud Talk or jitsi.
Both make use of STUN and TURN services so nginx needs to work with together with coturn.
In theory this shouldn't be much harder than just setting up a coturn instance - preferably on the same instance as nginx - and give ngnix instructions on redirecting traffic to coturn.
But I've yet to see someone actually done it, which makes me wonder if anybody ever who installed nextcloud in a jail tried to use nextcloud talk.
I'm reading through this atm: https://beta.cent.co/+k96cmv
But I'm struggling to bring the pieces together to get ngnix - coturn - nextcloud-talk to work.
 
Joined
Jan 27, 2020
Messages
577
I actually got a turn server working. Using a coturn ubuntu VM and directing traffic through a nginx revers proxy in a jail.
If someone is interested I would write up a guide and provide the resources to read up on the topic. It's not that hard.
 

samuel-emrys

Contributor
Joined
Dec 14, 2018
Messages
136
I actually got a turn server working. Using a coturn ubuntu VM and directing traffic through a nginx revers proxy in a jail.
If someone is interested I would write up a guide and provide the resources to read up on the topic. It's not that hard.
I'd be interested to see your reverse proxy settings if you had to change them substantially; can you post them here?
 
Joined
Jan 27, 2020
Messages
577
I just added the turn.mydomain.de subdomain as as vdomain in nginx.
I'm still testing the config, as I got mixed results in NC Talk via mobile network or with people across the globe.
I'll report back here if get improvements done.
 

Lessbones

Dabbler
Joined
Oct 17, 2016
Messages
22
I'm also attempting to do this--

I've got a Seafile server hosted at my main domain (https://xxxx.com) and i'm trying to host jitsi at https://jitsimeet.xxxx.com, but immediately i hit a snag with the nginx reverse proxy limiting the size of uploads to my seafile server at which point i reverted back to how I had it set up before. Also, it would be great to be able to host jitsi in a jail instead of a VM, but that would be for another topic. Anyway, i'll be following here to see if you have any luck, and I'll post myself if I get it working finally. Gonna spend some time on it today once the sun goes down--
 

samuel-emrys

Contributor
Joined
Dec 14, 2018
Messages
136
I'm also attempting to do this--

I've got a Seafile server hosted at my main domain (https://xxxx.com) and i'm trying to host jitsi at https://jitsimeet.xxxx.com, but immediately i hit a snag with the nginx reverse proxy limiting the size of uploads to my seafile server at which point i reverted back to how I had it set up before. Also, it would be great to be able to host jitsi in a jail instead of a VM, but that would be for another topic. Anyway, i'll be following here to see if you have any luck, and I'll post myself if I get it working finally. Gonna spend some time on it today once the sun goes down--

I believe this is easily rectified by setting the `client_max_body_size` directive in your `nginx.conf` file to something appropriate for your use case. An example:

Code:
http {
    ...
    client_max_body_size 10G;
    ...
    server {
        listen 80 default server;
        ...
    }
    ...
}


HTH
 
Joined
Jan 27, 2020
Messages
577
I just added the turn.mydomain.de subdomain as as vdomain in nginx.
I'm still testing the config, as I got mixed results in NC Talk via mobile network or with people across the globe.
I'll report back here if get improvements done.
I want to give an update on my attempt here.
So far it was not possible to host a turnserver behind a nginx reverse proxy. All successful connections via Nextcloud Talk were handled by the STUN service rather than the self hosted TURN service. Nextcloud provides its own STUN server (stun.nextcloud.de).
I somehow believe my Router cannot forward UDP-traffic through nginx, the stream derivate in the nginx config is not sufficient as it seems - first and foremost because it interferes with the nextcloud vdomain. I was not able to separate both accordingly because the stream module is only working globally.
Further readings:

You can test your turnserver config here: https://test.webrtc.org/
 

samuel-emrys

Contributor
Joined
Dec 14, 2018
Messages
136
I want to give an update on my attempt here.
So far it was not possible to host a turnserver behind a nginx reverse proxy. All successful connections via Nextcloud Talk were handled by the STUN service rather than the self hosted TURN service. Nextcloud provides its own STUN server (stun.nextcloud.de).
I somehow believe my Router cannot forward UDP-traffic through nginx, the stream derivate in the nginx config is not sufficient as it seems - first and foremost because it interferes with the nextcloud vdomain. I was not able to separate both accordingly because the stream module is only working globally.
Further readings:

You can test your turnserver config here: https://test.webrtc.org/

This issue provides some helpful hints on how to get this working. As you've discovered, the stream directive seems to be the key to getting this to work. I'm not sure what issue you're running in to, but you could try something like this (note I haven't tested it):

- Create a directory named `streams`, and a file for your cloud vdomain, i.e. `streams/cloud.domain.com.conf`. Populate it with the advice of the above thread:

Code:
server {
    listen 3478 udp;
    # server_name cloud.domain.com; # Edit: Invalid syntax - can't have a server_name directive inside a stream directive
    proxy_pass cloud.domain.com:3478
}


- Update your `nginx.conf` as follows:

Code:
stream {
    # Import server blocks for streams
    include "streams/*.conf";
}
http {
    ...

    # Import server blocks for subdomains
    include "vdomains/*.conf";
}


- Make sure your router forwards the UDP traffic on the port you're interested in to the reverse proxy.

Of course, none of this will work if your router is unable to forward the traffic - I wasn't really sure which was the issue in your last post. Was it the router not forwarding or the inability to separate the vdomain from the http block? Well, the above should solve the latter, anyway. Hope this helps.
 
Last edited:
Joined
Jan 27, 2020
Messages
577
Of course, none of this will work if your router is unable to forward the traffic - I wasn't really sure which was the issue in your last post. Was it the router not forwarding or the inability to separate the vdomain from the http block? Well, the above should solve the latter, anyway. Hope this helps.

Good advise, thank you. Unfortunately this is not possible.

Code:
"server_name" directive is not allowed here in /usr/local/etc/nginx/streams/cloud.mydomain.de.conf:3


As is described also here: https://stackoverflow.com/questions/45227491/nginx-server-name-inside-stream-block-possible

It may be possible with something like ssl_preread, but that seems overly complicated.
 

samuel-emrys

Contributor
Joined
Dec 14, 2018
Messages
136
Good advise, thank you. Unfortunately this is not possible.

Code:
"server_name" directive is not allowed here in /usr/local/etc/nginx/streams/cloud.mydomain.de.conf:3


As is described also here: https://stackoverflow.com/questions/45227491/nginx-server-name-inside-stream-block-possible

It may be possible with something like ssl_preread, but that seems overly complicated.
Ahhh, yes of course, this makes sense. Nginx uses the http 'Host' header to match the `server_name` directive. This header is not present in raw TCP/UDP headers, so it can't match it. Yeah, so your only solution seems to be to use the stream directive without a `server_name`:


Code:
stream {
    server {
        listen 3478 udp
        proxy_pass cloud.domain.com:3478
    }
}
http {
    ...

    # Import server blocks for subdomains
    include "vdomains/*.conf";
}


Have you tried just doing this? Obviously the implications of not having the server_name directive is just that you can't have multiple applications use the same port and discriminate between them; so you would *only* be able to use your turn server on this port. I'm not sure this is much of an issue since it appears (if the above example port is actually correct) to use a non standard port. If you needed multiple turn servers you'd just have to configure them to use different ports.
 
Joined
Jan 27, 2020
Messages
577
Have you tried just doing this?
This seems to work at least in such a way that I see incoming connections on my coturn-server. My tests are still not consistent, but I found out that trying to establish a connection via my mobile network is 100% not working. I suspect my carrier to filter non standard ports. When using a VPN service with obfuscation enabled it works most of the time via mobile network.

Nevertheless, your recommendation seems to work. Thank you!
It's crucial to import the stream modul in the nginx.conf because pkg does not activate it by default.
This is what my nginx.conf looks like for a reverse proxy setup with wordpress, nextcloud, plex(locally) and a coturn-turnserver, all in subdomains.

Code:
load_module /usr/local/libexec/nginx/ngx_stream_module.so;

worker_processes  auto;

        events {
                worker_connections  1024;
                multi_accept on;
                }


        stream {
                server {
                        listen 3478 udp;
                        # local adress of coturn instance inside ubuntu VM
                        proxy_pass 192.168.xxx.xxx:3478;
                        }
                }

        http {
                include mime.types;
                default_type application/octet-stream;
                sendfile on;
                keepalive_timeout 65;
                # set max body size to zero if you encounter max-file-size errors when uploading files to nextlcoud
                client_max_body_size 0;

                # Redirect all HTTP traffic to HTTPS

                server {
                        listen 80 default_server;
                        listen [::]:80 default_server;

                        return 301 https://$host$request_uri;
                        }

                        # Import server blocks for all subdomains
                        include "vdomains/*.conf";
                }
 

profzelonka

Explorer
Joined
Mar 29, 2020
Messages
67
@samuel-emrys I'm trying to setup a reverse proxy using your guide but without setting up a wildcard certificate (because I don't understand which "certbot-dns" to go with. My domain is on a shared webhost so I guess "py37-certbot-dns-cloudflare" won't work since cloudflare tells me my domain is already used by cpanel. ??)

Since your guide is for the wildcard I tried striping those steps out. Is this what my cloud.mydomain.com.conf should look like?

Code:
server {

        listen 443 http2;


        server_name cloud.mydomain.com;

        access_log /var/log/nginx/cloud.access.log;

        error_log /var/log/nginx/cloud.error.log;


        location / {

                proxy_pass http://192.168.9.2;

        }

}


I'm not sure if this is set up wrong or if I'm running into issues with my network again.
 

samuel-emrys

Contributor
Joined
Dec 14, 2018
Messages
136
@samuel-emrys I'm trying to setup a reverse proxy using your guide but without setting up a wildcard certificate (because I don't understand which "certbot-dns" to go with. My domain is on a shared webhost so I guess "py37-certbot-dns-cloudflare" won't work since cloudflare tells me my domain is already used by cpanel. ??)

Since your guide is for the wildcard I tried striping those steps out. Is this what my cloud.mydomain.com.conf should look like?

Code:
server {

        listen 443 http2;


        server_name cloud.mydomain.com;

        access_log /var/log/nginx/cloud.access.log;

        error_log /var/log/nginx/cloud.error.log;


        location / {

                proxy_pass http://192.168.9.2;

        }

}


I'm not sure if this is set up wrong or if I'm running into issues with my network again.
Well, the first question is, what are you trying to achieve? Are you still trying to terminate SSL at the reverse proxy but just use a concrete subdomain? i.e. subdomain.yourdomain.com instead of *.yourdomain.com?

If so, then the advantages of having a *.cert.conf file in snippets/ is lost. You would just add the ssl_certificate, ssl_certificate_key and ssl_trusted_certificate locations to your vdomain file for the host. i.e.:

Code:
server {
        listen 443 ssl http2;

        server_name subdomain.domain.com;
        access_log /var/log/nginx/subdomain.access.log;
        error_log /var/log/nginx/subdomain.error.log;

        ssl_certificate /usr/local/etc/letsencrypt/live/subdomain.domain.com/fullchain.pem
        ssl_certificate_key /usr/local/etc/letsencrypt/live/subdomain.domain.com/privkey.pem
        ssl_trusted_certificate /usr/local/etc/letsencrypt/live/subdomain.domain.com/chain.pem
        include snippets/ssl-params-intermediate.conf;

        location / {
                include snippets/proxy-params.conf;
                proxy_pass http://192.168.9.2;
        }
}

(This assumes you're still obtaining a certificate using certbot, but for subdomain.domain.com instead of *.domain.com)

Or, you could put these parameters into a snippets/subdomain.domain.com.cert.conf file and include that like I did previously. The important bit here is specifying the certificate files appropriate for the domain you obtain SSL/TLS keys/certificates for. It seems a bit strange that you're getting "already used" errors; do you own this domain? Regardless of whether it's a shared host, you should be able to get a wildcard certificate if you own the domain, and haven't set any CAA records for other certificate providers. Might be worth having a look at your Cloudflare configuration to see if anything has been added here from when you've previously configured it.
 

profzelonka

Explorer
Joined
Mar 29, 2020
Messages
67
I'm currently not using certbot. What I'm trying to achieve: I have nextcloud with it's own LetsEncrypt cert within it's jail, and another web-based jail where I will set up it's own cert as well. I'm simply trying to create a reverse proxy to allow web access to both jails from my FQDN.

Does my set up above look correct for this case? I'm unable to access the site once I switch port forwarding to my reverse proxy jail, and it may be an issue with my modem/DNS, unless my setting above is wrong?
 

samuel-emrys

Contributor
Joined
Dec 14, 2018
Messages
136
I'm currently not using certbot. What I'm trying to achieve: I have nextcloud with it's own LetsEncrypt cert within it's jail, and another web-based jail where I will set up it's own cert as well. I'm simply trying to create a reverse proxy to allow web access to both jails from my FQDN.

Does my set up above look correct for this case? I'm unable to access the site once I switch port forwarding to my reverse proxy jail, and it may be an issue with my modem/DNS, unless my setting above is wrong?

If you don't want SSL termination in the jail, you'd probably want something like this:

Code:
server {
        listen 443 ssl http2;

        server_name subdomain.domain.com;
        access_log /var/log/nginx/subdomain.access.log;
        error_log /var/log/nginx/subdomain.error.log;

        location / {
                include snippets/proxy-params.conf;
                proxy_pass https://192.168.9.2;
        }
}


So, your config is missing the proxy-params.conf include, and you don't proxy_pass to https (since the server you're proxying to handles its own SSL config). This is a departure from what I've configured though, so it's not tested - you might have to massage it a bit
 
Last edited:

profzelonka

Explorer
Joined
Mar 29, 2020
Messages
67
Code:
nginx: [emerg] no "ssl_certificate" is defined for the "listen ... ssl" directive in /usr/local/etc/nginx/vdomains/[subdomain].[domain].com.conf:1
nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
/usr/local/etc/rc.d/nginx: WARNING: failed to start nginx


So I removed "ssl" after 433 and nginx starts now, but when I port forward 443 and 80 to my reverse proxy jail IP, I can't reach my nextcloud jail from it's subdomain.domain.com url.

And again it could be that this is a network issue because I keep urnning into some sort of network issues. Just wanted to see if you knew and could confirm that this should be working.

I don't understand what "http2" does but I tried without it as well and same result. As well as changing 443 to 80.
 
Last edited:

samuel-emrys

Contributor
Joined
Dec 14, 2018
Messages
136
Code:
nginx: [emerg] no "ssl_certificate" is defined for the "listen ... ssl" directive in /usr/local/etc/nginx/vdomains/[subdomain].[domain].com.conf:1
nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
/usr/local/etc/rc.d/nginx: WARNING: failed to start nginx


So I removed "ssl" after 433 and nginx starts now, but when I port forward 443 and 80 to my reverse proxy jail IP, I can't reach my nextcloud jail from it's subdomain.domain.com url.

And again it could be that this is a network issue because I keep urnning into some sort of network issues. Just wanted to see if you knew and could confirm that this should be working.

I don't understand what "http2" does but I tried without it as well and same result. As well as changing 443 to 80.

Yeah, you'll probably need to remove all references to ssl. This isn't a use case I've had to deal with, so I'd probably have to spend some time with it to get it working and understand all of the factors. You could try removing the X-Forwarded-Ssl `proxy_set_header` directive in snippets/proxy-params.conf. Otherwise, your best bet is to inspect the requests manually (open developer tools, network tab in your browser), and refreshing the page to see what exactly is being sent/received, and researching what each of the headers does and whether you need it for your use case. You can then add and remove headers in the proxy-params.conf file as required
 
Top