vms and freenas questions

[zdenkers]

I've put together some hardware and now i just trying to decide how best to have freenas handle my needs. My thought is that i can either use plugins/jails and host the various services that way or i can use a single ubuntu vm to host docker. I am significantly more comfortable in the docker realm, i know how to handle reverse proxy of the services and all the services i intend to host have official docker images. The two main services i intend to host, zoneminder and plex, both have plugins already available. my hardware configuration is:

epyc 7402p
8x8tb 7200rpm seagate exos drives.
wd blue OS
samsung 970 evo plus (for fast storage of either the vm or jails/plugins)
intel 900p 280gb (SLOG)
mellanox 10gbe NIC

Pros and cons of jails vs the vm with docker route?

 

[sretalla]

Watch out on the WD blue for the OS... if that's the blue SSD, you may find it has the same TRIM bug as the green.

Jails:
+ Direct/native access to FreeNAS datasets via nullfs mounts (full speed, direct FreeBSD access).
+ Lightweight memory and file footprint (runs on top of already running FreeNAS OS).
- Jails difficult to manage if you want to stay on the latest versions of apps
- Often difficult to install and make it work (depending on your level of understanding)

VM + Docker containers
+ Latest versions usually available quickly and easy to update/install (if you do container storage properly to something like NFS for all unique data... config + app data)
+ Easy to manage (even more so if you use Portainer or Rancher server... or kubernetes if you're really into it).
- Storage access via network mapping (NFS probably best), so not at local access speed
- High resouce overhead of VM OS and memory consumption to make docker available... comparatively low against 1 VM per app though.


You don't mention the amount of RAM, so it's hard to see if a SLOG will help or not. I presume the RAM will be less than 128GB? SLOG may help a little, don't expect miracles.

 

[zdenkers]

Watch out on the WD blue for the OS... if that's the blue SSD, you may find it has the same TRIM bug as the green.

Its a WD blue nvme ssd, i didn't realize there were problems with those?

 

[zdenkers]

.

Jails:
+ Direct/native access to FreeNAS datasets via nullfs mounts (full speed, direct FreeBSD access).
+ Lightweight memory and file footprint (runs on top of already running FreeNAS OS).
- Jails difficult to manage if you want to stay on the latest versions of apps
- Often difficult to install and make it work (depending on your level of understanding)

VM + Docker containers
+ Latest versions usually available quickly and easy to update/install (if you do container storage properly to something like NFS for all unique data... config + app data)
+ Easy to manage (even more so if you use Portainer or Rancher server... or kubernetes if you're really into it).
- Storage access via network mapping (NFS probably best), so not at local access speed
- High resouce overhead of VM OS and memory consumption to make docker available... comparatively low against 1 VM per app though.


You don't mention the amount of RAM, so it's hard to see if a SLOG will help or not. I presume the RAM will be less than 128GB? SLOG may help a little, don't expect miracles.

i have 256gb of ecc 2933mhz memory in the system.

for the storage, i assume the filesystem can be mounted into the vm the same way as a jail. If thats true docker can mount that same mount and access will be identical in that way. also it will only really be one VM docker will handle lightweight virtualization of each app. But it is still going to be significantly heavier than jails like you say. Also i may not fully understand how the access to the datasets will work and thats the biggest reason i started this thread.

 

[sretalla]

I have seen nothing to confirm it for nvme, but the 3DNAND Sata ssd line seems to suffer from it.

 

[sretalla]

i assume the filesystem can be mounted into the vm the same way as a jail

No. Not possible. This was a folly of the much maligned FreeNAS Corral (version 10). The project was abandoned just days after reaching release due to the slowness of the filesystem integration component and it's lack of development support on FreeBSD.

Like I said, NFS is probably the best... you could try SMB or other cloud storage systems like minio.

 

[zdenkers]

No. Not possible. This was a folly of the much maligned FreeNAS Corral (version 10). The project was abandoned just days after reaching release due to the slowness of the filesystem integration component and it's lack of development support on FreeBSD.

So sounds like if i want fast direct access to the pool i need to go the jail route.

 

[sretalla]

So sounds like if i want fast direct access to the pool i need to go the jail route.

NFS over the host to guest bridge can theoretically be 10 Gbits, but I see plenty of threads lamenting the inability of NFS to get to those lofty heights, very rarely ones where folks are happy about it. Maybe it shouldn't be a showstopper if most of your apps are dealing in speeds under 1 Gbit.

I personally prefer the jail route to simplify my life in terms of concern about bottlenecks on the disk.

 

[zdenkers]

NFS over the host to guest bridge can theoretically be 10 Gbits, but I see plenty of threads lamenting the inability of NFS to get to those lofty heights, very rarely ones where folks are happy about it. Maybe it shouldn't be a showstopper if most of your apps are dealing in speeds under 1 Gbit.

I personally prefer the jail route to simplify my life in terms of concern about bottlenecks on the disk.

unfortunately i suspect i will be saturating my 10gbe nic so i think i better go the jail route. I will just have to figure out how to handle the reverse proxy from a plugin. i was just hoping to be able to use traefiq which has incredible docker support.

 

[Patrick M. Hausen]

Full local posix semantics is definitely a point for jails and nullfs mounts. I already complained to jkh about the lack of those in Corral. What I assume would be an interesting development project is a horizontal slicing of the fs at the vnode layer. If both upper and lower layer have similar (i.e. posix) semantics, that should be quite simple and efficient. One could start with FreeBSD guest - FreeBSD host first and see how that goes.

That would be great for use with Vagrant on Macs or Linux, too.

Kind regards,
Patrick

 

[Patrick M. Hausen]

I will just have to figure out how to handle the reverse proxy from a plugin.

I don't use plugin jails because I can update regular ones at my own pace. I create one jail per application - simplifies management a lot. For a reverse proxy I use Apache24 and dehydrated. I can post a rough HowTo and config snippets if people are interested.

Patrick

 

[zdenkers]

I don't use plugin jails because I can update regular ones at my own pace. I create one jail per application - simplifies management a lot. For a reverse proxy I use Apache24 and dehydrated. I can post a rough HowTo and config snippets if people are interested.

Patrick

sorry i said plugin but i meant jail haha. i will probably go the plugin route for apps that exist simply because i am not very literate with at the OS level. But i don't think there are any official plugins for reverse proxy as of now. devops/docker i have figured out though lol.

 

[danb35]

...and for other options on running a reverse proxy in a jail, see:

Reverse Proxy using Caddy (with optional automatic TLS)

Many users install a variety of web applications in jails on their FreeNAS servers, and often those applications run on non-standard ports like 6789, 8181, 7878, etc. These port numbers are far from intuitive, and the applications often either...

www.ixsystems.com

How to set up an nginx reverse proxy with SSL termination in a jail

Instructions are available here: https://www.samueldowling.com/2020/01/18/nginx-reverse-proxy-freenas-ssl-tls/ Recently I went through the process of standing up a Bitwarden server. As this is the second service that I plan to make available...

www.ixsystems.com

 

[zdenkers]

...and for other options on running a reverse proxy in a jail, see:

Reverse Proxy using Caddy (with optional automatic TLS)

Many users install a variety of web applications in jails on their FreeNAS servers, and often those applications run on non-standard ports like 6789, 8181, 7878, etc. These port numbers are far from intuitive, and the applications often either...

www.ixsystems.com

How to set up an nginx reverse proxy with SSL termination in a jail

Instructions are available here: https://www.samueldowling.com/2020/01/18/nginx-reverse-proxy-freenas-ssl-tls/ Recently I went through the process of standing up a Bitwarden server. As this is the second service that I plan to make available...

www.ixsystems.com

Thanks that nginx guide is going to be invaluable.