Has anyone tried virtualising unraid into Truenas scale?

[marshalleq]

Hi all, I'm just wondering if anyone has tried this? The reason I am back here again looking at truenas scale is simply just because unraid is extremely poor at the basic stuff you get with all other NAS's. And from time to time I get annoyed with it and want to shift. If I recall correctly the reason I didnt go to Truenas previously was because the docker interface was really clunky, vm's were quite inflexible and most responses to questions for help came from one individual that was quite unhelpful. Also I remember it was extremely limiting, where unraid lets you use any docker image, truenas at the time did not - I think it was meant to but it didnt work and the interface was so clunky you'd spend hours going over the same thins to no avail. Truecharts at the time did not have enough apps and good suggestions were rejected,so that was fairly limiting and I've had further rejections in time gone by - I honestly don't see the point in Trucharts, perhaps someone can enlighten me because, much time has passed and things have undoubtedly moved on since then.

Truenas remains for me probably the most polished NAS when it comes to the basics, file permissions, backups, replication, hardware failures etc. For example unraid recently included ZFS as standard but unfortunately made it worse than when it was only a plug-in. The implementation now makes it so that you have to stop your entire array to replace a failed disk (among other things), which is all due to the licensing being tied to a running array with a number of Licenced discs. They've just announced a new licensing model which is also based on a number of discs so I don't see this problem changing any time soon. And as is often true in life there's no perfect answer for anything but at this point I'm pretty tired with unraid and it's anti NAS abilities so I'm back here again.

There really isn't a reason why Truenas scale shouldn't be able to do everything unraid does and it has been a few years since I've been in so here I am having another look. Of course the core of each product is aimed at a different type of user and unraid serves its base very well. But, in a past life I did a lot of these things in large enterprise orgs, so I know what I'm missing.

To date my opinion is that for the management interface of docker and VM's unraid does a better job than everyone else - and how their App Store works leads the way. But for everything else truenas wins - except perhaps that my impression is proxmox is better as a stand alone hypervisor.

So I was thinking for docker and perhaps some things like reverse proxy a good interim is virtualise existing unraid for docker into TrueNAS, import all my ZFS disks (created in command line so should be pretty standard - I.e no unraid funkiness), perhaps migrate the vm's to TrueNAS native (currently qcow images which I like but I recall TrueNAS was not very flexible in this regard i.e cant over provision) and go from there?

Failing that I guess it's proxmox.

Thoughts?

Thanks.

Marshalleq

 

[artlessknave]

er. i admittedly haven't used scale all that much...but im *pretty sure* it's docker is just standard docker? I have installed docker containers from...the repos? (is that even correct?)

even if it wasn't, if you just want docker, there are VM OSes designed to be minimal builds specifically for docker. why unraid?

 

[Arwen]

If I recall correctly, TrueNAS SCALE does not support Docker, (Kubernetes != Docker).

TrueNAS SCALE is still under heavy development. For example, some people are asking for things that are not available in the currently used Linux kernel, (6.1). But, are available in the next LTS kernel 6.6.

I have no comments or knowledge of how easy it would be to run UnRAID in a VM under TrueNAS SCALE.

You might spin up a VM for testing and checking out TrueNAS SCALE's GUI.

 

[chuck32]

If I recall correctly, TrueNAS SCALE does not support Docker, (Kubernetes != Docker).

Launching docker images
Yes it's not the same, and I don't use that feature, but one could argue that scale supports running docker containers.

en if it wasn't, if you just want docker, there are VM OSes designed to be minimal builds specifically for docker. why unraid?

This. I use an Ubuntu server VM to deploy and manage my docker containers. Yes this adds the overhead of the Ubuntu VM, but I'd so would virtualization of unraid.

@Arwen gave you the pointer that scale is still actively developed. Yes not everything may work as expected, as of 23.10.1 you cannot change VM settings without a workaround (was working before though). But other than that I would say, at least in my experience, using a Linux VM as the base for docker containers is pretty smooth.

 

[LarsR]

The equivalent to the unraid community plugins would be truecharts, and since truecharts ported the unraid community plugins to truenas, you should nearly have the same apps/container/plugins available if you add truecharts. But i must agree, that unraids ui for adding and editing container is friendlier then the way truenas does things...

 

[artlessknave]

(Kubernetes != Docker).

my understanding is that this is both correct but also comparing apples to oranges. kubernetes is a system for managing docker (or other containers). my understanding is that scale uses kubernetes as a management layer to docker, and so docker commands work, because its docker.

as such, it would be best to just install a test SCALE and see if things work before trying to construct a workaround.
as scale has gone from concept to mostly usable in a fairly impressively short time, there have been many changes.

 

[LarsR]

In previous Versions k3s was using docker as runtime, but the k3s Project switchted to Containerd as runtime and therefore docker has been removed from the base OS.

 

[artlessknave]

but the k3s Project switchted to Containerd as runtime and therefore docker has been removed from the base OS.

im not sure that applies the way you state. it looks like you would use kubernetes to manage containerd which manages parts of docker...?

I find the docker side of containerization incredibly confusing. it moves too fast and there is management of the management for the management of the management and it makes my head implode...

"Docker uses Containerd as its runtime for creating Containers from images. Essentially, it acts as an interface (API) that allows users to use Containerd to perform low-level functionality."

Docker vs Containerd: A Detailed Comparison

This Containerd vs Docker comparison reviews the key features and use cases to help Developers achieve effective utilization.

bluelight.co

 

[Arwen]

As I said, it is my understanding that the Docker specific commands were removed from TrueNAS SCALE.

This has caused some people who used those Docker commands to be (highly) annoyed that an upgrade broke their setup. (Which was using an unadvertised command, in an unsupported way...)


I have said multiple times before, in various wording;

TrueNAS SCALE is not the end all for converged Apps and Storage.


And for practical software development, you have to start somewhere with (limited and) well defined goals. One of those was having dedicated boot devices with upgrades by whole replacement of the OS using alternate boot environments. This makes the OS more firmware like, and unsupported changes disappear after upgrades to SCALE. But also helps keep left over crud from potentially causing problems in new boot environment.

 

[artlessknave]

As I said, it is my understanding that the Docker specific commands were removed from TrueNAS SCALE.

the fact that your understanding, my understanding, and the OPs understanding are so different is part of why I haven't really gotten into SCALE
even the "release" versions feel more like BETA.

 

[Arwen]

the fact that your understanding, my understanding, and the OPs understanding are so different is part of why I haven't really gotten into SCALE
even the "release" versions feel more like BETA.

Yes, in someways using SCALE in production seems like Beta software.

On the other side, SCALE was not just a TrueNAS Core on Linux. SCALE added applications in place of BSD Jails. While it might have made sense to state that the 2nd release, (or even 3rd), would add Apps, and have the first release concentrate on making stability and GUI changes as appropriate for Linux. (And next generation of GUI changes they can't make on stable Core.)


SCALE has gotten to the point that I believe you almost need 2 releases. The "current" which only gets stability and bug fixes, but no extra features, not even Samba ones, (unless its a bug or stability issue). Then "next" release gets all the other updates.

Then, at a certain point, the "current" becomes "old", and "next" becomes the "stable & current" one. With development taking part in a new "next" release.

Core is more or less the precursor to TrueNAS Enterprise, so we community users of Core were the testers. But, we also got stuck in that anything that could reduce the Enterprise version's stability was not considered for update. that meant that GUI improvements were generally not accepted. (Barring "The Release That Shall Not Be Named!".)

 

[artlessknave]

"The Release That Shall Not Be Named!"

aw I was trying to forget that! I've been using truenas since it was a freshly born freenas split out from freenas 7.
out of all the truenas versions, I've defnitely found scale the least.....stable. I am working on getting some test scale up and running again to poke around with it. I am thinking of trying to mess with docker again, so if scale is not actually docker....I dunno, maybe that will just become debian instead.
my main storage and its backup are unlikely to leave Core, but I have a stupid amount of hardware to play around with.

 

[ThisTruenasUser]

Interesting. I am keen to know if anyone has been running proxmox as a virtual machine in truenas scale with nested virtualisation.
Stable?

So proxmox -has: windows 11 tpm emulation, easily backup & restore VM , and firewall options.
Truenas scale - none of the above.

Also If you have proxmox virtualised in truenas, as opposed to the opposite - no issues with passing though disks or need for hardware passthrough of flashed raid cards.

 

[probain]

Interesting. I am keen to know if anyone has been running proxmox as a virtual machine in truenas scale with nested virtualisation.
Stable?

So proxmox -has: windows 11 tpm emulation, easily backup & restore VM , and firewall options.
Truenas scale - none of the above.

Also If you have proxmox virtualised in truenas, as opposed to the opposite - no issues with passing though disks or need for hardware passthrough of flashed raid cards.

I did this, not with proxmox. But with XCP-NG.
It was not stable. Not at all. This is how I learned the real difference between type 1&2 hypervisors.

 

[marshalleq]

Interesting reading. I have to say, I have never understood why someone would want to virtualise a system that virtualises systems.

In the case of unraid, I was asking only because that's what I already had and it would offer some level of comfort to know I could spin up all my apps in a VM if I got into strife, rather than uninstall Scale and Reinstall Unraid. I would argue Unraid has the best docker / app capability for home use of any single solution I've ever seen. That's their target market so it's not surprising. Where it fails is everything that TrueNAS is good at. I just wish the two would learn from each other a bit - while not compromising their target markets of course.

Anyway, I have fully migrated to TrueNAS scale now, and am quite happy with it. I get the stability of the basics that I'm looking for, I get proper ZFS back which used to exist on unraid until they integrated it into their GUI - (perhaps they will fix that eventually but I'm not really convinced).

In scale the app implementation is sorely lacking, but I like everything else. It feels a lot smoother. Bootup and shutdown times are 100 times better than unraid, virtualisation is far more solid, I love not having to use a USB key to boot from and love having mirrored boot drives.

I always hated the UI for the unraid networking. But I really dislike a few TrueNAS features, like only being able to allocate high ports, I am yet to be impressed by any of the official or TrueCharts containers too. TrueCharts seem to be extremely complicated in an unnecessary way, as much as my corporate ops background knows how good Kubernetes can be, I don't understand its value in this context. So far, I hate PVC for storage and it's another red x in the box for me with TrueCharts - but each to their own, they do seem like a really great team now, which I couldn't say a few years back when I tried an early TrueNAS release and they've done some really solid work - probably with a less than solid architecture for apps to work with so that's impressive.

For now, I have settled on custom docker. This is because it doesn't do stupid things like corrupt itself and prevent container starts, make me rely on third party apps to back up my data ie some TrueCharts apps, or break / freeze during updates.

For me, doing everything manually in a custom docker (which is annoyingly still in kubernetes) is still faster than filling in a million TrueCharts fields I haven't learned and don't want to learn because I don't see any value in them yet. Perhaps that will change but I doubt it.

Oddly, I can't get permissions to work in a custom docker of plex and believe me I know how to set permissions, there is something wrong there I haven't yet put my finger on. The official TrueNAS plex container does not work with the P2000 GPU, despite other containers working with the same GPU and apparently other GPU's working. I logged a ticket for that and after some helpful digging, the response ultimately was it must be in the app and that was that. Except I know it isn't because that GPU worked on unraid.

Another oddity is reverse proxy. The only thing I've gotten to work properly is the TrueApps NGINX proxy manager. But I can never update it because when I do, everything stops working and I have to delete everything including the host path and reinstall everything from scratch. I'm thinking I might set up a light VM with Alpine OS in it and use that.

And finally one really annoying feature of TrueNAS apps, is the lack of a settable hostname. I mean some long string of annoying pod name is really just not good enough for many apps. I use Roon which requires me to relicense it and reset the config every single time I stop the container. That is hellish annoying. THere's another app I have that's the same too. Then there's just the aesthetics of it as well. Clients just have that horrible meaningless name to contend with showing up when you least want it.

I may still spin up an unraid VM, or perhaps portainer, or perhaps some other thing. If anyone knows any of those, I'd love to hear about it. Not looking forward to having to share via SMB or NFS again though - I thought those days were over!

Thanks for all your helpful replies, great discussion!

Marshalleq.

 

[artlessknave]

Oddly, I can't get permissions to work in a custom docker of plex and believe me I know how to set permissions,

does this include knowing that your username AND UID inside and outside the app might need to match?
they definitely do in jails on core; i don't really know scale well.

 

[NugentS]

@marshalleq take a look at what I believe IX are calling sandboxes. This is Jip-Hop's jailmaker. https://github.com/Jip-Hop/jailmaker

 

[marshalleq]

does this include knowing that your username AND UID inside and outside the app might need to match?
they definitely do in jails on core; i don't really know scale well.

Yes I am aware of that, I did go to the extent of dropping into the container, finding out what user plex wanted it to be (it was actually forcing a specific user) and adding that same UID etc to TrueNAS to no avail. I'm actually not sure how this is meant to work with custom apps (i.e if it's meant to match or not), but I tried both. :(

 

[marshalleq]

@marshalleq take a look at what I believe IX are calling sandboxes. This is Jip-Hop's jailmaker. https://github.com/Jip-Hop/jailmaker

Yes, I have a feeling this is the way I'm going to go - have read up a bit on it. Am thinking I might give RC1 a go - have you done that? Because it's pretty well baked in there right?

 

[Stux]

I've just finished converting all my docker compose apps that were running in a VM to running with jailmaker (in RC1). Quite happy with the results, and yes, it comes down to not really trusting anything else not to break.

My needs are simple enough that I can describe them concisely in a few compose files... mount a few host paths into the sandbox, and then mount those and away it goes.

Its working well.

I've looked into the apps, and like @marshalleq I see all the config parameters, and I just wonder why I'd bother changing... other than to get a pretty icon activated...

Regarding inbound reverse proxies etc, I use a pfSense VM running on TrueNAS scale. I've hardware passed through two ethernet ports wan/lan, and then can use HAProxy + ACME to do all the L5 routing that I recquire, SSL wrapping, certificates acquisition etc all in one place, with fairly strong separatation from TrueNAS and the rest of my lan... without actually adding any extra hardware...

And once you understand HAProxy and pfSenses' version of it... it seems to work quite well. I use 2 cores for the pfSense VM, and with Host Passthrough CPU mode, the AES-NI works and pfSense is capable of many gigabits per second of SSL.

pfSense also provides the OpenVPN service, firewall etc. Its a nice separation of concerns... would probably be better as an actually separate physical node... but well... this works :)

And dockge works nicely to control the compose files in the sandbox.