TrueNAS SCALE Project Start

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
Extracted from Announcement

SCALE is an exciting new addition to the TrueNAS software family. It uses much of the same TrueNAS 12.0 source code, but adds a few different twists.

For those eager to know more about the the goals of the SCALE project, they are defined by this acronym:

Scale-out
Converged
Active-active
Linux containers
Easy-to-manage

Since many in our community have already noticed and been speculating for a while, I’ll confirm now that SCALE is based upon Linux. Debian 11 (Bullseye) is currently being used as the platform for this new project. Linux is a key requirement to achieve some of the SCALE project goals. The ability to run OpenZFS 2.0 across both FreeBSD and Linux provides the TrueNAS family with the software diversity to service a wide variety of user infrastructure needs.

We’re pleased to let developers know that the source code for TrueNAS SCALE is already available on GitHub and under very active development. Over the next quarter, we will give you more detailed information about the architecture, download links to the installation images, and more details on how to get involved for collaboration. We’re currently hard at work finishing up some of the base functionality on a very early developer preview image. This nightly image will allow community developers and early technical preview audiences to kick the tires and get more involved in the early development stages of this new product. SCALE will be a development project for the remainder of 2020 with a planned release in 2021.
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
Contratz on the SCALE progress! :)
It might become an interesting competitor to Proxmox...

Some design questions:
- I read somewhere (might even be the repo itself or Jira) you guys are going for a "gluster" based setup, is this to keep ZFS as an underlaying storage backend?
- How is the expected iops performance in contrast to the (non-scale-out) TrueNAS solution?
- What type of Linux containers are you focussing on primarily? I know docker has been "in the pipeline", but will Docker be the primary focus when it comes to containerisation? If so, will there be a solid way of rebalancing containers when hosts "come and go" (aka fail and get back up) and will you guys implement solid fencing?
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Are we looking at a setup that has a gluster volume backed by ZFS and supports running stuff in Linux containers and allows some sort of centralized management? Because that sounds exactly like something I'd be interested in. Even a BSD version would be very interesting, but there is the odd Linux application I need to worry about...
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
Were it not for the linux containers part, I would point at Bastille.
Bastile is interesting, thanks for the link :)


there is the odd Linux application I need to worry about...
This really sums up the only reason for me to use Linux (And with this I also consider any sort of gpu transcoding an "odd linux application")...

Add to this some HA and centralised management from a trusted storage solution developer like IX systems and i'm in.
 

Lastb0isct

Explorer
Joined
Jun 15, 2011
Messages
96
Curious if these features will be extendable to other Linux Distro's? Rather than debian i prefer RHEL/CentOS...just interested to know that.
 

shadofall

Contributor
Joined
Jun 2, 2020
Messages
100
Curious if these features will be extendable to other Linux Distro's? Rather than debian i prefer RHEL/CentOS...just interested to know that.

most likely not to begin with, i suspect the choice to use Debian in part is due to 11 utilizing kernel 5.5 i believe. not to mention the ability to possibly make use of the Ubuntu/Proxmox kernels for ZFS support (or even just reference) would likely streamline the initial development process.

i do recall somewhere tho that they are trying to make the webui a bit more portable. so its possible in the future you might could effectively build out truenas on other distros
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
Curious if these features will be extendable to other Linux Distro's? Rather than debian i prefer RHEL/CentOS...just interested to know that.
The features themselves in theory are, the UI too.

The middleware is another story, while it is python, it's not completely platform agnostic.
While I don't think it would be an issue to port to debian-based distro's, RHEL is a totally different beast in soms aspects.

Considering they focus on the TrueNAS SCALE distro, I highly doubt you will see any progress on porting it to any other distro before (at least) 2025.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
Are we looking at a setup that has a gluster volume backed by ZFS and supports running stuff in Linux containers and allows some sort of centralized management? Because that sounds exactly like something I'd be interested in. Even a BSD version would be very interesting, but there is the odd Linux application I need to worry about...

More info to come, but you can confirm your guesses in the GitHub repo....
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
More info to come, but you can confirm your guesses in the GitHub repo....
Something more technical and I think something that won't be in a pressrelease any time soon:
Considering the glusterfs considerations and average size of TrueNAS/FreeNAS systems currently, is it considered to also allow GUI based setup of arbitrator nodes, or would this probaly be a feature at a later stage than launch?
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
> Scale-out
> Converged

Hyper-Converged, even? A TrueNAS Scale cluster with compute and storage nodes, and orchestration, coming together as a hyper-converged virtualization platform: Is that part of the vision?
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
Hyper-Converged, even? A TrueNAS Scale cluster with compute and storage nodes, and orchestration, coming together as a hyper-converged virtualization platform: Is that part of the vision?
Hyperconverged is actually not having seperate compute and storage nodes, but having all nodes do a little bit of storage AND compute... Well theoretically at least. (there are some providers having different opinions and truely hyperconverged has the tendency to be not very efficient at larger scales.)

But yes, when we talk "hyper converged" we talk (at the very least) about having: Storage, Virtualisation, virtual networking and compute in one node.
 
Last edited:

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
Agreed on compute and storage in one node, and, as you scale, you may find you’re overweighted one way or the other, hence adding “purely compute” or “mostly just storage” nodes.
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
Agreed on compute and storage in one node, and, as you scale, you may find you’re overweighted one way or the other, hence adding “purely compute” or “mostly just storage” nodes.
True, so thats a "yes" to your own answer ;)
Also by digging into the work they've been doing with adding gluster and linux containers into the mix,you can be pretty sure thats what they meant with hyper converged and scaling out...

I know because i've been following this project (I giggled a bit reading they just started it) for months ;)
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
To add something constructive into the mix:
For those of us with some experience with development, the build instructions quite doable and the buildscripts are actually quite nice:

My compliments!
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
> Scale-out
> Converged

Hyper-Converged, even? A TrueNAS Scale cluster with compute and storage nodes, and orchestration, coming together as a hyper-converged virtualization platform: Is that part of the vision?

"Converged" implies Compute and Storage.. can be separate or together. Its the general case.
"HyperConverged" implies they are together... and doesn't help the SCALE acronym :) Do you want SCALE or SHALE?
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
"Converged" implies Compute and Storage.. can be separate or together. Its the general case.
"HyperConverged" implies they are together... and doesn't help the SCALE acronym :) Do you want SCALE or SHALE?
SHALE sound a bit like an STD to be frankfully honest.
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
Okey for those interested in "what is TrueNAS SCALE currently like":

Install wise:
Debian without GUI Installer.

GUI wise:
FreeNAS 11.3, nothing changed (which seems fair considering it's based on the same github repo)
So don't expect fancy Gluster or Docker settings in GUI yet.
Basically: Nothing to see here except a new logo

Update wise:
Update seems to be hacked together a bit, it's a bit TBD..
But it should work and (for those interested) the update url is (currently) this:

Download URL for ISO's (while I expect them to already be constructed after the daily update file is made by the CI), is currently still unavailable.
Although building the ISO really is doable using the readme in the github repo. I think everyone should be able to do it using virtualbox and a latest debian iso. Really, I verified it.

Under the hood:
Some work is already done writhing the middleware portions of both Docker and Gluster support.
I need to do more testing to verify how much of said middleware and/or API is already functional.
 

abcslayer

Dabbler
Joined
Dec 9, 2014
Messages
42
Linux + OpenZFS + Docker (containerd) + iXSystems huge know-how about operating ZFS system = absolutely right!!!
But GlusterFS instead of Ceph would be quite interesting, so there is another early-stage QuantaStor? (QuantaStor support Ceph for quite sometime now).
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
But GlusterFS instead of Ceph would be quite interesting
The reason for going gluster seems obvious: IX wants to focus on ZFS as underlaying filesystem, something which isn't possible to do (production ready) with CEPH.

But it is interesting, I'm very interested myself in a performance comparison between ceph and Gluster on ZFS, with the new Special Allocation Classes.
 
Top