TrueNAS SCALE Project Start

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
TrueNAS SCALE will be optimized for baremetal 1st.... clouds will be later. But, any developer can use the code and try it out on cloud instances.
 

NickF

Guru
Joined
Jun 12, 2014
Messages
760
Not to keep this thread too far off topic, but AMD attempted to break into the ARM market 3 or 4 years ago seems to have abandoned the project. They initially slapped an Opteron logo on a standard Cortex A-57 design:
They began working on custom silicon under the helm of Jim Keller back in 2013. It was dubbed K12, which for you oldies out there was a nod at it becoming the successor for their K10 (Phenom) and unofficial K11 (Bulldozer) x86 designs. It was supposed to release in parallel to Zen 1, but it never did.

There is no official evidence that Intel is working on an ARM processor design. Intel sold off the XScale line in 2006 to Marvell and there is no indication that they have looked back.

Both of these giants took the time to research ARM and put substantial efforts into its development, and then chose to stick with X86 anyway. If they didn't jump on it, they must feel that the advantages for servers are simply not there.

Just because Apple wants to vertically integrate their product stack doesn't mean the rest of the market is going to follow suit. In mobile ARM makes sense, but in high-performance compute X86 is still king and will be for the foreseeable future. Power and efficiency improvements over the years exist in both X86 and ARM. Performance has also increased in both fronts. Please see https://research.cs.wisc.edu/vertical/papers/2013/hpca13-isa-power-struggles.pdf

In any case, whatever future R&D holds on the processor front, IX has alot on their plate right now with the transition to TrueNAS Core and the introduction of SCALE. I would hope that they are utilizing their resources more wisely than trying to compile an ARM compatible version of SCALE.
 
Last edited:

JoeAtWork

Contributor
Joined
Aug 20, 2018
Messages
165
Linus said in an interview the same thing, that when Apple makes a decent Arm desktop/laptop that he could use one. Until then all the arm systems are different and junky compared to what we expect in a high end laptop or workstation.
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
Okey, some analysis here, We need to seperate a few things:
Market
Power consumption
Performance

Market
The primary target audience for apple products is consumers with a low performance requirement
The secondary target audience for apple products is specific professional use, with a significant above-average budget

The primary target audience for Truenas is Enterprisess with a high-performance medium-size (a few servers/systems) requirement for storage
The Secondary target audience for Truenas is Enterprises without high performance requirement but with still medium size storage requirement (aka 1 or 2 storage servers)
The Tertiary target audience for Truenas is consumers with an above average requirement for network storage and requiring a premade solution
The quartiary target audience for TrueNAS is consumers with an above average requirement for network storage that want to build it themselves

The primary audience for ARM is mobile
The secondary target audience for ARM is low-power desktop and servers

The primary audience for X86_64 is Every system that requires a decent amount of performance (desktop or server) and doesn't have heavy power constraints

Power consumption and Performance
The Power consumption of ARM is lower, yes.... But the max performance is less. Simply put: ARM IPC-Per-Core (IPCPC) doesn't scale very well.

When it comes to Laptop, Low-use Desktop and NAS use, less power consumption is an improvement and most people don't really need much CPU anyway.
But when it comes to servers: Those are often selected based on peak-performance-requirement. They NEED it to be able to handle the peak load. Peak-load means they have a high IPCPC requirement


Conclusion

Okey, so what does this all mean?
Most people don't use their PC heavily, that much is clear. Thats why Apple slowed their update schedule for their "real" pro lineup a decade ago. The VAST majority of their customers don't need a "real" MAC-Pro.
In the same way the vast majority of their consumers don't need the full featureset and power of a x86_64 processor.
So I totally understand why Apple has decided to go for ARM, it fits their sales pattern rather nicely and there aren't many technical reasons not-to move to ARM for Apple.

When it comes to CPU requirement the situation is different for storage. ZFS storage can be seperated into two catagories:
1. Home/SMB NAS-storage
These systems already often use ARM hardware. While it won't provide the best performance with ZFS, said performance is well acceptable for small SMB and Home use.

2. Enterprise storage
These systems often need to push large amounts of data rather quickly. The CPU's get hammered quite often: Compression, Checksumming and Encryption at 1GB/s+ aren't easy, nor are they cheap performance wise. Just because a home users gets by with an ARM system, doesn't mean these servers can. Actually, A lot of those systems are CPU bottlenecked already.

As I wrote before, TrueNAS focusses at Enterprise storage primarily... It just isn't viable development wise to move to ARM just for their tertiary or quartiary audiences.

The Future of TrueNAS SCALE

Most of the packages used by TrueNAS SCALE, already have ARM equivalents or are platform agnostic. If IX-system would want to move to ARM (which isn't likely any time soon), it would be relatively easy.

ZFS and ARM
I will keep this short:
Yes OpenZFS supports ARM.

But where ARM servers shine is throwing a lot of weak threads at a task (for example see 256 core ARM servers, those ain't x86_64 performance grade cores!), however: ZFS threading highly depend on load. The more datastream you have, the more threads it can handle. Simply put: 100 people writing files is a lot more threadable than 1 user with 100 times the write-speed requirement.

This is a gross over-simplification, but ZFS compression scaling (the main use of CPU) is rather complex. While working on ZSTD-on-ZFS (See github for more info), We had a lot of discussion about doing realistic testing... Not every writestream is the same.

That being said: ARM is nice in a situation with a lot of users accessing a lot of data. It isn't great for big single-stream transfers.
 
Last edited:

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
The primary audience for ARM is mobile
Today, yes... near future, maybe every apple desktop/laptop including high performance and flow-on maybe servers too ... need to re-think with future in mind. What Data Center manager wouldn't want lower power consumption/cooling requirement and options for higher density with existing facilities?
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
Today, yes... near future, maybe every apple desktop/laptop facilities?
Considering you responded faster than I could edit the post, I highly doubt you read my post carefully.
If you did, you would've noticed, I actually said I think ARM is a good fit for most desktop and laptop usecases.

including high performance and flow-on maybe servers too ...
Single thread performance (high IPC+High clockspeed) on ARM is not-that-great, this makes it not-very-suitable for most high-performance use-cases

What Data Center manager wouldn't want lower power consumption/cooling requirement and options for higher density with existing facilities?
The Data Center Manager that has customers that have high IPC+Clockspeed requirements. As I explained servers are selected on peak load, not average load. With ARM you hit the peak-performance-per-core earlier than X86_64. BY DESIGN.

Why ARM isn't magically going to be X86_64-grade
You seem to think ARM could magicallybe MUCH more performant. Newsflash: IT WONT.
Why? Because it's a leaner architecture. Simply put: Less transistors to do the same work. Often due to having more generalised instructions.
X86_64 is a giant pile of specialist instructionssets, which means it also has a huge load of transistors to support all those more-and-more specialist instructions.

What does that mean in practice?
In Practice this means ARM is more energy efficient (less or smaller transistors means less power required to keep it all up-and-running, simply put), It also means that X86 would have a WAY higher IPC if it is processing a computation for which a specialist instructionset is used. The more often specialist instructionsets can be used, the higher the performance and the more specialist instructionsets there are, the more often the higher performance could be used.

Could we just add instructionssets to ARM for high-performance applications? Yes. But that would be very silly, because you would be creating basically the same thing that x86_64 does already and would be adding energy consumption. If we start this route, in a few years we would be having the same thing we have in x86_64 already.

So when CAN we use ARM?
So, The reason ARM works for apple: They don't need much performance
The reason this works for some datacenters: They often have a load that is easily spread over a lot of lighter cores
The reason this often won't work with ZFS? It doesn't often scale nicely over insane amounts of cores... it's very load dependent.
The reason this often don't work with high performance applications? They often still need fast single core performance and often rely heavily on specialist instructionssets to get optimal performance out of those cores.

My vision for the future?
I personally expect more-and-more consumer oriented devices moving to ARM. Consumers simply don't push their systems hard enough to warrant the requirement for x86_64

But I don't see ZFS enterprise storagechanging this way any time soon. Unless HUGE changes to ZFS are done (including ALL compression algorithms), which I don't see happening considering how much work goes in just adding one faster algorithm.
 

NickF

Guru
Joined
Jun 12, 2014
Messages
760
I agree with Ornias in most of his reply. As someone who literally is a Data Center Operations Manager, I would not consider an ARM processor at this time. That's assuming that products actually exist and are readily available. But that is not even the case.
With AMD EPYC putting 128 cores and 256 threads, HIGH PERFORMANCE CORES, in a 2u box, all I can say is most of the benefits I would gain with ARM I could gain by retiring old BladeCenters or Cisco UCS M series chassis and replacing them with EPYC...or even the new 3rd Gen Intel Scalable in some use cases.
To be very clear, my datacenters are designed with specific power capacity and cooling in mind already. I definitely have horror stories at some of my smaller sites, but largely, I don't think I would gain much from the ARM use case. But I would lose legacy compatbility...Making the barrier to entry far too high for negligible gain.

Can we please drop this topic??
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Can we please drop this topic??
Consider it dropped (by me at least), following this response.

Considering you responded faster than I could edit the post, I highly doubt you read my post carefully.
I freely admit to regularly jumping to conclusions and not reading posts in full before responding, but in this case even after re-reading your (edited) post carefully, I still have another opinion.
I regret conveying (unintentionally) any sign of disrespect as you clearly spent time formulating your opinion and articulating it in the post.

I would not consider an ARM processor at this time.
I am in no way suggesting immediate or even complete replacement of Intel architecture with ARM in Data Centers.

What I intended to add (as my opinion) is that since apple will replace the CPUs in their entire line of Macs (which means including Mac Pro) in the coming 2 years, the chips available on the ARM side may significantly change in the direction of high performance (even if that is directed at specific workloads... perhaps some of those being encryption, compression or other tasks which have a potential overlap with storage processing needs).

For 8K (and beyond) video editing, it will be necessary to have excellent IO on those boards/chips, so I would expect to see some of that spilling over into some server applications.

So in short, I think the ARM architecture could prove to be interesting in the coming few years, but we'll need to watch it evolve to be sure. There may be some benefit to spending a small amount of time considering the effort that would be required to have a product like TrueNAS SCALE running on ARM (once it's delivered on Intel) and maybe taking steps to avoid ruling the possibility out by any decisions to be made in the short term.

Anyway, that's what I wanted to contribute, so consider me done.
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
I freely admit to regularly jumping to conclusions and not reading posts in full before responding, but in this case even after re-reading your (edited) post carefully, I still have another opinion.
I regret conveying (unintentionally) any sign of disrespect as you clearly spent time formulating your opinion and articulating it in the post.
My respect for actually admitting to it. You offense taken at all. I was more amazed, because you mostly don't make that mistake ;)
I understand we have a difference in opinion, my amazement was more the fact that some nuance got lost in translation.
 

diskdiddler

Wizard
Joined
Jul 9, 2014
Messages
2,374
I was surprised to see Kris mention some kind of downloadable beta in the next few months for Scale, this really surprised me - is that actually true, anyone know?
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,737

diskdiddler

Wizard
Joined
Jul 9, 2014
Messages
2,374
I just found that, thanks. I will have a play with this on my spare machines.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
I was surprised to see Kris mention some kind of downloadable beta in the next few months for Scale, this really surprised me - is that actually true, anyone know?

Its not a BETA, but a NIGHTLY. Internally, we call its a "Technology Preview". Its a foundation for development and testing as new features get added and bugs get fixed. Because its based on TrueNAS 12.0 and OpenZFS it is more stable than usual, but we don't recommend anyone store their only copy of data on this. However, we really do appreciate feedback from everyone that does test and report issues.
 

silverback

Contributor
Joined
Jun 26, 2016
Messages
134
I have been unable to connect to an SMB share from either OSX or Windows. The creation process works fine and the service starts, but errors out after authentication. Is it still to early for this service?
 

DavidSpek

Cadet
Joined
Jul 4, 2020
Messages
2
I've been using unRAID as a media server for a while now, but I am debating the switch to TrueNAS (SCALE) due to my increasing needs. My main reasoning for using unRAID was easy storage expansion, but that may come at some point with RAIDZ expansion.

I just spun up Truenas SCALE in a VM, with 8 vdisks for a pool, and was able to successfully launch a docker container with a dataset mapped through for config storage. I could access the container webUI at the forwarded port and everything looks to be running correctly.

I have a question about the SCALE clustering implementation though. At the moment I am using Rancher and RKE to deploy kubernetes nodes and manage the pods on each node. In unRAID, the docker UI is essentially an XML template for a docker run command. Containers deployed by docker compose or another method do show up, but lack the usual configuration features (logically). Will the kubernetes support be through a solution similar to RKE (where kubernetes runs within docker containers) or will kubernetes be deployed directly on the system? Also, is the idea that the docker or clustering UI will (also) be used to manage a kubernetes cluster and delegate node roles? My idea was to possible use FreeNAS as the base OS for a kubernetes cluster and built in management feature similar to Rancher would make that very appealing.
 

Lennie

Cadet
Joined
Jun 9, 2020
Messages
2
Why ARM isn't magically going to be X86_64-grade
You seem to think ARM could magicallybe MUCH more performant. Newsflash: IT WONT.

Now I don't expect we'll see lots of ARM storage servers larger than 5 HDDs in the near future, but ARM is now at the #1 position of the TOP 500 list. Now maybe this isn't because of single core performances, but from it's new design: probably because it sets more in between CPU-based and GPU-based super computer design and very high performance interconnect makes it all come together. It's also more like a SOC for example it has memory on-die.
Some article claimed this: "While the A64FX processor is compliant with the Armv8.2-A spec and has the SVE extensions, it has a custom core that inherits the superscalar processing, out-of-order execution, and branch prediction capabilities of the Sparc64 architecture. You can just block copy this stuff right in and make a better Arm chip, and this is what Fujitsu has done."
 

NickF

Guru
Joined
Jun 12, 2014
Messages
760
1594600676015.png


I know this is about a year old and doesn't factor in recent launches...
But X86 is getting wider than ever, and is still the IPC king.

Just stop :)
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
I just spun up Truenas SCALE in a VM, with 8 vdisks for a pool, and was able to successfully launch a docker container with a dataset mapped through for config storage. I could access the container webUI at the forwarded port and everything looks to be running correctly.

I have a question about the SCALE clustering implementation though. At the moment I am using Rancher and RKE to deploy kubernetes nodes and manage the pods on each node. In unRAID, the docker UI is essentially an XML template for a docker run command. Containers deployed by docker compose or another method do show up, but lack the usual configuration features (logically). Will the kubernetes support be through a solution similar to RKE (where kubernetes runs within docker containers) or will kubernetes be deployed directly on the system? Also, is the idea that the docker or clustering UI will (also) be used to manage a kubernetes cluster and delegate node roles? My idea was to possible use FreeNAS as the base OS for a kubernetes cluster and built in management feature similar to Rancher would make that very appealing.

SCALE provides docker CLI and API. Using these you can deploy Rancher and Kubernetes.

There is also an option of deploying Kubernetes natively on SCALE nodes. We are looking for user feedback, write-ups and contributions.
 

silverback

Contributor
Joined
Jun 26, 2016
Messages
134
I have been unable to create a bridge network for vm's either through the GUI or with virsh. The bridge is made but the when activated I lose the web configurator. Docker network seem to work right out of the box. Not an expert at all so could be operator error.
 
Top