Custom NAS Build and Advice/Suggestions

[Maximilious]

Hey All,

Well I'm finally fed up enough with QNAP to get my hands dirty and build my very own first ever server. Two different units died on me after 3 years and 2.5 years respectively and if I'm going to throw more $$$ at a solution it might as well be one I have control over! If I had know years ago when starting my homelab what I know now I would have gone the FreeNAS/TrueNAS route, but we're here now.

I'm looking to build a custom rackmount but I'm limited in what I can do because my rack is only 22" deep. I did find what I think is a viable solution in components, but before committing I figured it would be best to post here first.

My current QNAP is running five 10TB Ironwolf drives in a RAID5 and comprised of about 30TB of data in the 35TB pool (When moving to Truenas Scale I'd purchase a sixth and move it to RAIDZ-2 volume). I'm also running my Plex server from the QNAP with an Nvidia GTX 1050Ti.

I'll be moving the data over to six 8TB drives I have for the move, and I'd like to move the Nvidia video card to the new build and utilize Plex in a container (or are they jails in TrueNAS?). This unit would connect up to my network via 10GB fiber - I already have the SFP's and cable for that on hand to go with the motherboard.

Here's the build I've come up with:

Case: https://www.amazon.com/gp/product/B0BTY185DY
Motherboard: https://www.amazon.com/dp/B07534B8FS
CPU: https://www.amazon.com/dp/B07V3SWL51
RAM: https://www.amazon.com/dp/B094DTKB88
PSU: https://www.amazon.com/dp/B08W47MX46
Heatsink: https://www.amazon.com/dp/B091HQJ1M1
SAS Cables: https://www.amazon.com/dp/B017CO6JRO
USB to SATA Cable (Optional for boot drive mirror): https://www.amazon.com/dp/B011M8YACM

A few questions I have about the build:

• Will it work?
  • I believe all the components will go together nicely but a second (or thousand) set of eyes can never hurt.
• Memory Amount - Is it too much?
  • I've read about deduplication memory consumption for the table but I'm not certain how much would really be in use. I just reinstalled my repaired QNAP with QuTS Hero for ZFS support and memory consumption doesn't seem that large even though I have deduplication turned on for about 10TB of data (2GB of 32 used currently). Is it only consumed for the amount of data that's deduped, or the size of the volume as a whole?
• SATA DOM vs M.2 vs USB SATA for the Scale OS
  • I searched a bit and it seems there's not much concern for the OS performance/speed. Is this also the case if using deduplication? Would a USB SATA mirror perform just as well as an M.2 for the OS?
  • I'd hate to let the M.2 slot go to waste, would there be any issue setting it up with a mirror on another drive for the OS, or possibly read cache?
• Drive Identification
  • This is my first time building a server from scratch using SAS cables. I had to do some reading to even figure out that SAS and SATA are very similar! What's the best way to identify which drive bay corresponds to the correct port identifier on the motherboard? This is a HUGE concern for me in the event of drive failure.
• Drive Capacity Limit
  • How can I know what the capacity limit is of this build? Perhaps this is my traditional way of thinking, but certain NAS devices have an upper limit of the amount of TB/PB they can host - what dictates this limit?
• Video Card and Plex
  • I just saw a recent post on the forum regarding the need for two video cards to pass through to a VM on TrueNAS. I assume using the video card for hardware transcoding in a container/jail would work without issue but I want to ask the question to ensure no issue.

I THINK that's all the questions I have. Thanks to anyone that's made it this far!

P.S. - First Post! Hello TrueNAS Community!

 

[jgreco]

I've read about deduplication memory consumption for the table but I'm not certain how much would really be in use.

You don't want deduplication. Unless you have an application where you would have dozens of copies of exactly identical data, the amount of effort you have to put into getting dedup to be stable is horrifying. Think on the order of 5-10GBytes of ARC per deduplicated TB. Stilez has an excellent resource on dedup in the Resources section.

Be aware that Linux memory management is crap and therefore SCALE is crappy too. If we say 64GB of ARC, expect to need 128GB of RAM.

This is my first time building a server from scratch using SAS cables. I had to do some reading to even figure out that SAS and SATA are very similar! What's the best way to identify which drive bay corresponds to the correct port identifier on the motherboard? This is a HUGE concern for me in the event of drive failure.

Get a Brother labelmaker and put the serial number of your drives on the front of each tray. Also feel free to read the SAS Primer.

How can I know what the capacity limit is of this build? Perhaps this is my traditional way of thinking, but certain NAS devices have an upper limit of the amount of TB/PB they can host - what dictates this limit?

Memory and drive bays. With 12 bays and 22TB drives, your practical limit is 264TB raw or maybe around 140TB usable if you do RAIDZ3 with a spare and 80% maximum utilization. You would do well to have between 64GB-128GB of ARC, so if using SCALE, shoot for 128-256GB RAM for that size pool.

You can practically scale ZFS far north of that as long as you have I/O capacity. I believe iXsystems is sticking 20 petabytes into a 45U rack. You'd need a larger server than you have just to handle the memory and I/O devices, but if you only needed a slow system or a sedately lazy system, you can go quite far on reasonably inexpensive hardware.

North of 20 petabytes is specialist territory at this time.

I just saw a recent post on the forum regarding the need for two video cards to pass through to a VM on TrueNAS. I assume using the video card for hardware transcoding in a container/jail would work without issue but I want to ask the question to ensure no issue.

Between your 12 drives, CPU, and talk of a GPU, your PSU is probably undersized. Please see

Proper Power Supply Sizing Guidance

I've seen about 1,000 threads like this one where people decide that they can power a dozen hard drives off a 360 watt supply. DO NOT DO THIS. I've seen another 1,000 threads where people decide to buy the cheapest power supply that they can...

www.truenas.com

 

[Maximilious]

You don't want deduplication. Unless you have an application where you would have dozens of copies of exactly identical data, the amount of effort you have to put into getting dedup to be stable is horrifying. Think on the order of 5-10GBytes of ARC per deduplicated TB. Stilez has an excellent resource on dedup in the Resources section.

Be aware that Linux memory management is crap and therefore SCALE is crappy too. If we say 64GB of ARC, expect to need 128GB of RAM.

Thanks - I'm used to Nimble/HPE deduplication from work and it appears TrueNAS implementation of it is pretty barebones and dedupes at the file level rather than block-level. I'm only saving about 250GB on my 30TB now with compression and deduplication on, so you offer sound advice.

Get a Brother labelmaker and put the serial number of your drives on the front of each tray. Also feel free to read the SAS Primer.

Your write-up here is awesome, thanks for the knowledge dump. I never know SATA could use SAS ports. I always suspected, but never wanted to
try due to SAS not fitting into SATA.

Memory and drive bays. With 12 bays and 22TB drives, your practical limit is 264TB raw or maybe around 140TB usable if you do RAIDZ3 with a spare and 80% maximum utilization. You would do well to have between 64GB-128GB of ARC, so if using SCALE, shoot for 128-256GB RAM for that size pool.

Okay, so my target of 128GB should track well. My expected target is around 65TB usable between 6x10TB and 6x8TB drives in two RAIDZ2 pools and using a Plex Container/Jail. I have Dell T430 running ESXi for any other VM's, I'm just trying to replicate what my current QNAP is providing me now.

Between your 12 drives, CPU, and talk of a GPU, your PSU is probably undersized. Please see

Proper Power Supply Sizing Guidance

I've seen about 1,000 threads like this one where people decide that they can power a dozen hard drives off a 360 watt supply. DO NOT DO THIS. I've seen another 1,000 threads where people decide to buy the cheapest power supply that they can...

www.truenas.com

Again, another awesome writeup! I tried to run my components through a PSU calculator, but calculations for server components are pretty slim pickings. The numbers showed about 450watt which is why I chose a 750. Adding up the max wattage including your 35w per drive I'm at about 850w so I'll bump up to 1200.

As for the M.2 drive - Does TrueNAS have a read or write cache option, and if so would having a single drive cache potentially cause data loss in the event of a component or power failure? This will be plugged into an APC C1500 UPS but I have had some longer power outages here.

I'm also thinking of taking some advice from other threads I've seen and installing the OS on a pair of SATA SSD drives mirrored and connected via USB interface. Would this hinder performance at all in relation to compression/read/write operations as opposed to installing to the M.2 and a mirrored SATA SSD for redundancy?

Lastly - Will I also need an HbA for this build? That component actually slipped my mind. I assume yes but wanted to ask.

 

[jgreco]

Thanks - I'm used to Nimble/HPE deduplication from work and it appears TrueNAS implementation of it is pretty barebones and dedupes at the file level rather than block-level. I'm only saving about 250GB on my 30TB now with compression and deduplication on, so you offer sound advice.

Incorrect, it's a hot and heavy full block deduplication system based on checksums. The problem is that you incur significant additional CPU load (because every written block has to be checksummed for the dedup) and also ZFS uses a variable block size strategy that can make your blocks up to 1MB large, and so usually some tuning is required to optimize your workload for deduplication. ZFS dedup is much more practical today than it was ten years ago, because we have stuff like lots of cheap RAM and also stuff like NVDIMM/Optane/etc. which can be hammered into a practical deduplication strategy. Dedup creates new system stresses that most of the users here aren't very familiar with. Therefore, it's not for everyone and definitely not recommended unless you have a realistic use case for it.

Lastly - Will I also need an HbA for this build? That component actually slipped my mind. I assume yes but wanted to ask.

I'm looking back and I see *me* saying 12 drives. Did I just jump to conclusions based on your 12 bay chassis? I have no familiarity with that board but it does support 12x SATA3 (6Gbps) from the 622 PCH. The LSI HBA's will actually be a bit slower than native Intel SATA AHCI ports. But the LSI HBA's can handle at least a few hundred drives (through SAS expanders) so it's a pick-yer-poison kind of situation.

 

[Etorix]

I'm only saving about 250GB on my 30TB now with compression and deduplication on, so you offer sound advice.

With such a poor figure, don't bother with ZFS dedup.

Your write-up here is awesome, thanks for the knowledge dump. I never know SATA could use SAS ports. I always suspected, but never wanted to
try due to SAS not fitting into SATA.

Note that, unless you're throwing an undisclosed SAS HBA, your setup is pure SATA, from the C622 SATA controller to SATA drives. The picture of a reverse breakout cable in the Amazon page for the case clearly indicates that it only has a passive backplane. No SAS here—except in the name (and the NAME only) of some connectors—and no support for SAS drives.

As for the M.2 drive - Does TrueNAS have a read or write cache option, and if so would having a single drive cache potentially cause data loss in the event of a component or power failure? This will be plugged into an APC C1500 UPS but I have had some longer power outages here.

Read cache = L2ARC. No risk of data loss. But use and benchmark the system first before considering a L2ARC; with 128 GB RAM to begin with, there's a fair possibility that a L2ARC may bring no benefit.
There's no "write cache" for ZFS. SLOG is NOT a cache, and is only useful for synchronous writes, which you have not mentioned yet.

I'm also thinking of taking some advice from other threads I've seen and installing the OS on a pair of SATA SSD drives mirrored and connected via USB interface. Would this hinder performance at all in relation to compression/read/write operations as opposed to installing to the M.2 and a mirrored SATA SSD for redundancy?

No. A boot drive is for… booting. It has no impact on performance in operation.

Lastly - Will I also need an HbA for this build? That component actually slipped my mind. I assume yes but wanted to ask.

Only if you use SAS drives, or need to expand beyond 12 drives.

 

[Maximilious]

Incorrect, it's a hot and heavy full block deduplication system based on checksums. The problem is that you incur significant additional CPU load (because every written block has to be checksummed for the dedup) and also ZFS uses a variable block size strategy that can make your blocks up to 1MB large, and so usually some tuning is required to optimize your workload for deduplication. ZFS dedup is much more practical today than it was ten years ago, because we have stuff like lots of cheap RAM and also stuff like NVDIMM/Optane/etc. which can be hammered into a practical deduplication strategy. Dedup creates new system stresses that most of the users here aren't very familiar with. Therefore, it's not for everyone and definitely not recommended unless you have a realistic use case for it.

Okay, that's good to know. I should have read up more on it's technology as I also assumed it's inner workings. Seems it is closer to HPE/Nimble than I thought where you would want your read/write cache (possibly happens at the OS level?) on high-performing SSD/NVMe for that process, along with the memory for the Dedup table and CPU to crunch the numbers. All speaking laymans terms here, I'm more a man of many hats rather than focused on a specific technology!

I'm looking back and I see *me* saying 12 drives. Did I just jump to conclusions based on your 12 bay chassis? I have no familiarity with that board but it does support 12x SATA3 (6Gbps) from the 622 PCH. The LSI HBA's will actually be a bit slower than native Intel SATA AHCI ports. But the LSI HBA's can handle at least a few hundred drives (through SAS expanders) so it's a pick-yer-poison kind of situation.

You assumed correctly. I'm going from a 6 bay to a 12 bay for future growth. Silly me in my research forgot that HBA's are for expansion. I was having a hard time finding an mATX board to fit in that case that would handle 12 drives.

I imagine I will be splitting out my current data from hosting everything in one Pool to Pool1 hosting Plex media and Pool2 hosting everything else (family photos/docs, games, container volumes, ect.). I could just clean up what isn't watched, but who has time for that??

Thanks for all the help!!

 

[Maximilious]

Note that, unless you're throwing an undisclosed SAS HBA, your setup is pure SATA, from the C622 SATA controller to SATA drives. The picture of a reverse breakout cable in the Amazon page for the case clearly indicates that it only has a passive backplane. No SAS here—except in the name (and the NAME only) of some connectors—and no support for SAS drives.

That's correct, I'll be using SATA with no HBA at all and using the backplane provided by the case. The case does specific SATA/SAS availability, but you're correct that it may be passive. Either way, I have no inclination of using SAS drives for some time (if ever). I haven't seen a need to go beyond the 6GB capability of SATA on the backplane and doubt I ever will for personal/family use.

Read cache = L2ARC. No risk of data loss. But use and benchmark the system first before considering a L2ARC; with 128 GB RAM to begin with, there's a fair possibility that a L2ARC may bring no benefit.
There's no "write cache" for ZFS. SLOG is NOT a cache, and is only useful for synchronous writes, which you have not mentioned yet.

Thanks for the explanation here and you're probably right - with no deduplication and ample memory I would probably be okay without an L2ARC. I quickly read up on SLOG and I can only think of a single use case for one in my environment. The only share that may benefit would be my Container share where I'm hosting roughly 20-30 container root file systems out of, but I can see how everything performs without an SLOG first - I assume just fine without though.

Thanks again!

 

[Maximilious]

Chiming it to say the system is built and TrueNAS is up and running! Currently rsync'ing data over from my QNAP and once done I'll steal the disks from it and install a second pool on the TrueNAS deployment. Happy clam here!