New NAS: Taking The Plunge... (Part Two)

[JTT0]

Hi all,
I am preparing to purchase components for my next generation NAS Server and would love some thoughts on what I have before diving in!

For starters, my existing NAS is aging (built in 2014) and is limited on performance. This is my driving force for an upgrade (the old server will become my backup system). Throughout this time, I have also had local backups to offline HDDs but now would like to have an online solution as well.

New Backup Strategy:

• Performance NAS (FreeNAS RAIDZ Stripe of Mirrored Pairs [ex: RAID 10])
  • This thread is discussing this one
• Retired parts NAS (FreeNAS RAIDZ Stripe (2x RAIDZ2) [ex: RAID 60])
  • This thread discussed it's build
  • Previous setup was 2 pools - 7 in RAIDZ3 and 1 standalone
• Offline/Offsite HDDs (RAID LACKING)

My primary Use-Cases for this system is as follows:

1. FreeNAS Encryption [ZFS/GELI Encryption]
2. Time Machine Backups (Fast)
3. File Storage (Backed up and safe)
4. Media Serving
   1. I use Emby as the server-in-jail with occasional transcoding but primarily focused on direct play
   2. Devices being served [Kodu HTPC, Apple TV, Macintosh PCs, Windows PCs, phones]
   3. Likely a max of 3-4 simultaneous streams at the same time
5. Code Server (Git Server)
6. Nice To Have: Run VMs directly from shares

Derived Requirements:

1. More Threads!
2. RAM Must be ECC
3. 8 x Hot Swappable HDD
4. Boot Drives must not be USB (too many have died)
   1. M.2 NVME or SATA SSD

Derived Nice-To-Haves:

1. SAS on Motherboard (Prevent from burning a PCIe bus)
2. 10GBe on Motherboard (Prevent from burning a PCIe bus)

Primary Areas For Assistance Request:

1. RAM - I have no idea where to go with 6 channels!
2. Heat Concerns with higher TDP CPU and U-NAS Case
3. Motherboard/CPU for occasional transcoding
4. Anything I might be missing!

Existing NAS Parts (For ease of viewing without going to link above):
OS: FreeNAS 11.2
Boot Drives: 2 x (mirrored) SanDisk 32GB Cruzer Fit USB Flash Drive
Case: U-NAS NSC-800
Motherboard: ASRock E3C224D2I
CPU: Intel Core i3-4130T
RAM: Kingston 16GB (2 x 8GB) 240-Pin DDR3 1333MHZ
PSU: SeaSonic SS-350M1U SS-350M1U 350
SATA Port Extender / Raid Controller: IBM M1015 (Flashed to IT Mode)
Storage: WD Red (8 x 3TB)
Other: Various Extensions, etc.

New NAS Parts (Owned):
OS: FreeNAS 11.3
Case: U-NAS NSC-810A
Storage: WD Red (8 x 3TB)

New NAS Parts (Under Consideration):
Boot Drives: 2 x (mirrored) Kingston 120GB A400 SATA 3 2.5" Internal SSD SA400S37/120G
Motherboard: Supermicro X11SPM-TF
CPU: Intel CPU BX806954216 Xeon Bronze 3204 6C 6T 1.9GHz 8.25M FC-LGA14B
CPU Cooler: Supermicro SNK-P0067PS LGA 3647-0 1U X11 Purley Platform CPU Heat Sink
RAM: Samsung M393A4K40BB1-CRC 32GB DDR4-2400 LP ECC Reg Server Memory (2 x 16GB)
PSU: Seasonic SS-400H1U 80+ 1U 400W
Other: 2 x CableDeconn 0.7M Internal Mini SAS 36-Pin to SFF-8087 Cable
Motherboard Manual:

https://www.supermicro.com/manuals/motherboard/C620/MNL-1939.pdf

Motherboard Technical Specs:

X11SPM-TPF | Motherboards | Products | Super Micro Computer, Inc.

X11S, X11SP, X11SPM, X11SPM-TPF

www.supermicro.com

Alternative Parts Under Consideration (Not as a unit):
Case: SilverStone Technology CS381B
Case: SilverStone Technology DS380B
Boot Drives: 2 x (mirrored) Kingston A400 120G Internal SSD M.2 2280 SA400M8/120G
RAM: Supermicro Certified MEM-DR432L-SL01-ER24 Samsung 32GB DDR4-2400 LP ECC REG
Other: Supermicro AOC-SLG3-2M2 PCIe Add-On Card for up to Two NVMe SSDs
CPU: Intel Xeon Silver 4110 2.1Ghz Processor BX806734110
Possible GPU?

Alternative Setup:
Motherboard: X11SCL-F (Reference)
CPU: i3-9100F (Reference)
RAM: 2 x Crucial CT8G4DFS824A 8GB DDR4 2400 [or a 32GB setup] (Reference)
SATA Port Extender / Raid Controller: IBM M1015 (Flashed to IT Mode)
10GBase-T Card: StarTech.com 1 Port PCI Express 10GBase-T/NBASE-T Ethernet Network Card - 5-Speed Network Support: 10G/5G/2.5G/1G/100Mbps - PCIe 2.0 x4 (ST10GSPEXNB)


Thank you so much for your time and assistance!

 

[PhiloEpisteme]

New Backup Strategy:

• Performance NAS (RAID 10)
  • This thread is discussing this one
• Retired parts NAS (RAID 60)
  • This thread discussed it's build
• Offline/Offsite HDDs (RAID LACKING)

Can you clarify if this is referring to some set of other non-FreeNAS/zfs machines? The reason I ask is a bit nitpicky, I admit, but FreeNAS uses zfs and zfs doesn't have RAID10/60/etc.

1. FreeNAS Encryption [ZFS/GELI Encryption]

You may already know this; but encryption adds some extremely important steps in maintenance. Specifically, keep copies of both keys for every pool and unless you are extremely sure of your technique, follow the User Guide to the letter with respect to drive replacement, expanding pools, adding cache|slog devices, etc.

1. Likely a max of 3-4 simultaneous streams at the same time

What your machine needs to accomplish this depends a lot on whether you're transcoding or not and at what resolution. If you're using Plex they have a really useful passmark score guide that gives you a rough idea.

1. SAS on Motherboard (Avoid an HBA as to prevent burning a PCIe bus)
2. 10GBe on Motherboard (Avoid an HBA as to prevent burning a PCIe bus)

What are you saving those PCIe lanes for? Where possible I actually prefer NOT to have things built into the MOBO and prefer more PCIe lanes because they are quite versatile. Want to upgrade your SAS2 card with a SAS3 card? No problem? Want to upgrade your SAS2 MOBO with a SAS3 MOBO, much harder. Definitely a bit of personal preference though.

Boot Drives must not be USB (too many have died)

1. M.2 NVME or SATA SSD

I recommend SATA SSD where possible. Save any M.2 w/NVMe for purposes that really benefit from it, such as SLOG devices.

 

[JTT0]

Hi @PhiloEpisteme,
Thank you for your insight into some important aspects of the build process. First off, due to the "Wife Acceptance Factor", the U-NAS NSC-810A case has been chosen for me. My original post has been updated to reflect this and a few other changes.

Can you clarify if this is referring to some set of other non-FreeNAS/zfs machines? The reason I ask is a bit nitpicky, I admit, but FreeNAS uses zfs and zfs doesn't have RAID10/60/etc.

I have updated my original post to specify my intentions but both systems will be FreeNAS and using RAIDZ setups for redundancy.

You may already know this; but encryption adds some extremely important steps in maintenance. Specifically, keep copies of both keys for every pool and unless you are extremely sure of your technique, follow the User Guide to the letter with respect to drive replacement, expanding pools, adding cache|slog devices, etc.

Thank you for the concern around this area as it is very important to not miss on any of these points or data loss will occur. I have backed up all encryption keys accordingly and have not have a single bit of data loss in the past 5 years :). As for adding a cache/SLOG device, I have not needed one to date but have been researching and debating using it in my next system for quite some time.

What your machine needs to accomplish this depends a lot on whether you're transcoding or not and at what resolution. If you're using Plex they have a really useful passmark score guide that gives you a rough idea.

I am using Emby and generally not transcoding (Mostly Direct-Play) but do need to be able to play media on mobile devices, etc. so some transcoding is likely to take place. The primary format being transcoded would be 1080p but 4k media is also present.

What are you saving those PCIe lanes for? Where possible I actually prefer NOT to have things built into the MOBO and prefer more PCIe lanes because they are quite versatile. Want to upgrade your SAS2 card with a SAS3 card? No problem? Want to upgrade your SAS2 MOBO with a SAS3 MOBO, much harder. Definitely a bit of personal preference though.

As the U-NAS NSC-810A case has been chosen, it only allows for two PCIe cards to be installed. I have relaxed my requirement of having a motherboard with 10GBe and SAS but my need is to have 10GBe and 10x SATA connections (8 x HDD and 2 x OS SSD). I have also been toying with adding a PCI card for another M.2 SDD or two for caching/SLOG usage. The motherboard may or may not have one/two on it. In addition to required I/O, I guess my thought process was that the less real-estate taken up by cards, etc. in the case would reduce heat and allow better airflow. Thoughts?

I recommend SATA SSD where possible. Save any M.2 w/NVMe for purposes that really benefit from it, such as SLOG devices.

I have updated my initial post and will move toward using two SATA SSD for an OS mirror. Thanks for the guidance here!

Thanks for the help and advice!

 

[JTT0]

After receiving limited responses on this thread, I went back to the drawing board to review my choices and perform some additional in-depth research. I have since narrowed my focus to the following setup and am still looking for some additional guidance on the system for my use cases (outlined in my initial post).

Setup:
OS: FreeNAS 11.3
Case: U-NAS NSC-810A
Motherboard: Supermicro X11SSH-CTF
CPU: E3-1275V6
RAM: Supermicro 16GB DDR4 2400 ECC UDIMM Memory RAM [64GB Unbuffered ECC UDIMM, DDR4-2400MHz] (4 x 16GB)
Boot Drives: (2 x mirrored) Kingston 240GB A400 SATA 3 2.5" Internal SSD
Storage: WD Red (8 x 3TB - Oversized RAM allows room to expand HDD sizes over time)
PSU: Athena AP-MFATX50P8 500W OR Seasonic SS-400H1U 80+ 1U 400W
Cable: 2 x CableDeconn 0.7M Internal Mini SAS 36-Pin to SFF-8087 Cable

Thank you so much for your time and assistance!

 

[JTT0]

Due to some sourcing issues, I have revised my RAM to the following complete setup.

Setup:
OS: FreeNAS 11.3
Case: U-NAS NSC-810A
Motherboard: Supermicro X11SSH-CTF
CPU: E3-1275V6
RAM: Crucial 16GB DDR4 2666 MT/s ECC UDIMM RAM [64GB Unbuffered ECC UDIMM, DDR4-2400MHz] (4 x 16GB)
Boot Drives: (2 x mirrored) Kingston 240GB A400 SATA 3 2.5" Internal SSD
Storage: WD Red (8 x 3TB - Oversized RAM allows room to expand HDD sizes over time)
PSU: Athena AP-MFATX50P8 500W (Also considered Seasonic SS-400H1U 80+ 1U 400W)
Cable: 2 x CableDeconn 0.7M Internal Mini SAS 36-Pin to SFF-8087 Cable

Thank you so much for the guidance!