First NAS Build - Revisited

[stavroswtf]

Hello guys.

So I have been reading around some more on TrueNas and I have been having issues settling for a couple of things. My main use case will be SMB Sharing between my NAS and 5 users with files that are all smaller than 10 mb (PDFs, Documents, Excel Files e.t.c). I would like to have a Windows Server VM running 24/7 and possibly having the possibility of video transcoding for multiple devices. My internet connection is limited to 10 MB/s, so picking the right drives that won't be overkill is an issue.

The build is the following:

PCPartPicker Part List

CPU Cooler: Thermalright AXP90-X36 42.58 CFM CPU Cooler (€39.00)
Memory: Kingston Server Premier 32 GB (1 x 32 GB) DDR4-3200 CL22 Memory (€84.58)
Memory: Kingston Server Premier 32 GB (1 x 32 GB) DDR4-3200 CL22 Memory (€84.58)
Storage: Samsung 870 Evo 250 GB 2.5" Solid State Drive (€41.50)
Storage: Seagate IronWolf NAS 4 TB 3.5" 5900 RPM Internal Hard Drive (€96.90)
Storage: Seagate IronWolf NAS 4 TB 3.5" 5900 RPM Internal Hard Drive (€96.90)
Storage: Seagate IronWolf NAS 4 TB 3.5" 5900 RPM Internal Hard Drive (€96.90)
Storage: Seagate IronWolf NAS 4 TB 3.5" 5900 RPM Internal Hard Drive (€96.90)
Video Card: ASRock Intel Arc A380 Challenger ITX Arc A380 6 GB Video Card (€147.30)
Case: Inter-Tech IM-1 POCKET MicroATX Mini Tower Case (€73.52)
Power Supply: Corsair SF750 750 W 80+ Platinum Certified Fully Modular SFX Power Supply (€186.46)
Custom: Gigabyte Mainboard MC12-LE0 Re1.0 (€59.00)
Custom: AMD Ryzen 5 PRO 4650G Processor (€98.45)
Custom: Asus Hyper M.2 X16 Card v2 (€41.86)
Total: €1243.85

So I have some questions I need help with. Those are:
1. Is Samsung 870 Evo 250 GB good enough for the boot pool? I was thinking of only using 1 drive since I can just backup and restore the config if something goes wrong.
2. What drives (and sizes) should I choose for installing apps and running the Windows VM that needs around 50-60GB? Would NVME be overkill? My conservative approach would be 2x Samsung 870 Evo SSD 500GB.
3. I was thinking of adding 2x NVMEs for mirrored metadata vdev so I could make accessing the directories faster when using SMB Sharing with multiple concurrent users. What size would be a good start and also have room for expandability? Would be choosing NVME/M.2 be overkill for my disk size and internet connection?
4. Could I just somehow use just 3-way big NVMEs and split them for installing apps / VMs and metadata vdev?
5. Should I choose RAIDz2 or mirror setup for my 4 HDDs? I was thinking of RAIDz2 since my internet connection quite limits me for the extra performance the 4-drive mirror.

For the above drives, could you also recommend some specific brands or/and specific models I could use? I am kinda buffled because I have been seeing people recommending certain brands and models that other say they have issues with.

Lastly, I want to thank DigitalMinimalist for helping me out through PMs with a couple of things regarding restructuring my build.

 

[Etorix]

1. Yes, anything cheap (and small) goes. On this board my preference would go to any cheap M.2 NVMe drive, since there's no other good use for the x1 (!) M.2 slot. (Optane M10 16 GB: $7 on eBay, free shipping from Shenzhen included. Help recycle e-waste and grab a few.)
2. Whatever you want. NVMe is a good way to save the six SATA ports for HDDs.
3. "Expandability" is a given for mirrors (2- or 3-way special vdev) but a dubious concept for L2ARC—and a persistent metadata L2ARC would not require much space. With a comfortable 64 GB RAM for a comparatively puny data pool, it's possible you don't even need either: Pool metadata will just be a permanent resident once ARC is warm.
4. You could, but that would be asking for trouble.

Why a 4650G PRO? APUs can only bifurcate x8x4x4, and you have no apparent use of the Vega iGPU. Get a 3600 or 5600 and enjoy x4x4x4x4 bifurcation.

 

[stavroswtf]

1. Yes, anything cheap (and small) goes. On this board my preference would go to any cheap M.2 NVMe drive, since there's no other good use for the x1 (!) M.2 slot. (Optane M10 16 GB: $7 on eBay, free shipping from Shenzhen included. Help recycle e-waste and grab a few.)
2. Whatever you want. NVMe is a good way to save the six SATA ports for HDDs.
3. "Expandability" is a given for mirrors (2- or 3-way special vdev) but a dubious concept for L2ARC—and a persistent metadata L2ARC would not require much space. With a comfortable 64 GB RAM for a comparatively puny data pool, it's possible you don't even need either: Pool metadata will just be a permanent resident once ARC is warm.
4. You could, but that would be asking for trouble.

Why a 4650G PRO? APUs can only bifurcate x8x4x4, and you have no apparent use of the Vega iGPU. Get a 3600 or 5600 and enjoy x4x4x4x4 bifurcation.

5. Should I choose RAIDz2 or mirror setup for my 4 HDDs? I was thinking of RAIDz2 since my internet connection quite limits me for the extra performance the 4-drive mirror.
If I choose the 5600, will I have a problem with that specific ECC Ram and motherboard?

 

[chuck32]

5. Should I choose RAIDz2 or mirror setup for my 4 HDDs? I was thinking of RAIDz2 since my internet connection quite limits me for the extra performance the 4-drive mirror.

Aside from IOPS the performance for read write should be on par between the both.

Are all users remotely accessing or why do you see the ISP speed as a limit?

RAIDZ2 isn't as flexible as striped mirrors (you can just add another mirror to add capacity) but offers more resilience. Any two drives can fail vs. maybe two drives can fail, depending whether they are part of the same Vdev or not.

Ultimately you have to decide - how likely are you to run out of storage midterm?

 

[stavroswtf]

Are all users remotely accessing or why do you see the ISP speed as a limit?

Thanks for all the insight man.

As for ths ISP concern, I thought SMB shares are limited to your internet connection. Am I wrong?

 

[DigitalMinimalist]

Either dGPU & 5600, or 4650G

The APU iGPU is fine unless you need to transcode multiple 4k streams in parallel - and saves you one PCIe slot.

The m.2 slot is perfect for OS: 32GB are more than sufficient for TNS.

SSD for VMs is a good idea.
I would go with striped mirror for the storage pool and use NVMEs in Asus Hyper M.2 for VMs. Then you have two additional SATA available for later storage increase…

But I would really consider to go 2x16TB mirror - easier to handle, and saves power due to less disks

 

[chuck32]

As for ths ISP concern, I thought SMB shares are limited to your internet connection. Am I wrong?

Yes, you're wrong. Why would your ISP speed limit your local SMB speeds? It's not like the data is taking a stroll through the Internet just because of SMB.

If all users / machines are on the same LAN then you are not limited by your ISP speed. The throughout is then determined by the NICs and Switch(es).
If the users are not local, then it's limited by your ISP speed, but in that case it would be limited by that for any type of share you provide, not just SMB.

 

[stavroswtf]

Yes, you're wrong. Why would your ISP speed limit your local SMB speeds? It's not like the data is taking a stroll through the Internet just because of SMB.

If all users / machines are on the same LAN then you are not limited by your ISP speed. The throughout is then determined by the NICs and Switch(es).
If the users are not local, then it's limited by your ISP speed, but in that case it would be limited by that for any type of share you provide, not just SMB.

Oh I see, haven't really tried SMB. I just thought that was the case. Thank you for clarifying man!

 

[stavroswtf]

Either dGPU & 5600, or 4650G

The APU iGPU is fine unless you need to transcode multiple 4k streams in parallel - and saves you one PCIe slot.

The m.2 slot is perfect for OS: 32GB are more than sufficient for TNS.

SSD for VMs is a good idea.
I would go with striped mirror for the storage pool and use NVMEs in Asus Hyper M.2 for VMs. Then you have two additional SATA available for later storage increase…

But I would really consider to go 2x16TB mirror - easier to handle, and saves power due to less disks

Thanks a lot again man.

I am kinda afraid of using 2x16TB drives just because I can only lose and not lose everything.

Also, I think 16TB is too much space for now, maybe doing 4x8TB would be better for my sanity I guess haha

 

[chuck32]

2x16 is the same as 4x8 ;)

As for the data loss fear: remember that raid is an availability solution, not a backup. You still need backups with any raid configuration.

 

[stavroswtf]

2x16 is the same as 4x8 ;)

As for the data loss fear: remember that raid is an availability solution, not a backup. You still need backups with any raid configuration.

Yes I know.

I was thinking of 4x8TB so that I could use 2x8TB for my work stuff, 1x8TB for personal (movies, photos .e.t.c) and 1x8TB for spare backup for the 2x8TB mirror. Otherwise, I will be sticking with 4TB drives.

That could work if I could work up some more money, right now it's tight I guess.

 

[chuck32]

Sorry, I didn't read the whole thread. I thought we were talking a single pool of 4 HDDs.

4*8 TB will leave you with 16 TB usable, regardless whether you go RAIDZ2 or mirrors. You cannot use 2*8TB for work, 1 for personal etc. 2x the drive capacity is gone for parity.
Create a pool and make separate datasets to manage your data.
Your latest config would be 1 mirror, 1 stripe and 1 spare. The stripe would not have redundancy.

Edit:
I'm thrown off by the price difference, but I'm also looking at Seagate exos for my next expansion. They run you around 250 for 16 TB.

I just realized we were talking 4x4 TB not 8 TB ;) scale down what I posted accordingly.

 

[Etorix]

5. Should I choose RAIDz2 or mirror setup for my 4 HDDs? I was thinking of RAIDz2 since my internet connection quite limits me for the extra performance the 4-drive mirror.

Assuming one pool, with appropriate datasets, mirrors are more flexible (can add a further two drives later) but may become dangerous as you swap in more drives (the scare about 2*16 TB is a right one!) and going to 3-way mirrors is not financially attractive; raidz2 is safer, especially with larger drives later on, but cannot be extended.
My choice would be six drives right now in raidz2. Whatever size you can afford. 4 TB is probably more expensive per TB than 10+ TB at this point, and I'm always amazed that people still go for such small sizes.

If I choose the 5600, will I have a problem with that specific ECC Ram and motherboard?

Most desktop Ryzen support ECC. I think all 3000 and most 5000 except for the very latest batch such as 5500. Check on AMD's site—but 5600 does ECC.

 

[DigitalMinimalist]

Oh I see, haven't really tried SMB. I just thought that was the case. Thank you for clarifying man!

First of all: ONE storage pool with RAIDZ2, or (striped) mirrors - independent which HDD size… (even though, mirrors allow for easier size increase)
Purposes and access are managed below in Datasets…

HDDs: best price per TB are currently 16-18TB drives… 4TB for 94€, if 16TB is 280€?
As a rule of thumb: each HDD adds 5W power consumption in idle if not in spindown.

How are the users accessing the data? Via Internet, or Intranet (same local network)?
Very different scenarios…
If Intranet: consider 10Gbit Network

 

[stavroswtf]

First of all: ONE storage pool with RAIDZ2, or (striped) mirrors - independent which HDD size… (even though, mirrors allow for easier size increase)
Purposes and access are managed below in Datasets…

HDDs: best price per TB are currently 16-18TB drives… 4TB for 94€, if 16TB is 280€?
As a rule of thumb: each HDD adds 5W power consumption in idle if not in spindown.

How are the users accessing the data? Via Internet, or Intranet (same local network)?
Very different scenarios…
If Intranet: consider 10Gbit Network

Thanks for your input man. Although I have a question:

How would 10Gbit Intranet help with my case? Let's assume 5-10 users access small files concurrently, wouldn't the 1Gbit be enough? I am not asking condescendingly, just trying to understand.

 

[DigitalMinimalist]

What is exactly the application scenario for your SMB shares?

1Gbit is 125MB/s

Define small size…
If this is photo editing and 5 users are uploading/downloading 25MB files (small for me) then you run into bottleneck…

Especially if the requested data are in the ARC (RAM), from where you can access much faster…

 

[stavroswtf]

Define small size…

Files that are all smaller than 10 mb (PDFs, Documents, Excel Files e.t.c). Mostly smaller than 5MB.
Users will be reading files a lot throughout the day but rarely uploading any. Reading/Writing files would probably have a 30/1 ratio.

 

[chuck32]

Reading / Writing is both capped to that speed ;)

@DigitalMinimalist has a point, if you have the 5 users on a regular basis in parallel it may make sense to give the server a 10G NIC. This way it can supply 5x 1G to the end users without being a bottleneck. A single user writing a larger file will take up bandwidth.
OTOH it's nothing you can't retrofit if you feel the need.
Especially with a lot of reading you would lose your HDDs as a bottleneck, given that the requested files (if multiple users read/edit the same files) are probably in memory anyway.
A switch and 10G NIC will probably run you upwards of 200 Eur though.

 

[DigitalMinimalist]

A switch and 10G NIC will probably run you upwards of 200 Eur though.

Get one of the cheap unmanaged switches from AliExpress for 50€ + Intel X520-DA2 for 30€ and you are good.

If 1GBit connection speed for all 5 users is sufficient: don’t be bothered with special vDev and other stuff: 100% rust (HDD) pool is fast enough to saturate

 

[stavroswtf]

Get one of the cheap unmanaged switches from AliExpress for 50€ + Intel X520-DA2 for 30€ and you are good.

If 1GBit connection speed for all 5 users is sufficient: don’t be bothered with special vDev and other stuff: 100% rust (HDD) pool is fast enough to saturate

Will be getting a switch anyways so buying that Intel X520-DA2 could actually be worth it in the future (if I manage to actually find it on a low price, in my country there are shops that sell it for 300€ haha).