Advice on system setup

[jps229]

Hi
After having bitter experience with a USB setup that wasn't tested enough..., I would now like to double-check that I'm aiming for a good setup.
I am planning my system based on my old desktop:
- Board: ASUS R.O.G. Maximus V Formula, LGA 1155, Intel Z77 (6+2 SATA ports)
- CPU: Intel Core i7 3770K, 3.50 GHz, 4-core
- RAM: 32GB DDR3
- Broadcom's SAS 3008, 12Gbit controller
- Boot partition: 2x 960GB Kingston A400 SSD (direct connection to board)
- Data pool for media files: 3x 8TB Seagate HDD 7200rpm (via SAS Controller)
- Data pool for important data (including network boots for various other systems and NFS volumes for a Docker Swarm cluster): 3x 2TB Crucial SSD (direct connection to board)
- Data pool for backups/reserve: 2x 1.5 TB HDD 7200rpm (via SAS Controller)

Questions:
- - Are there any problems in this setup?
- In particular, I still have my former system disk left, a Samsung 840 Pro Basic, 256 GB SSD (approx. 500MB/s write rate). Does it make sense to use this as a cache or metadata disk for one of the pools or does it not benefit me?
- I also have a 2TB and a 3TB WD HDD (7200rpm) left that I could also use as cache/metadata. But I didn't find any precise recommendations in the documentation.
- Since these are not new disks, what happens if they fail? Can the data pool then simply continue to operate without a cache?

Thx for any help!
JPS

 

[Pitfrr]

Hello,

Here is my feedback on your set-up.

Are there any problems in this setup?

The motherboard is consumer/gaming grade motherboard that I wouldn't use in such setup. It also has unnecessary components (wifi, bluetooth, audio) and I'm not sure about the extra controller (AsMedia).
You didn't mention it but you seem to have pickup regular RAM (i.e. non ECC) and anyway the motherboard does not support it (and the CPU also not).

It's not mentioned but I expect the SAS3008 controller to be flashed in IT, right?

Why do you use 2 disks for the boot device? I'd say, for a home usage, one is enough (and you make a backup of your configuration). Of course, it's working and there's nothing wrong in doing so (you can spare some bucks for something else).

You don't say anything about the type of volume you're going to set up (mirror, RAIDz1 or 2...) but I guess, the media might be RAIDz1 (with 3 disks)? Then I would really discourage RAIDz1 with HDD greater than 2TB...

Does it make sense to use this as a cache or metadata disk for one of the pools or does it not benefit me?

No.
But why don't you use the Samsung 256GB SSD as a boot device (instead of the 2x 960GB)?
Because you'll loose most of the space on the 960GB for nothing (and won't be able to use it). Whereas with the 256GB SSD you'll loose less space...

Before considering using cache, you have to max out the RAM (and I'm not talking about the 32GB of the board but then rather more about something like 128GB or so). And for a home usage, you will most likely get no benefit out of it.

I also have a 2TB and a 3TB WD HDD (7200rpm) left that I could also use as cache/metadata.

Same remark as above related to cache.

Since these are not new disks, what happens if they fail?

You'll need to replace them when they fail...
I would recommend you to burn in your disk before using them in a pool (independently if they're new or old by the way) and to keep a close monitoring on them while in use (SMART tests and scrubs).
But this can also have an impact on your choice for redundancy on your pool... If you're using old disks, then maybe a RAIDz2 pool would be advisable for example, instead of a mirror or RAIDz1.

 

[Davvo]

Pitfrr already gave you valuable advice so I will be adding up to that.

- Are there any problems in this setup?

You are running a "gaming" board, but the NIC is Intel so you should be good. You will be using an old (read inefficent) CPU and non-ECC DDR3, but I suppose you are OK with this.

Your HBA needs to be flashed in IT mode.

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

There is little point in mirroring the boot pool; if you want a redundant boot-pool, please read the following resource.

Highly Available Boot Pool Strategy

I've pounded a few versions of this out over the years, but I hate explaining over and over. TrueNAS allows a ZFS boot mirror pool to be created to increase the reliability of the NAS. This sounds great in theory, but there's a flaw. Due to...

www.truenas.com

Going RAIDZ1 with bigger drives than 2 TB is not suggested.

Assessing the Potential for Data Loss

This guide was written to be read from top to bottom without jumps, with the intent of spreading awareness to both new and experienced users; the author of this document assumes the understanding of the concepts explained in the following...

www.truenas.com

- [...] Does it make sense to use this as a cache or metadata disk for one of the pools or does it not benefit me?

- I also have a 2TB and a 3TB WD HDD (7200rpm) left that I could also use as cache/metadata. But I didn't find any precise recommendations in the documentation.

Not only would a metadata disk harm your performance instead of improving it, but it would even endanger your whole pool since losing a metadata VDEV (it's a VDEV even if you use a single drive) will result in losing your entire pool: you have to match the redundancy of the pool (in your case having a RAIDZ1 VDEV you have to setup at least a 2-way mirror as metadata VDEV); about L2ARC you want at least 64 GB before considering it. In any case, you want SSDs (preferably NVMes due to the low latency) for such tasks.

- Since these are not new disks, what happens if they fail? Can the data pool then simply continue to operate without a cache?

L2ARC is disposable, the metadata VDEV is not. Again, having spinning rust as L2ARC is a great idea to kill performance.

 

[jps229]

You don't say anything about the type of volume you're going to set up (mirror, RAIDz1 or 2...) but I guess, the media might be RAIDz1 (with 3 disks)? Then I would really discourage RAIDz1 with HDD greater than 2TB...

Thank you both for the answers.

What is the specific recommendation for a pool with 3x 8TB disks if it is not Raidz1?

 

[somethingweird]

What is the specific recommendation for a pool with 3x 8TB disks if it is not Raidz1?

Answer - 3 way mirror (hehe)

Personally it okay to be RaidZ1 with 3x 8TB as long you have backups and your okay if a 2nd drive fails.

 

[Davvo]

What is the specific recommendation for a pool with 3x 8TB disks if it is not Raidz1?

• 2x drives in a 2-way mirror and a hotspare
• 3x drives in a 3-way mirror
• 4x drives in RAIDZ2
• 4x drives in two 2-way mirrors
• 5x drives in RAIDZ2

ZFS Storage Pool Layout

This amazing document, created by iXsystems in February 2022 as a "White Paper", cleanly explains how to qualify pool performance touching briefly on how ZFS stores data and presents the advantages, performance and disadvantages of each pool...

www.truenas.com

 

[joeschmuck]

There is one big piece missing which is very critical in planning to build a system... What will it be used for? Be specific. What does it need to do for you? How much storage do you need and how important is the data? This drives the number and size of drives as well as the redundancy.

You likely have parts that will operate, but as you already know, they are not idea, for a NAS to store important data unless you maintain a backup elsewhere. If you know what it needs to do, then you can pick the proper components.

EDIT: Damn, your first post does cover some of these. And after reading those requirements, you have the wrong hardware. You need to purchase a system with at least 4 drives, one SSD boot drive, At least 16GB ECC RAM (more is better if you plan to run VM's) and the CPU and motherboard which support ECC. HDD rotational speed can be important for specific needs like low latency or high IOPS, but your pool layout will be crucial as well.