Please check the computer specifications for TrueNAS and give me advice.

[HaNue]

After decades of storing photos and videos on a single hard drive, I found out about TrueNAS while searching for data storage because of the increasing number of hard drives and the smaller rooms.

As a test, I installed it on a computer with z390 / i9-9900k / 64gb / 2070s / 630w and succeeded in sharing zfs and smb, so I'm building a new PC and looking for hardware.

The specifications I'm thinking of
Ryzen5 5600 / ASUS ROG STRIX B550-A / 128GB ECC(64x2) / WD 22TB x 4ea / SF-750W / SATA Controller / 4U Rackmount

I'm thinking about it like this, but is there anything I need to fix?

 

[Davvo]

The following resources are a good way to start understanding TrueNAS.

Introduction to ZFS

This is a short introduction to ZFS. It is really only intended to convey the bare minimum knowledge needed to start diving into ZFS and is in no way meant to cut Michael W. Lucas' and Allan Jude's book income. It is a bit of a spiritual...

www.truenas.com

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Hardware Recommendations Guide

This is the latest edition of the FreeNAS Community hardware recommendations guide. The current major version is R2, dated January 2021, with the last minor update on 2021-01-24. The format has moved away from the forum post form factor, to...

www.truenas.com

That being said there are no issues with your configuration per se, proven you use a good HBA: it will probably work well, but since you are going for rackmount, 22 HDDs (please use CMR) and 128GB of ECC RAM you might as well pick server-grade hardware (things like IPMI are a very nice thing when you don't have an iGPU or a dedicated GPU). It would also open a path to future expansion (scale-up), especially RAM-wise... with that amout of drives, 128GB might feel tight.

Do also note that 2.5 Gbps is a bit notorious for creating issues, even with Intel iirc, and that your PSU might not be enough (see resource linked and verify yourself).

Proper Power Supply Sizing Guidance

I've seen about 1,000 threads like this one where people decide that they can power a dozen hard drives off a 360 watt supply. DO NOT DO THIS. I've seen another 1,000 threads where people decide to buy the cheapest power supply that they can...

www.truenas.com

Finally, 22 drives is quite a lot and won't fit on a single vdev... Are you thinking of 2 vdevs composed by 11 drives in RAIDZ3 each?
This resource can help you understanding the performance you will get by various pool layouts.

https://www.ixsystems.com/wp-content/uploads/2022/02/ZFS_Storage_Pool_Layout_White_Paper_February_2022.pdf

 

[HaNue]

The following resources are a good way to start understanding TrueNAS.

Introduction to ZFS

This is a short introduction to ZFS. It is really only intended to convey the bare minimum knowledge needed to start diving into ZFS and is in no way meant to cut Michael W. Lucas' and Allan Jude's book income. It is a bit of a spiritual...

www.truenas.com

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Hardware Recommendations Guide

This is the latest edition of the FreeNAS Community hardware recommendations guide. The current major version is R2, dated January 2021, with the last minor update on 2021-01-24. The format has moved away from the forum post form factor, to...

www.truenas.com

That being said there are no issues with your configuration per se, proven you use a good HBA: it will probably work well, but since you are going for rackmount, 22 HDDs (please use CMR) and 128GB of ECC RAM you might as well pick server-grade hardware (things like IPMI are a very nice thing when you don't have an iGPU or a dedicated GPU). It would also open a path to future expansion (scale-up), especially RAM-wise... with that amout of drives, 128GB might feel tight.

Do also note that 2.5 Gbps is a bit notorious for creating issues, even with Intel iirc, and that your PSU might not be enough (see resource linked and verify yourself).

Proper Power Supply Sizing Guidance

I've seen about 1,000 threads like this one where people decide that they can power a dozen hard drives off a 360 watt supply. DO NOT DO THIS. I've seen another 1,000 threads where people decide to buy the cheapest power supply that they can...

www.truenas.com

Finally, 22 drives is quite a lot and won't fit on a single vdev... Are you thinking of 2 vdevs composed by 11 drives in RAIDZ3 each?
This resource can help you understanding the performance you will get by various pool layouts.

https://www.ixsystems.com/wp-content/uploads/2022/02/ZFS_Storage_Pool_Layout_White_Paper_February_2022.pdf

Thank you for your answer.

1. I'm glad you said there's no big problem with my composition. Then should I change the composition a little bit and buy it?

2. (Proven you use a good HBA) Does this answer mean SAS/SATA raid controller?

3. Currently, we are planning to upgrade step by step after purchasing four HDDs for WD company's 22TB NAS.
(물론 메모리도 증설 할 것입니다)

4. I'm thinking of two vdevs. What I'm curious about here is, can I use the three vdevs, which are 4 pieces of 22TB, 8 pieces of 16TB, and 8 pieces of 8TB?

5. For the 2.5G network, we need to replace the motherboard, right?

 

[ChrisRJ]

2. (Proven you use a good HBA) Does this answer mean SAS/SATA raid controller?

No, please read the linked resource about HBAs and RAID controllers.

 

[jgreco]

No, please read the linked resource about HBAs and RAID controllers.

(a *thud* echoes thru the forum)

 

[Etorix]

1. I'm glad you said there's no big problem with my composition. Then should I change the composition a little bit and buy it?

If you've not yet bought the system, you may improve by going for a server-grade motherboard, such as AsRockRack B550D4U / X570D4U / X470D4U family of boards (to keep with the Ryzen CPU).

2. (Proven you use a good HBA) Does this answer mean SAS/SATA raid controller?

Basically any SAS controller based on LSI 2008/2308/3008.

3. Currently, we are planning to upgrade step by step after purchasing four HDDs for WD company's 22TB NAS.
(물론 메모리도 증설 할 것입니다)

To upgrade memory beyond 128 GB you need to look into EPYC or Xeon Scalable. (With thanks to Google Translate here.)

4. I'm thinking of two vdevs. What I'm curious about here is, can I use the three vdevs, which are 4 pieces of 22TB, 8 pieces of 16TB, and 8 pieces of 8TB?

It is possible to mix different vdevs in the same pool, but it is best if the vdevs have the same geometry.
Raidz# is appropriate for bulk storage of photos/videos, and 8-wide is a reasonable size. You could have a 8-wide raidz2 (or raidz3) vdev with 8*16 TB and another with 8*8 TB, but then it would be best to have 8*22 TB drives. (You cannot extend a 4-wide raidz2 to 8-wide later.)
Or you could do everything in 4-wide raidz2 vdevs, but that would not be a very efficient use of space.

With drives as large as 22 TB, 2-way mirrors are not advisable, and 3-way mirrors are expensive.

5. For the 2.5G network, we need to replace the motherboard, right?

Best pick a motherboard with server-grade Intel NICs (some AsRockRack motherboards even have 10 GbE on-board).
Otherwise, use a PCIe slot for a Chelsio/Solarflare/Intel card.

 

[HaNue]

No, please read the linked resource about HBAs and RAID controllers.

oh i got it
Thank for your answer

 

[HaNue]

If you've not yet bought the system, you may improve by going for a server-grade motherboard, such as AsRockRack B550D4U / X570D4U / X470D4U family of boards (to keep with the Ryzen CPU).


Basically any SAS controller based on LSI 2008/2308/3008.


To upgrade memory beyond 128 GB you need to look into EPYC or Xeon Scalable. (With thanks to Google Translate here.)


It is possible to mix different vdevs in the same pool, but it is best if the vdevs have the same geometry.
Raidz# is appropriate for bulk storage of photos/videos, and 8-wide is a reasonable size. You could have a 8-wide raidz2 (or raidz3) vdev with 8*16 TB and another with 8*8 TB, but then it would be best to have 8*22 TB drives. (You cannot extend a 4-wide raidz2 to 8-wide later.)
Or you could do everything in 4-wide raidz2 vdevs, but that would not be a very efficient use of space.

With drives as large as 22 TB, 2-way mirrors are not advisable, and 3-way mirrors are expensive.


Best pick a motherboard with server-grade Intel NICs (some AsRockRack motherboards even have 10 GbE on-board).
Otherwise, use a PCIe slot for a Chelsio/Solarflare/Intel card.

Thank you for your answer.

I haven't bought it yet.

Is it better to use all the products as a server?
The revised edition of Hardware recommendations 2021 states that low-cost cpu is sufficient if it is not used for file sharing purposes anymore.

 

[Etorix]

Server-grade motherboards come with better NICs, presumably better ECC support (relevant for a Ryzen build, where ECC support is unofficial), and possibly IPMI for remote management.

A low cost CPU, Core i3 or Ryzen 3, is indeed sufficient for a home NAS. But you mentioned a 4U rack, lots of drives (4*22 TB + 8*16 TB + 8*8 TB), including big drives (16 and 22 TB, with the prospect that 8 TB drives may be upgraded to larger drives as well). That's a lot of data, much more than what was considered when writing the recommendations, and ZFS will benefit from a lot of RAM to handle such a large pool. 128 GB RAM should be comfortable, but if you ever want or need to increase RAM beyond that, you'll need a different platform.

 

[HaNue]

Server-grade motherboards come with better NICs, presumably better ECC support (relevant for a Ryzen build, where ECC support is unofficial), and possibly IPMI for remote management.

A low cost CPU, Core i3 or Ryzen 3, is indeed sufficient for a home NAS. But you mentioned a 4U rack, lots of drives (4*22 TB + 8*16 TB + 8*8 TB), including big drives (16 and 22 TB, with the prospect that 8 TB drives may be upgraded to larger drives as well). That's a lot of data, much more than what was considered when writing the recommendations, and ZFS will benefit from a lot of RAM to handle such a large pool. 128 GB RAM should be comfortable, but if you ever want or need to increase RAM beyond that, you'll need a different platform.

If i use the asrock rack x570,
Can I use up to 3 controllers?

I could only see 3 pci slots

The controller wants to use the product from the link below, is it okay?

https://www.amazon.com/16-PORT-Int-6GB-Sata-Pcie/dp/B003UNP05O?tag=linus21-20&geniuslink=true

Finally, I need more than 200TB of capacity.

So I wanted to make it into a disk with a large capacity on a single disk(ex. 22TB * 10ea)


PS. Once the storage has been built, it will only work when the files are moved and found, and the rest of the time will be closed

 

[jgreco]

Can I use up to 3 controllers?

Yes, but why would you? You only need a single controller. The one you just linked to is a relatively old and overpriced controller.

Find a chassis with SAS expander support, such as a Supermicro SC846BE1C, and then use an LSI 3008 based card like the LSI SAS 9300-8i HBA.

 

[Davvo]

So I wanted to make it into a disk with a large capacity on a single disk(ex. 22TB * 10ea)

Please don't use RAIDZ1. Also, what is "ea"?
Do note that with a 10 disk vdev in RAIDZ2 (using 22TB drives) your usable space (80%) will be 129.84 TB.

 

[jgreco]

Also, what is "ea"?

Each. One of several notations sometimes used to specify quantity.

Do note that with a 10 disk vdev in RAIDZ2 (using 22TB drives) your usable space (80%) will be 129.84 TB.

Given the previous discussion of 8 x 8TB (48TB of RAIDZ2 pool space), if you then added two 8 x 16TB RAIDZ2 vdevs (96TB of RAIDZ2 pool space each), you get 240TB of pool space, or 192TB of usable space.

 

[HaNue]

Yes, but why would you?

Isn't it possible to fit only 8 hard disks with 8 inputs? That's what I understood.

If there are 6 sata pins and 3 pci slots on the main board,
I can fit 30 hdd, right?

 

[jgreco]

Isn't it possible to fit only 8 hard disks with 8 inputs? That's what I understood.

You understood incorrectly. Please check out the SAS Primer.

Don't be afraid to be SAS-sy ... a primer on basic SAS and SATA

With the introduction of SAS 12Gbps, seems like "it's time" to do a braindump on SAS. Work in progress, as usual. History By the late '90's, SCSI and PATA were the dominant technologies to attach disks. Both were parallel bus multiple drop...

www.truenas.com

An SAS expander takes 4 SAS lanes and can make it possible to service 24 or 36 hard drives from that.

If you cascade expanders, you can probably run about a thousand drives off a single PCIe slot. Now, it might not be SMART to do that, I'm just saying you can theoretically do it.

 

[HaNue]

Each. One of several notations sometimes used to specify quantity.



Given the previous discussion of 8 x 8TB (48TB of RAIDZ2 pool space), if you then added two 8 x 16TB RAIDZ2 vdevs (96TB of RAIDZ2 pool space each), you get 240TB of pool space, or 192TB of usable space.

The capacity will continue to grow, so I hope the maximum capacity is high from the beginning (the largest capacity in the future, not the current available capacity)

I consulted with the CEO of the company I'm currently working at, but he said he doesn't need to go to the server level, so is it okay to give up ecc and go to the regular PC level?

Usage is for simple file backups and for finding files when needed. I won't always leave the server on and I'm going to turn it off.

 

[HaNue]

You understood incorrectly. Please check out the SAS Primer.

Don't be afraid to be SAS-sy ... a primer on basic SAS and SATA

With the introduction of SAS 12Gbps, seems like "it's time" to do a braindump on SAS. Work in progress, as usual. History By the late '90's, SCSI and PATA were the dominant technologies to attach disks. Both were parallel bus multiple drop...

www.truenas.com

An SAS expander takes 4 SAS lanes and can make it possible to service 24 or 36 hard drives from that.

If you cascade expanders, you can probably run about a thousand drives off a single PCIe slot. Now, it might not be SMART to do that, I'm just saying you can theoretically do it.

The computer world is very difficult.

I'll try my best to understand Thank you!

 

[jgreco]

I consulted with the CEO of the company I'm currently working at, but he said he doesn't need to go to the server level, so is it okay to give up ecc and go to the regular PC level?

Only if you don't mind some risk of data loss. Show me a "regular PC" with 24 hard drives. It's a server. Refusing to call it a server doesn't make it "not-a-server."

 

[Etorix]

I consulted with the CEO of the company I'm currently working at, but he said he doesn't need to go to the server level, so is it okay to give up ecc and go to the regular PC level?

ZFS without ECC is probably at least as safe as any other file system on ECC. But if this is company storage, holding quite a lot of data, presumably important company data, it is a pity not to do it by the textbook and skimp on ECC RAM.
The IPMI in Server-grade boards is useful to remotely manage and administer storage racks in the basement. What is saved today by going for a consumer board will be wasted time tomorrow when the technician will have to bring a screen and keyboard to the data center to investigate an issue with the server.