Storage for 60 disks

[leksand]

Help to choose the optimal solution for creating a storage for 60 disks.

supermicro SSG-6049P-E1CR60L server:

• 2 x Intel CLX 4210R processors (10C/20T);
• 64 GB of RAM (16 GB x4), and there is no budget to increase the amount of RAM in the next six months or a year. I would like 512 GB of RAM, there is a small chance that in 3-4 months it will be possible to increase it to 128. There are a lot of slots, the platform supports up to 6 TB of RAM;
• 2 x sata ssd 480 GB intel 4620 for the operating system.
• 60 sas hdd 7200 hard drives of 10 TB each in the disk basket of the server .
• Broadcom 3008 AOC raid controller; IT mode - you can create an array through it or install all disks in JBOD, but it seems to me that there is not enough RAM for zfs.
• 2 x 10 Gb RJ45 LAN (integrated into the platform)

Functionality - it will be file storage + webdav and S3 (built into TrueNAS mininio).

How to split zpool into vdev or leave raid6 from hw raid?
Speed is not needed - it is slow storage of data processed on another server.
It is important to have redundancy to protect against data loss and maximum storage size.
I thought to make 2 disks under hot spare, but the width of the vdev is inconvenient.

P.S.: There were similar queries in the search, but I didn't find an explicit answer, either in the manual.
P.P.S.: If there are best practices and recommendations for splitting zpool and vdev and more detailed information with recommendations / calculation for RAM (in the system requirements 8 GB for basic operation and 1 GB of RAM per disk) - please send a link.

 

[Patrick M. Hausen]

Broadcom 3008 AOC raid controller; IT mode - you can create an array through it or install all disks in JBOD, but it seems to me that there is not enough RAM for zfs

If it is in IT mode you cannot create an array. If it is in IR mode, you cannot use it with TrueNAS. No, not even when you configure each disk as an individual JBOD volume. You simply cannot.

How to split zpool into vdev or leave raid6 from hw raid?

You cannot use HW RAID.

I would create VDEVs 8 or 10 disks wide. 6x 10 disk RAIDZ3 (420 TB net), or 7x 8 disk RAIDZ2 plus 4 hot spare (420 TB net) ... something like that. And plan to upgrade the memory as soon as possible. More VDEVs gives you more IOPS. RAIDZ3 gives you more redundancy than RAIDZ2. Pick your poison.

 

[Ericloewe]

Yeah, 64 GB is on the low side for so much storage. Not necessarily a problem, but definitely the main concern I'd have, all talk of HW RAID aside.

 

[leksand]

If it is in IT mode you cannot create an array. If it is in IR mode, you cannot use it with TrueNAS. No, not even when you configure each disk as an individual JBOD volume. You simply cannot.


You cannot use HW RAID.

I would create VDEVs 8 or 10 disks wide. 6x 10 disk RAIDZ3 (420 TB net), or 7x 8 disk RAIDZ2 plus 4 hot spare (420 TB net) ... something like that. And plan to upgrade the memory as soon as possible. More VDEVs gives you more IOPS. RAIDZ3 gives you more redundancy than RAIDZ2. Pick your poison.

I have already turned on the jbod controller for sure.
I see disks in truenas and can create a zpool from them.

will there be any problems when working with an array of 420 TB and 64 GB of RAM?

 

[leksand]

Yeah, 64 GB is on the low side for so much storage.w Not necessarily a problem, but definitely the main concern I'd have, all talk of HW RAID aside.

that's why I'm thinking about hw raid - there is not enough memory for zfs

 

[Ericloewe]

There is no situation in which hardware RAID would be a solution. For starters, a SAS3008 would be extremely puny and you'd almost certainly have to get a higher-end controller, which would probably cost more than the extra memory.
Since there is no real data integrity or reliability concern from having 64 GB of RAM, there's also no harm in trying it out.

Thing is, this is a really silly configuration unless you happen to have the hardware in-house. The 60-bay chassis are rather expensive and 10 TB HDDs are pedestrian and probably not super price competitive. A 36-bay chassis with 20 TB disks (or even 22 TB, but those still command a premium) would provide similar storage at a probably lower cost, in the same rack space, with less power.

 

[Patrick M. Hausen]

I have already turned on the jbod controller for sure.
I see disks in truenas and can create a zpool from them.

will there be any problems when working with an array of 420 TB and 64 GB of RAM?

Have you turned on "JBOD mode" or have you flashed it with IT firmware? You cannot use the former. You must flash it with IT firmware and then all RAID capabilities will be gone.

Also regarding your consideration of using HW RAID - if you build a giant RAID6 or RAID60, TrueNAS still puts ZFS on that virtual volume. You cannot use any other filesystem but ZFS for storage in TrueNAS. So that does not change the memory situation in any way.

But your data will not be in danger with 64 GB of memory - so agreeing with @Ericloewe, just try it.

But don't even think of using RAID. You will lose data. You have been warned.

HTH,
Patrick

 

[leksand]

There is no situation in which hardware RAID would be a solution. For starters, a SAS3008 would be extremely puny and you'd almost certainly have to get a higher-end controller, which would probably cost more than the extra memory.
Since there is no real data integrity or reliability concern from having 64 GB of RAM, there's also no harm in trying it out.

Thing is, this is a really silly configuration unless you happen to have the hardware in-house. The 60-bay chassis are rather expensive and 10 TB HDDs are pedestrian and probably not super price competitive. A 36-bay chassis with 20 TB disks (or even 22 TB, but those still command a premium) would provide similar storage at a probably lower cost, in the same rack space, with less power.

The equipment is already available, including disks. I didn't choose this configuration - my task is to run it...

The controller is already part of the platform, the description is at the link below, squeeze:
Chipset - Intel ® C621 chipset
SAS - SAS 3 (12Gbps) via Broadcom 3008 AOC; IT Mode
SATA- SATA3 (6Gbps) with RAID 0, 1, 5, 10

6049P-E1CR60L | SuperStorage | Products | Super Micro Computer, Inc.

www.supermicro.com

 

[leksand]

Have you turned on "JBOD mode" or have you flashed it with IT firmware? You cannot use the former. You must flash it with IT firmware and then all RAID capabilities will be gone.

Also regarding your consideration of using HW RAID - if you build a giant RAID6 or RAID60, TrueNAS still puts ZFS on that virtual volume. You cannot use any other filesystem but ZFS for storage in TrueNAS. So that does not change the memory situation in any way.

But your data will not be in danger with 64 GB of memory - so agreeing with @Ericloewe, just try it.

But don't even think of using RAID. You will lose data. You have been warned.

HTH,
Patrick

it mode on the controller is already from the manufacturer. in

1) I ask for advice on splitting and doing hot spare or not?
2) what is the minimum comfortable amount of memory needed?
3) what is the optimal amount of memory guaranteed to solve the needs of the system?

As I understood the main options
6 vdev of 10 raidz2 disks
5 vdev of 12 raidz3 disks
smaller ones - for example, 7 vdev with 8 raid 1 disks plus 4 hot spare.

I have the maximum amount of storage in my tasks with some kind of redundancy/fault tolerance.

 

[Patrick M. Hausen]

Ah - so the SATA RAID controller is probably driving a maximum of 4 SATA disks for booting, connected to the bays in the rear. The SAS (IT mode) HBA drives the storage bays. That makes sense.

If possible disable the RAID function and set the SATA controller to AHCI. Then just perform a mirrored install to both of your SSDs. The installer will do the right thing without any special setup if you pick two drives to install to.

As for your pool layout options see above - I made two suggestions.

HTH,
Patrick

P.S. Quite the beast. I would definitely upgrade the memory and depending on the number of concurrent users consider getting the 6 NVMe drive option for two metadata special VDEVs.

 

[Ericloewe]

1) I ask for advice on splitting and doing hot spare or not?

I would be comfortable doing RAIDZ2 + one or two hot spares. You need to evaluate your own risk, your ability to quickly replace disks, the effect of downtime, etc... More risk/more risk-averse? RAIDZ3. Less so? RAIDZ2.

Either way, have additional cold spares on hand. I keep three cold spares on hand for 24 disks + some other devices that use the same disks. Again, you need to judge for yourself what satisfies your risk mitigation requirements.

2) what is the minimum comfortable amount of memory needed?

I wouldn't spec such a system with less than 128 GB of RAM and I might aim for 256 GB depending on the circumstances.

3) what is the optimal amount of memory guaranteed to solve the needs of the system?

No one can answer that without a very detailed understanding of your actual workload. The answer anyone else can give you is "as much as you can fit in". And more RAM is useful, but it's not magical.

Get started with the 64 GB and analyze the situation from there. You may even get away with not needing more, depending on your workload.

 

[leksand]

By volume it turns out: I didn't change the cost of the disk on the calculator, the disks are already in stock, 10 TB each):
Как я понимаю будет 383-400-414 Тб максимум при использовании zfs

6x 10d raidz2
(Tb) (TB) (%)
Total raw storage capacity: 545.696821 600.000000 100
Zpool storage capacity: 543.000000 597.034814 99.51
Reservation for parity and padding: 129.285714 142.151146 23.69
Zpool usable storage capacity: 413.714286 454.883668 75.81
ZFS usable storage capacity: 400.785714 440.668553 73.44

5x 12d raidz3
((Tb) (TB) (%)
Total raw storage capacity: 545.696821 600.000000 100
Zpool storage capacity: 545.000000 599.233837 99.87
Reservation for parity and padding: 148.636364 163.427410 27.24
Zpool usable storage capacity: 396.363636 435.806427 72.63
ZFS usable storage capacity: 383.977273 422.187476 70.36

7x 8d raidz1 + 4 hot-spare
((Tb) (TB) (%)
Total raw storage capacity: 509.317033 560.000000 100
Zpool storage capacity: 507.500000 558.002151 99.64
Reservation for parity and padding: 80.131579 88.105603 15.73
Zpool usable storage capacity: 427.368421 469.896548 83.91
ZFS usable storage capacity: 414.013158 455.212281 81.29

For comparison:
raid 6 - 560 tb (2 disks failure + 2 hot-spare)
raid 60 - 540 tb (2 disks failure + 2 hot-spare)

 

[Patrick M. Hausen]

58 is an odd number to split into VDEVs, that's why I suggested 7x 8-wide and 4 hot spares even if the 4 hot spares might be overkill. I would definitely not go 13-wide.

 

[leksand]

Ah - so the SATA RAID controller is probably driving a maximum of 4 SATA disks for booting, connected to the bays in the rear. The SAS (IT mode) HBA drives the storage bays. That makes sense.

If possible disable the RAID function and set the SATA controller to AHCI. Then just perform a mirrored install to both of your SSDs. The installer will do the right thing without any special setup if you pick two drives to install to.

As for your pool layout options see above - I made two suggestions.

HTH,
Patrick

P.S. Quite the beast. I would definitely upgrade the memory and depending on the number of concurrent users consider getting the 6 NVMe drive option for two metadata special VDEVs.

the system is already on a 2x sata ssd (sata ahci)

from memory - I expect to add memory in the future.

1) What am I risking: will it slow down when there is a lack of memory or maybe something else? Will the consequences start from the large size of the created zpool or when it is filled with information larger than a certain volume?

2) Are there safe options to get more usable space - 500+ tb?

 

[leksand]

I would be comfortable doing RAIDZ2 + one or two hot spares. You need to evaluate your own risk, your ability to quickly replace disks, the effect of downtime, etc... More risk/more risk-averse? RAIDZ3. Less so? RAIDZ2.

Either way, have additional cold spares on hand. I keep three cold spares on hand for 24 disks + some other devices that use the same disks. Again, you need to judge for yourself what satisfies your risk mitigation requirements.


I wouldn't spec such a system with less than 128 GB of RAM and I might aim for 256 GB depending on the circumstances.

No one can answer that without a very detailed understanding of your actual workload. The answer anyone else can give you is "as much as you can fit in". And more RAM is useful, but it's not magical.

Get started with the 64 GB and analyze the situation from there. You may even get away with not needing more, depending on your workload.

this is a cold storage - there will not be a high load: 5-15 users, and not simultaneous. Load: Write data from the computing server and sometimes watch/copy them.

the option to buy up to 256 GB of memory in the future + at least 2 replacement disks is a good option. It's probably 6-12 months, but now there's a new server and new disks - I hope they won't last that long).

at the same time, the question remains is it possible to make 500+ Tb?

 

[leksand]

С

58 is an odd number to split into VDEVs, that's why I suggested 7x 8-wide and 4 hot spares even if the 4 hot spares might be overkill. I would definitely not go 13-wide.

I agree - 13-wide bad configuration

 

[leksand]

The largest in terms of volume seems to be obtained
6x 10d raidz1 - 514 Tb.

Failure of no more than 1 disk in the array and quick replacement by hand - without hot-spare.

Will something close turn out, but with a failure of up to 2 disks in the array or raidz1 + at least 2 hot-spare?

((Tb) (TB) (%)
Total raw storage capacity: 545.696821 600.000000 100
Zpool storage capacity: 543.000000 597.034814 99.51
Reservation for parity and padding: 60.333333 66.337202 11.06
Zpool usable storage capacity: 482.666667 530.697612 88.45
ZFS usable storage capacity: 467.583333 514.113312 85.69

 

[Davvo]

I wouldn't go RAIDZ1: the second something unexpected happens, you lose your entire pool.
Use RAIDZ2 at the minimum since this is an enterprise-level system.

If you want max space with some degree of protection you could go with 5x RAIDZ2 VDEVs composed of 12 drives each. No hotspares though.

You should consider leaving at the very least 10% of the pool empty.

 

[Patrick M. Hausen]

I would never use RAIDZ1 with 10 TB drives. But you do you. I would be happy to have such a monster with 400 TB of usable space.

 

[leksand]

I discussed the situation with the project manager:

we agreed on the optimal option with a bias in reliability:

6x 9disks in raidz2 and 6 hot spare disks -396 Tb usable storage capacity.

Question: for future recovery (for example, the motherboard burned down) is it worth splitting zpool into 3 pieces: 2 vdev in 9 raidz2 disks?

P.S.: In which case it is easier to find a server with 18 disks than with 60 disks