disk arrangement question

[urobe]

Hey there,

I'm currently building my first nas. I have a supermicro x9SCL-F mainboard with an Xeon CPU and 32 gb ecc ram (max what the mainboard supports).

As a case I have a SC823TQ (https://www.supermicro.com/products/chassis/2U/823/SC823TQ-653LPB).

I have now several options to create my zfs2:

- 6x6tb = 36 tb (voilating the one gb ram / tb hdd)
- 6x4tb = 24 tb (would be enough for now, but run out of space sooner than I'd want to)

Getting a different case that supports 8 hdds:

- 8x4tb = 32 tb
this would actually be my preferred choice, but I think to remember from one of the beginners guide that a zfs2 should be setup in 2^n+2 numbers of drives (4, 6, 10,...)
I can't find where I read about this rule, but I think to remember that it had something to do with compression turned on or not.

Once the first system is up and running, I want to build a second one for an off site backup. For this one, I was considering using 8x4tb 2.5" as it would only be synced once a day for about 30-60 minutes.

this setup would give about 6-8 years of freedom. As the system is ram wise maxed out, I'd build a new nas after that time.

So my two main questions are:

- is it a no-no to use 8 drives in a zfs2
- are the seagate barracuda 4tb 2.5" hdds a bad idea for the backup nas (I would like them, because they are much lighter. The daily backup would cover the important work data. Once a month I would bring the backup on site and sync non essential data (movies and such), which are too much traffic for the wifi bridge the off site backup is connected with. Therefor, I wouldn't mind a light setup)

Any help and input is greatly appreciated.
-Peter

 

[Stux]

voilating the one gb ram / tb hdd

should be setup in 2^n+2

Both of those rules of thumb are outdated.

You should be fine. Use whatever size disks you want, in 5-9 way RaidZ2.

If you find your ARC is not delivering the performance you want, then investigate more RAM.

Enjoy.

 

[Arwen]

First, the 1GB RAM per 1TB storage is a rule of thumb. Not carved in stone. You can violate it just fine with 32GB of RAM.

Second, the stripe width and number of disks is much less relevant today when you use compression. See below. Basically
you should enable LZ4 compression on your top level dataset, and not worry about it. LZ4 bails out early for less compressable
data.

https://www.delphix.com/blog/delphi...or-how-i-learned-stop-worrying-and-love-raidz

My suggestion is to go for 6 x 8TB disks. With only 6 disk slots, you have less choices in upgrades. Meaning you can't easily
add a second 6 disk vDev to grow your pool.

 

[urobe]

Both of those rules of thumb are outdated.

You should be fine. Use whatever size disks you want, in 5-9 way RaidZ2.

If you find your ARC is not delivering the performance you want, then investigate more RAM.

OK. Great. I wanted to be sure before I commit.

First, the 1GB RAM per 1TB storage is a rule of thumb. Not carved in stone. You can violate it just fine with 32GB of RAM.

Second, the stripe width and number of disks is much less relevant today when you use compression. See below. Basically
you should enable LZ4 compression on your top level dataset, and not worry about it. LZ4 bails out early for less compressable
data.

https://www.delphix.com/blog/delphi...or-how-i-learned-stop-worrying-and-love-raidz

My suggestion is to go for 6 x 8TB disks. With only 6 disk slots, you have less choices in upgrades. Meaning you can't easily
add a second 6 disk vDev to grow your pool.

Thank you for the Link. Looks like an interesting read which I'll have over lunch.
6x8tb! Wohoo. This would give me even more freedom. I might have to check that with my treasury secretary...

Anyone has an opinion on the seagate 2.5" drives? Or is it such a bad idea, not worth to be answered?

-Peter

 

[Stux]

Anyone has an opinion on the seagate 2.5" drives? Or is it such a bad idea, not worth to be answered?

No real opinion. Yes, it should be lighter... your call :)

 

[urobe]

No real opinion. Yes, it should be lighter... your call :)

:) Alright. Will see what kind of case and mainboard I get, and see if it makes sense.

 

[Stux]

:) Alright. Will see what kind of case and mainboard I get, and see if it makes sense.

Keep us updated, I'm curious what you come up with :)

 

[Arwen]

...
Anyone has an opinion on the seagate 2.5" drives? Or is it such a bad idea, not worth to be answered?

-Peter

Since they are thicker than some lower density drivers, (15mm high), you would need a chassis that allows them.

They do appear to spin slower, (5400 RPM), than Enterprise drives, which tend to be at 7200, 10K or even 15K RPM.
So that's helpful keeping the heat down.

By the way, Seagate does make a 5TB version of the 2.5" Barracuda drives. Another way to reduce weight.

 

[urobe]

Keep us updated, I'm curious what you come up with :)

Will do!

Since they are thicker than some lower density drivers, (15mm high), you would need a chassis that allows them.

They do appear to spin slower, (5400 RPM), than Enterprise drives, which tend to be at 7200, 10K or even 15K RPM.
So that's helpful keeping the heat down.

By the way, Seagate does make a 5TB version of the 2.5" Barracuda drives. Another way to reduce weight.

They do make a 5tb version. Didn't know that. Thanks for the hint.

Regarding the backup nas, I don't need it to be fast, would 8 gb or 16 gb of ram be enough for the 36 / possibly 48 tb dsik space?

 

[Arwen]

...
Regarding the backup nas, I don't need it to be fast, would 8 gb or 16 gb of ram be enough for the 36 / possibly 48 tb dsik space?

Yes. If you go with 8GB of RAM, just make sure you can add more. Meaning don't fill all your memory slots up with that 8GB of RAM.

In general, ZFS likes RAM for it's ARC, (Adaptive Replacement Cache), to improve read performance. So, for a backup NAS, which is mostly write only, you can get away with less RAM. That said, there are some tuning rules for ZFS and it's write transaction. It's based a bit on the performance of the disks, pool layout and network performance, giving the size of the RAM ZFS would prefer. I don't have further information handy. But, the worst case, your backup NAS simply writes slower.

Here is some information on ZFS write transactions;

https://www.delphix.com/blog/delphix-engineering/zfs-fundamentals-transaction-groups

Edit: Fixed wording. Added pool layout part.

 

[urobe]

Yes. If you go with 8GB of RAM, just make sure you can add more. Meaning don't fill all your memory slots up with that 8GB of RAM.

Great! Will do. I'll start with one single 8gb module, and will add more if need be.

 

[Robert Trevellyan]

ARC

<cough>Adaptive Replacement Cache</cough>

 

[Arwen]

<cough>Adaptive Replacement Cache</cough>

I don't know what you are talking about... . My post is perfect, (after some editting).

 

[urobe]

Alright, the die is cast... 6x8tb on the main machine and 6x5tb on the backup.
On the working NAS, I'd like ZFS2, resulting in 32 tb usable space. On the backup drive, I'd would only run ZFS1, giving me 25 tb of usable disk space.

Regarding the 7 tb difference in space, there is a buch of data that is nice to have, but really not essential (OS iso images, and movies) nothing that I can't live without, and all that is possible to get somewhere else from again, quickly (enough).

My thinking behind the ZFS1 on the backup, is that it would first be highly unlikely to need it at all (fire, flooding, lighting,...) and that on top of that two drives would fail on the backup before and can copy it seems rather be unlikely?

Other than that, the two machines use the same mainboard, ram and cpu, so I'd have replacement hardware very quickly.

Does this sound like a plan?

 

[danb35]

On the working NAS, I'd like ZFS2, resulting in 32 tb usable space.

Keep in mind that 32 TB is a little under 29 TiB, and if you keep that less than 80% full as you should, that really gives you only about 23 TiB.

 

[Robert Trevellyan]

ZFS2 ... ZFS1

To minimize potential confusion, please use proper ZFS terminology. Presumably you mean RAIDZ2 and RAIDZ1.

 

[urobe]

Keep in mind that 32 TB is a little under 29 TiB, and if you keep that less than 80% full as you should, that really gives you only about 23 TiB.

Thank you for pointing that out. It's difficult to estimate our data hunger for the future. Right now we have about 8 tb of accumulated data over the years. Looking at the structure of the data, and how it grew, this should give us peace of mind for about 6-8 years.

To minimize potential confusion, please use proper ZFS terminology. Presumably you mean RAIDZ2 and RAIDZ1.

Yes, you're right. That's what I meant to say. Thanks for the correction.