More Storage for my Server - pool layout

[IceBoosteR]

Hello out there,

I want to upgrade my main freenas server. So lets start with what I have now.
I have a Dell T20, which has a Xeon E3-1225v3, 24GB of DDR3 ECC RAM and 4x4TB WD Red Drives. I also have a working backup on a Synology box, also with ECC.
With this pool, I store everything on it. Media, photos, documents,... This one is in RAIDZ-1. You can read this, why I choose it (https://serverfault.com/questions/634197/zfs-is-raidz-1-really-that-bad)
But I observed that I run out of space, and it takes forever to move through my photo folder. I have a loooot of photos, stored in JPG and RAW, as I have a DSLR and the rest of the story is clear ;)
So I would also like to fasten things up here.

First, I will move to a bigger case, the Nanoxia Deep Silence 6 Rev. B would fit my needs. I will also need another PSU, something like the new Seasonic Focus Plus Platinum 550W ATX 2.4 (SSR-550PX). I know, the G-Series of Seasonic was often recommended. I found a company in germany which also makes adapters, so the properatary Dell to normal ATX.
I am sure, this all should be fine, especially for drive temperatures.

So coming to the pool. First of all, a new HBA will be required in IT mode (IBM M1015) or something similar. Disks should be some WD Reds again, I like them.
So for the pool I have some options I thought about, and yes, I know RAIDZ-2 is recommended.

1.
Buy 6x8TB or 6x6TB drives for a new pool.
Migrate everything to the pool using ZFS send and recieve (I defenetly need to read some docs before doing this (syntax), and destroy the old pool. With those drives, built another pool with mirrored vdevs, for photos and zvols for my ESXi servers. On the other pool, media will placed. I have then a little bit more safety in disks failure if 1 drive per vdev dies (hopefully never), but would I expect more IOPS here? So noticeable more? But I can easily upgrade the pool with ore vdevs.
1.1 Maybe adding L2ARC to this pool would be an option.

2.
Buy a lot more 4TB drives, destory the pool, make a new one with eg. 9x4TB in RAIDZ-2 and rebuilt from backup
2.1 Add L2ARC

3.
Adding another vdev with 4x4TB in RAID-Z1 to the existing pool (refering to the link I have posted. In case of a URE I can ask ZFS which file is affected and recreate from backup). This is the cheapest solution of all.

4.
Maybe you got some ideas.

I don't want to put tooo much mony on this, especially when most of this is for media, which is not that important and I have a backup...
And I am not stupid, I know that the extra level of security cost some cash, so I think I can hear the people telling me to choose option 1 ;)
And thats fine, because it's maybe "the best".

Any comment to this, with you thoughts would be nice. Also please consider about the power consumption of more drives. I have seen 12TB drives out in the darkness of the internet :D
Questions to this, because of bad english -> please ask.

Regards
Ice

 

[BigDave]

I don't want to put so much mony on this, especially when most of this is for media, whcih is not that important and I have a backup...

And I am not stupid, I know that the extra level of security cost some cash, so mostly I can hear the people telling me to choose option 1

I'm going to answer #3 because it's cheap! Cheap is ALWAYS the best!
Of course you realize I'm just saying what you want to hear, right...

 

[MrToddsFriends]

2.
Buy a lot more 4TB drives, destory the pool, make a new one with eg. 9x4TB in RAIDZ-2 and rebuilt from backup

My 2 cent:

2*. (modified version of 2.)
Buy 6 more 4TB drives, destroy the pool, make a new one with 2 x (5 x 4TB in RaidZ2) and restore from backup.

Merely by two 4TB disks more expensive than option 3. (two RaidZ1 vdevs) and more IOPS than a pool consisting of a single vdev.

 

[IceBoosteR]

I'm going to answer #3 because it's cheap! Cheap is ALWAYS the best!
Of course you realize I'm just saying what you want to hear, right...

Thats too much ironie in 2 lines...
No I just put some thoughts together, and yes in that case #3 is the cheapest. Never said its the best, and I maybe used som 'derp' spelling. If you guys are telling me, that regardless of anything else, #1 is the best, then I have to think about to do this.
So maybe some missunderstandings here, but I want to have a discussion in those options....

(Edited the initial post for better understandings....)

 

[IceBoosteR]

My 2 cent:

2*. (modified version of 2.)
Buy 6 more 4TB drives, destroy the pool, make a new one with 2 x (5 x 4TB in RaidZ2) and restore from backup.

Merely by two 4TB disks more expensive than option 3. (two RaidZ1 vdevs) and more IOPS than a pool consisting of a single vdev.

Thank you. Maybe in this case I also need to think about L2ARC?

 

[BigDave]

Thats too much ironie in 2 lines...
No I just put some thoughts together, and yes in that case #3 is the cheapest. Never said its the best, and I maybe used som 'derp' spelling. If you guys are telling me, that regardless of anything else, #1 is the best, then I have to think about to do this.
So maybe some missunderstandings here, but I want to have a discussion in those options....

(Edited the initial post for better understandings....)

I apologize for the sarcasm. In the interest of discussion, it's my opinion that using 4TB drives in
RAIDz1 is bad. So option number 3 is doing nothing more than doubling that bad-ness.
In a nutshell, I would not even consider that an option (#3) ;)

 

[MrToddsFriends]

Thank you. Maybe in this case I also need to think about L2ARC?

With 24 GB RAM in your Xeon E3-1225v3 based system? Most possibly not the best idea.
http://doc.freenas.org/11/zfsprimer.html

 

[Stux]

I wouldn't use more than 64GB of L2ARC with just 24GB of RAM. In my tests performance degraded at 128GB.

L2arc very well might help with your photos.

Probably a good idea to step up to either 6 or 8 way RaidZ2 by adding 4TB drives. Then from there grow your pool with additional vdevs and bigger chassis (is 24 bay 4U rack mount)

 

[BigDave]

@Stux has tested a lot with this stuff, so the quote below is not intended as a rebuff or
disagreement. I'm just pointing out the OP's
listed amount of RAM needs to be bumped to 32GB first? (assuming that's maximum for his platform)

"As a general rule, L2ARC should not be added to a system with less than 32 GB of RAM, and the size of an L2ARC should not exceed ten times the amount of RAM."
Quoted from The ZFS Primer

 

[IceBoosteR]

Thanks you guys first for picking this up.
L2ARC was only an idea, because of the photos. But if I am under the recommendations with 24GB.... So I might wait for a plattform upgrade with more RAM, as spending more money on DDR3 is something like waste of money (did you see those prices tho? :O )

listed amount of RAM needs to be bumped to 32GB first? (assuming that's maximum for his platform)

Yes you are right, maximum is 32GB.

So Stux, if I am recreating my pool with 8x4TB drives, I would also have the speed/IOPS of one HDD, so there is no improvement in performance in the first way.
What about option 1, with a dedicated pool for those photos with mirrored vdevs. Easy to increase, and more performance.

Just for interest. I have no jails running (only MiniDLNA for some hours a month), so everything in RAM is for FreeNAS only. Stux do you have some threads or sources with your tests with L2ARC?

Thanks

 

[IceBoosteR]

Push :D

I have seen there is something called Seagate Archive HDD v2 with 8TB available, this would fit to my requirements of cold storage. But what I have seen, there is no ERC for those drives. So no usable for NAS :/
Nevermind, but I would like to hear more opinions^^

Edit: I have read a lot now about vdev performance and things like that.
My thoughts are now, to built 2 pools.
1st one for thoise things which need more IO, so i will go wiht a vdev of 2x4TB or 2x6TB. In case of failure, I have one disk up and running.
2nd pool for media, I would go with one vdev with 6,8 or 10 disks with 4TB. Thats all in the budget and does not require that much power. I have calculated, that 20 watts less power consuption result in a free 3TB drive a year. LOL :D

What do you think? Is pool #1 maybe less protected?

 

[IceBoosteR]

Attached I have a screenshot of the drives which would fulfill my requirements (after checking out the resources section).
Those 10TB drives looks awesome xD


BIG EDIT:
Okey well, this is all awesome to read. After spending hours on hte internet, I found an awesome article which is showing, that moving from RAIDZ1 with 4x4TB to RAIDZ2 with 10x4TB is giving me a performance boost. So maybe all is fine with even this configuration. All I have to do is starting from scratch then. FN11.1 hopefully and all new.
Hers the link:
https://calomel.org/zfs_raid_speed_capacity.html
And another one:
https://icesquare.com/wordpress/zfs-performance-mirror-vs-raidz-vs-raidz2-vs-raidz3-vs-striped/

 

[Stux]

Stux do you have some threads or sources with your tests with L2ARC?

Not so much. I was testing iscsi performance, and the max ram I could afford was 24GB, more was better than less. Then I tested various l2arc sizes, and I found performance degraded with larger l2arcs.

 

[IceBoosteR]

I wouldn't use more than 64GB of L2ARC with just 24GB of RAM. In my tests performance degraded at 128GB.

L2arc very well might help with your photos.

Probably a good idea to step up to either 6 or 8 way RaidZ2 by adding 4TB drives. Then from there grow your pool with additional vdevs and bigger chassis (is 24 bay 4U rack mount)

So this can be marked as closed. I have spend now 1500€ which are about 1780$ for all the stuff. I will recreate my pool now going to 10x4TB in RAIDZ2. Thanks for your suggestion Stux.
I will also create a little blog here I think. Hopefully you will like it ...

 

[Chris Moore]

10x4TB in RAIDZ2. Thanks for your suggestion Stux.

I am not sure you got from this what was intended, but let us know how it goes.