25 TB storage: how to setup the smart way?

Mr. Slumber

Contributor
Joined
Mar 10, 2019
Messages
182
I have to setup up 25TB of useable storage (not raw) in a FreeNAS machine. Still no disks were purchased so still some flexibility left. What would be a smart way?

a) 4x 10TB HDDs, one ZFS Pool with one vdev and raidZ1. pros: just 4 disks cons: expensive (cost per GB storage isn't the best with 10TB drives), just one disk can fail

b) 6x 6TB HDDs, one ZFS Pool with one vdev and raidZ2. pros: cheaper (better cost per GB ratio), 2 disks can fail cons: 6 disks, not quiet 25TB space useable

c) ?

Of course there always has to be compromise between performce (one or more vdevs), cost ($ per GB ratio), redundancy (raidZ1, raidZ2).

Thanks for thoughts and tips! :)
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Here is a link to a fair calculator to use in deciding how much "usable" capacity will remain after all overhead is accounted for AND the fact that you should not fill a ZFS storage pool above around 80% capacity.
https://wintelguy.com/zfs-calc.pl

Also, keep in mind that, as a best practice, you are not supposed to use less than RAIDz2 with spinning disks larger than 1TB.
The attached graphic is a suggestion:

1553007839010.png
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
PS. I didn't make the cost accurate.
 

Mr. Slumber

Contributor
Joined
Mar 10, 2019
Messages
182
the fact that you should not fill a ZFS storage pool above around 80% capacity

Thank you Chris, I did not know that. :) So I may have to reconsider on how many drives I'll have to buy.

Which way would you go? More smaller drives (risking that which each drive you add to a system you also add a one more component that could fail) or better go with fewer but larger drives? (cost is not irrelevant but speaking of storage main concern is data integrity).
 
Last edited:

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,464
Which way would you go?
6 x 8 TB RAIDZ2. Shuck the 8TB disks from WD External drives at $150 each (either Amazon, the WD Elements line, or Best Buy, the WD EasyStore line on sale).
 

Mr. Slumber

Contributor
Joined
Mar 10, 2019
Messages
182
Shuck the 8TB disks from WD External drives at $150 each (either Amazon, the WD Elements line, or Best Buy, the WD EasyStore line on sale).

Heard about this but first thought was: a real smart way to get some drives cheap but are these drives of good quality? I don't know anything about HDD manufacturing but as an engineer I would say: let's not put enterprise class drives in our external hdd products and sell them for cheap... ;)

What I mean is, is it smart to build an entire FreeNAS system with ZFS, ECC Ram and all those things to uphold data integrity and then implant some cheap drives... ;) Like building a Ferrari and then change the motor to a VW Beetle motor ;)

But thanks for this tip! :)
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,464
are these drives of good quality?
All indications is that they're simply WD Red drives (sometimes with the red label, sometimes with a white one). Those have a pretty good reputation around here.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Like building a Ferrari and then change the motor to a VW Beetle motor
Not really the same. The CPU is the motor, the hard drives are more like the tires.
Take a look at this video about taking the drives out:
https://www.youtube.com/watch?v=vVKCVzf8_4w

The manufacturer does modify the firmware to run the drives at a lower RPM, from what I have seen, but these drives appear to be the same drives that are sold as WD Red drives when not inside a USB enclosure.
 

Mr. Slumber

Contributor
Joined
Mar 10, 2019
Messages
182
Take a look at this video about taking the drives out:
https://www.youtube.com/watch?v=vVKCVzf8_4w

Thank you that me be a way to go. Definitely a clever move "cost per TB"-wise... :cool:
But it still bothers me wether these drives are as reliable as the WD Red ones without the firmware mods and so on. And the WD Red (sorry, it's not my intention to start a flame war here! ;)) are of course good and reliable drives but no enterprise class drives.
But getting them for this cheap... hmm, quiet seductive :)
 

melloa

Wizard
Joined
May 22, 2016
Messages
1,749
I have to setup up 25TB of useable storage (not raw) in a FreeNAS machine.

I might have missed, but how many drives can your setup handle?

But it still bothers me wether these drives are as reliable as the WD Red ones without the firmware mods and so on. And the WD Red (sorry, it's not my intention to start a flame war here! ;)) are of course good and reliable drives but no enterprise class drives.

That is true, but I've never lost a volume due to HD failure and all are WD Red ;)
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
the WD Red (sorry, it's not my intention to start a flame war here! ;)) are of course good and reliable drives but no enterprise class drives.
WD Red drives are not perfect by any means and I am no champion for WD. If you really want enterprise class drives, I have a server filled with 60 of the Seagate Exos 10 TB drives at work that have been flawless in their first five months of operation. I wouldn't hesitate to suggest those as being reliable and worthy.
 

Mr. Slumber

Contributor
Joined
Mar 10, 2019
Messages
182

Mr. Slumber

Contributor
Joined
Mar 10, 2019
Messages
182
6 x 8 TB RAIDZ2. Shuck the 8TB disks from WD External drives at $150 each

Couldn't resist so I ordered some for €145 = US $164 (which is an unbelievable low price over here). Definitely will post here on how it went. Thanks again for this hint :)
 

anmnz

Patron
Joined
Feb 17, 2018
Messages
286
I have to setup up 25TB of useable storage (not raw) in a FreeNAS machine. Still no disks were purchased so still some flexibility left. What would be a smart way?

What will the storage be used for?

RAID-Z2 is great but it's going to make you sad if you need high IOPS, e.g. backing storage for VMs.
 

Mr. Slumber

Contributor
Joined
Mar 10, 2019
Messages
182
FOR THOSE ABOUT TO SHUCK... ;)

Found a very good WD Easystore 8TB Compendium regarding very many aspects. Also some more info on different drive types.

Also this video as a complement to the one Chris Moore posted (thanks again :)) for all "shuckers" ;) (it's about the 10GB version but helpful too)

If you want to shuck the drives in a non destructive way (e.g. for warranty re-insertion) I found this toolkit very very useful and really like the quality for the price: iFixit Essential Electronics Toolkit

Be warned, maybe these drives won't work with your setup (3,3V Pin, please read the compendium) and maybe (but these are just my 2 cent) a company which is selling HDDS in large quantities worldwide may have some good (internal) reason for selling 8TB this cheap which in most cases is not altruism... ;) (Or as my grandma used to say: "I can't afford cheap" ;))

For me I will try out this way and if it doesn't work out: ok, lesson learned and move on. :)




EDIT: 2019.03.21 added some more links
EDIT: 2019:03.23 added a link for "shucking tools"
 
Last edited:

Mr. Slumber

Contributor
Joined
Mar 10, 2019
Messages
182
What will the storage be used for?

VM backups (nightly), and lots and lots of office data including videos (no databases included). So not really high performance needed I think.
 

Mr. Slumber

Contributor
Joined
Mar 10, 2019
Messages
182
Ok, my 8TB WDs "My book" have arrived.

Just a quick tip: if you own a Mac (also possible with Linux and Win of course; please take a look at the compendium mentioned above) and want to know what you get before you "free" the HDD I recommend DriveDX for MacOS (can be done via CLI of course too).
Just connect the drive as you would normally do to your Mac, fire up DriveDX and... see for yourself :)

So this is most definitely an enterprise class HGST Ultrastar He10-8 SATA but you may have the 3.3V pin problem. You know that before you shucked it and now you can decide if you really want to risk further trouble. For me I must say I'm not quite sure but still hoping that having a very modern PSU I may not have the 3.3V problem with this drive... :oops:


WD.jpg
 
Last edited:

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,464
For me I must say I'm not quite sure but still hoping that having a very modern PSU I may not have the 3.3V problem with this drive
I haven't had the problem with my Supermicro backplane, but that's the only way I've used them. However, a Molex power adapter is an easy way to address the issue if you encounter it.
 

Mr. Slumber

Contributor
Joined
Mar 10, 2019
Messages
182

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,464
Top