Before I pull the trigger...

[FreeBug]

I'd like to ask a few questions

The Plan: Replace my Patriot Javelin S4 with a FreeNAS box with a max capacity of 32TB. (Max because I am strictly following the 1GB of RAM for every 1TB of storage philosophy)

Server Uses: PLEX, Audio/Video/Photo Production Storage

Planned Build (If you're curious. It's pretty unexciting. I am following CyberJock's thread, and am buying all recommended parts, minus the chassis)

MOBO: SuperMicro X9SCM-F
CPU: Xeon E3-1230 v2
RAM: 8GB to 32GB
HDD: Up to 8 Consumer Grade NAS Drive (Suggestions Welcome)
PSU: 80Plus Gold (Suggestions Welcome)
CHASSIS: NZXT Source 210
EXPANSION: IBM ServeRAID M1015 (If or when I need it)
FANS: Undecided (Suggestions Welcome)

Questions

I have been consolidating my data over the past year, deleting copies, and organizing. I'm projecting that when I complete this task I will have somewhere in the neighborhood of 1.5 to 2TB of data. I know my future plans (ripping my Blu-ray collection, and getting more into audio, video, and photo production) will necessitate a lot more storage than I currently have, but who knows how long it will take me to accumulate enough data fill a 32TB server.

So, my primary question is should I max out the system now, or let it grow with me? I'm always worried that something will become scarce if I don't buy what I need now. If any body remembers Samsung's so called "magic memory" I bought 12GB of it for my gaming rig a few years ago. Now, it's difficult to find and I take personal offense to that empty DIMM slot. :p Anyway, I also tend to buy hardware as I find awesome deals. The drawback of course is it takes longer to get my system fully functional. I could very likely use two old 3TB Barracudas in RAID1 for the time being, and purchase actual NAS drives when I find awesome deals, but I'd like your opinions before I cause a mess, which brings us to the next question

RAID10 or RAIDZ: According to this online calculator a 32TB RAID10 system will nearly max out my gigabit LAN. That's a tempting proposition, because you know SPEED and whatnot, but RAIDZ has better fault tolerance and the useable space is at least 30% more than RAID10. I understand it's a trade off, but I'm tempted to start out with RAID10, and switch over to RAIDZ once I reach the RAID10's capacity. I'm sure at that point it would be a huge pain to migrate all my data from RAID10 to RAIDZ, but again it's best to ask before the mess.

Last question, HDD capacity? Since I am following a 1:1 ratio of RAM to storage, it's probably best if I buy 4TB drives (easy math). But, when I calculate the unit prices (Price / Capacity) of HDDs 3TB is always the cheapest. As an example, I attached a chart comparing the unit prices of WD Red Drives (Non Pro). I know, I know, you're think WTF, but I'm nitpicky with unit price, and a couple of stickies claim that 4TB is the sweet spot, so if I've been wrong all this time I'd like to correct that.

Alright, that's it. Thanks for sticking with it this far. I appreciate it.

 

[Bidule0hm]

RAID-Z (RAID-Z1) is not recommended for more than 3 drives and for drives bigger than 1 TB. Generally people use RAID-Z2 with 6 to 12 drives per vdev (2 drives for the parity, the others for data). Personally I use RAID-Z3 with 8 drives (3 parity drives, 5 data drives) and even if RAID-Z3 is the slowest of the three I've more than enough speed to saturate a gigabit link (more than 4 in fact...) so don't base your RAID strategy on speed, whatever you chose (or almost, don't be dumb either...) you'll saturate a gigabit link ;)

Note that you don't need to follow exactly the 1GB/1TB rule, especially for a home server.

If you include the cost of the server the 4TB are cheaper than the 3TB drives for the same storage space speaking of $/TB.

 

[Ericloewe]

RAID-Z (RAID-Z1) is not recommended for more than 3 drives and for drives bigger than 1 TB. Generally people use RAID-Z2 with 6 to 12 drives per vdev (2 drives for the parity, the others for data). Personally I use RAID-Z3 with 8 drives (3 parity drives, 5 data drives) and even if RAID-Z3 is the slowest of the three I've more than enough speed to saturate a gigabyte link (more than 4 in fact...) so don't base your RAID strategy on speed, whatever you chose (or almost, don't be dumb either...) you'll saturate a gigabyte link ;)

Note that you don't need to follow exactly the 1GB/1TB rule, especially for a home server.

If you include the cost of the server the 4TB are cheaper than the 3TB drives for the same storage space speaking of $/TB.

Gigabit. Gigabit link.

A gigabyte link would be nearly as fast as 10GbE.

 

[Bidule0hm]

Yes, sorry for that, I know the difference but it was nearly 5 am here when I posted... I'll correct that ;)

 

[FreeBug]

Gigabit. Gigabit link.

A gigabyte link would be nearly as fast as 10GbE.

And besides, who would want something like a gigabyte link anyway? We were all perfectly happy with 10/100 :p

Back on subject though, Bidule0hm makes some good points. I'll go with integrity over performance. Let's consider this question answered.

For HDD capacity, if you calculate the cost per terabyte based on the total cost of the rig then larger capacity drives would always be cheaper. By my math, a 6TB rig would be cheaper than a 4TB, even though a 6TB drive is more expensive per gig. I hate to keep beating this drum but I feel like I'm missing something here. Would 6TB be the cheapest option?

How far away from the 1GB:1TB rule can I deviate before I will see a performance hit? I can have a max of 32GB of RAM. That's a given. Now, if my statement above about HDD capacity is true (larger drives are always cheaper) and if I were to go with 7 5TB HDDs or 6 6TB HDDs would I notice any kind of performance hit? This is rig is for PLEX, but I will I began exploring advanced features of FreeNAS in the future. I don't want to take a performance hit because I am demanding too much of the system.

A new question: Is Kingston RAM still a gamble? According to CyberJock's hardware recommendations Kingston is playing games with their part numbers. When I search Google for builds with my MOBO I usually see Kingston kits in the parts list. I've checked out the Tested Memory page on SuperMicro for my board but am having trouble finding the memory listed there. I'm not left with many options outside of Kingston it seems.

As always, thanks for the help. I appreciate it.

 

[jgreco]

32TB. (Max because I am strictly following the 1GB of RAM for every 1TB of storage philosophy)

That rule is much squishier out there at 32GB. Don't freak out about it. It's more important at the lower end.

Last question, HDD capacity? Since I am following a 1:1 ratio of RAM to storage, it's probably best if I buy 4TB drives (easy math). But, when I calculate the unit prices (Price / Capacity) of HDDs 3TB is always the cheapest. As an example, I attached a chart comparing the unit prices of WD Red Drives (Non Pro). I know, I know, you're think WTF, but I'm nitpicky with unit price, and a couple of stickies claim that 4TB is the sweet spot, so if I've been wrong all this time I'd like to correct that.

Add up the cost for your entire system with 3TB drives, then divide that by the number of usable TB you get.

Add up the cost for your entire system with 4TB drives, then divide that by the number of usable TB you get.

Then think about costs to expand from 3 to 4 or 6TB later, and whether or not you'd be likely to have to do that.

Comparing per-TB costs of individual hard drives is a chump's game. It has nothing to do with the actual capex requirements for your filer.

 

[cyberjock]

A new question: Is Kingston RAM still a gamble? According to CyberJock's hardware recommendations Kingston is playing games with their part numbers. When I search Google for builds with my MOBO I usually see Kingston kits in the parts list. I've checked out the Tested Memory page on SuperMicro for my board but am having trouble finding the memory listed there. I'm not left with many options outside of Kingston it seems.

If you go to Crucial's website you'll find they have recommended modules that people have used and have no reported issues.

 

[Jailer]

Ok considering what you offered up as your proposed build if you want to get to 32TB of space with a RAID Z2 vdev using 8 drives you are going to have to use 6TB drives. This will leave you with 32.7TB of formatted space. BUT, you must also take into account the 80% figure for usable space before performance tanks so your practical usable space would be around 26.8TB. So you'll have to go with more drives to reach your proposed capacity of 32TB if that is indeed what you truly need.

ETA: The motherboard you have listed in your OP only has 6 SATA connectors. If you want more than 6 drives you'll have to purchase an add in HBA card or choose a different motherboard with more SATA connectors.

 

[Z300M]

And besides, who would want something like a gigabyte link anyway? We were all perfectly happy with 10/100 :p

Back on subject though, Bidule0hm makes some good points. I'll go with integrity over performance. Let's consider this question answered.

For HDD capacity, if you calculate the cost per terabyte based on the total cost of the rig then larger capacity drives would always be cheaper. By my math, a 6TB rig would be cheaper than a 4TB, even though a 6TB drive is more expensive per gig. I hate to keep beating this drum but I feel like I'm missing something here. Would 6TB be the cheapest option?

How far away from the 1GB:1TB rule can I deviate before I will see a performance hit? I can have a max of 32GB of RAM. That's a given. Now, if my statement above about HDD capacity is true (larger drives are always cheaper) and if I were to go with 7 5TB HDDs or 6 6TB HDDs would I notice any kind of performance hit? This is rig is for PLEX, but I will I began exploring advanced features of FreeNAS in the future. I don't want to take a performance hit because I am demanding too much of the system.

A new question: Is Kingston RAM still a gamble? According to CyberJock's hardware recommendations Kingston is playing games with their part numbers. When I search Google for builds with my MOBO I usually see Kingston kits in the parts list. I've checked out the Tested Memory page on SuperMicro for my board but am having trouble finding the memory listed there. I'm not left with many options outside of Kingston it seems.

As always, thanks for the help. I appreciate it.

The Hynix RAM I bought (which is on the Supermicro list) was a dollar or two cheaper than the Kingston RAM I had bought earlier (which was not on the Supermicro list but on the Kingston list -- until it wasn't and was then even discontinued altogether). I bought my Hynix RAM from Memory America (with whom I have no connection apart from being a one-time purchaser).

 

[FreeBug]

Thanks for the help. I really do appreciate it :)

I ended up going with a few different parts than listed above. I bought a e3 1270 v2, 4x8GB of Samsung M391B1G73QH0-YK0, and though it doesn't play a vital role, a NZXT Source 220. I've also got a couple 6TB Reds on the way. I'm going to slowly add drives over the next year or so until I reach a max of 8 in RAIDZ2. Concerning additional drives, can I add drives and update my RAID configuration easily or will I have to rebuild the array from scratch?

... you must also take into account the 80% figure for usable space before performance tanks so your practical usable space would be around 26.8TB. So you'll have to go with more drives to reach your proposed capacity of 32TB if that is indeed what you truly need.

I've been doing some light reading on the 80% figure you sited. Oracle mentions something called immutable files, but I'm not sure it's a definite solution. Kind of a bummer that I can only get 80% before performance suffers.

I've also done some reading on how to calculate my PSU wattage. Frankly, I'm stuck and frustrated with this part of the research. I haven't found the spin-up amps for the 6TB Reds, and I'm about to say screw it and just buy a 700w PSU. This is my noob way of begging someone to tell me what wattage I need ;)

Last, but most important of all, who knew building a box for the sole purpose of holding ones and zeroes would be so expensive :p Don't get me wrong, I'm excited to have a new toy but the Bank of FreeBug isn't.

ありがとうございます！

 

[Bidule0hm]

Look here (FYI found with wd red 6tb specs on google) ;)

 

[jgreco]

I'm going to slowly add drives over the next year or so until I reach a max of 8 in RAIDZ2. Concerning additional drives, can I add drives and update my RAID configuration easily or will I have to rebuild the array from scratch?

Unfortunately ZFS has no way to alter the width of a RAIDZ vdev.

I've been doing some light reading on the 80% figure you sited. Oracle mentions something called immutable files, but I'm not sure it's a definite solution. Kind of a bummer that I can only get 80% before performance suffers.

For what they probably mean by immutable files, you can go higher than 80%. The problem is that block allocation becomes more difficult as a pool becomes closer to full. For most fileserver applications, 80% is where things start to tank.

I've also done some reading on how to calculate my PSU wattage. Frankly, I'm stuck and frustrated with this part of the research. I haven't found the spin-up amps for the 6TB Reds, and I'm about to say screw it and just buy a 700w PSU. This is my noob way of begging someone to tell me what wattage I need ;)

The classic method is still useful. Assume 2A per 3.5" hard drive - it will be close enough to right.

https://forums.freenas.org/index.php?threads/so-you-want-some-hardware-suggestions.12276/

The technique in there is still viable. Note that most modern power supplies have fairly flat efficiency curves over the range 20-70% utilization. The X9 and E3 will consume about 30-40W idle, and 8 drives will consume between 50-80W idle, plus fans, etc., so on the low end you need to supply 80-120W, when busy probably 200W, and when spinning up an additional 200W, so 400 watts peak. But you don't want to order a 400W power supply, because that puts you right at the limit of the thing, and stressing a power supply is bad. A 500-600W supply seems approximately right.

Done that too many times, it's too easy ;-)

Last, but most important of all, who knew building a box for the sole purpose of holding ones and zeroes would be so expensive :p Don't get me wrong, I'm excited to have a new toy but the Bank of FreeBug isn't.

ありがとうございます！

We feel that pain. But loss of data is more expensive.