Citadel - Build Plan and Log

[ctag]

Hi,

First of all, thank you for dropping by! I know forums can be a chore for those sharing knowledge to complete noobs, but I really appreciate that this community exists.

I'm hoping to use this thread as a staging area to plan, receive feedback, and then carry out my first FreeNAS project. I'll try to respect your time and not get too off topic, but I do want to be thorough as well.

Current situation:

Personal computers:

• Desktop - 100G root, 500G home, 3.6T lvm-raid
• Laptop - 100G

Home servers:

• IRC box - 200G root (20G used)
• HTTP box - 150G root (10G used)
• NAS - 3.7T x-raid
• Security Camera - SD card root, streams video to desktop lvm-raid and internet

My home network is consumer stuff; but it is gigabit, and wired. The home servers are just repurposed old laptops.

The NAS is a sticking point. It's a Netgear ReadyNAS NV+ that I got a while back and installed 3x 2T WD Red disks in. It's just atrociously slow; enough so that I've given up using it to keep regular backups of the other computers and only store long-term documents/pictures/video on it.

The lvm-raid on my desktop is for stuff I'm not planning to keep forever, like the security footage. So it doesn't get bulletproof solutions. I honestly can't even remember what the scheme is for it: if it has any redundancy or is just a few disks striped together... Need to look that up.

FreeNAS plan:

I'd like to replace the ReadyNAS with a FreeNAS box, and possibly consolidate some of the other systems into it as well. So I'm looking for long term file storage, media streaming, maybe website hosting, and maybe small services like IRC. To be worthwhile, I wouldn't want to build it for less than 8 terabytes usable space, but I'm leaning more toward 16 so there's plenty of room to grow into. I'd like to spend less than $2500, but my budget philosophy is that I'd rather splurge and do it right the first time than have to build it twice, or worse lose my data.

Just because I feel like it, I name all of my computers after fictional spacecraft from movies and games. The ReadyNAS is the sole exception, because it's garbage. Anyway, my next NAS box is going to get the hostname citadel after the large space station from Mass Effect. So that's why the thread title is what it is.

Below is the list of hardware. It's very generic, and taken mostly from the hardware guide without any competency. But hopefully it's a starting point for figuring out overall system price and making changes.

• Chassis
  • Silverstone Technology CS380B Silverstone DIY ATX NAS Storage Case with Hot Swap Cases
  • https://www.amazon.com/SilverStone-Technology-Mini-ITX-Computer-DS380B/dp/B01MA6NJE8
  • $125
• Motherboard
  • Supermicro Motherboard MBD-X10SRL-F
  • https://www.amazon.com/Supermicro-Motherboard-MBD-X10SRL-F-B-E5-1600-LGA2011/dp/B00PS67III
  • $260
• CPU
  • Intel Xeon E5-1650
  • https://www.ebay.com/p/Intel-Xeon-E5-1650-3-2GHz-Six-Core-CM8062101102002-Processor/127385868
  • $100
• HDD
  • WD Red 4TB
  • https://smile.amazon.com/Red-4TB-Hard-Disk-Drive/dp/B00EHBERSE/ref=sr_1_1
  • $134
  • 8T drives cost a bit less per gig, but they just seem like overkill...?

If I do a single Vdev with RaidZ2, then I'd need 6 of those NAS drives to get 16T, which would be $804. So the total system cost is at $1289, and I still haven't figured out what to do about RAM, PSU, fans, etc.

FreeNAS conceptions:

I'm not sure if what I've picked up from reading around is accurate or up-to-date, so here's some of the stuff floating around in my head:

• Don't use more than 80% of the storage available, because ZFS will slow way down.
• Pools can be expanded with additional vdevs. Vdevs cannot be changed. (Exception: vdevs can be increased by replacing drives with larger capacity ones) (Exception: Can add additional disks to a mirror vdev.)
• Pools just wrangle vdevs. Vdevs are where all of the redundancy lives.
• RaidZ is on the way out because larger drives (4T+) take too long to resilver, and the risk of losing a second drive while that's going on is increased.
• All the RAM.
• Server grade is worthwhile, and doesn't necessarily cost way more than consumer.
• Keep HDD temps below 40 degrees C.

That's it for now, I'll keep reading and try to improve things on my own, but if you have any input for me it'd be greatly appreciated!

 

[LTCM]

FreeNAS conceptions:

I'm not sure if what I've picked up from reading around is accurate or up-to-date, so here's some of the stuff floating around in my head:

• Pools can be expanded with additional vdevs. Vdevs cannot be changed.

Not 100% true. You can increase the size of a vdev by replacing all the drives with larger variants. A vdev of 4tb drives could be expanded in capacity by swapping in 8tb (or 10tb, 12tb, etc)drives over time. You only get the benefit once everything is change over, however.

 

[ctag]

Not 100% true. You can increase the size of a vdev by replacing all the drives with larger variants. A vdev of 4tb drives could be expanded in capacity by swapping in 8tb (or 10tb, 12tb, etc)drives over time. You only get the benefit once everything is change over, however.

Thank you for the clarification!

 

[Chris Moore]

Are you willing to consider second hand server equipment?

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[ctag]

Are you willing to consider second hand server equipment?

Sent from my SAMSUNG-SGH-I537 using Tapatalk

Yes, if that's something which is considered pretty safe to do. But I don't know where to go looking for second hand parts.

Searching online brings up eBay and a bunch of specialty outlets.

 

[Jailer]

Looks like a good choice of hardware. Only thing I might question is the cooling of the drives in that case by the way it's set up. But never having used one I can't say for sure how it would do, just looks like it wouldn't cool well.

 

[ctag]

Looks like a good choice of hardware. Only thing I might question is the cooling of the drives in that case by the way it's set up. But never having used one I can't say for sure how it would do, just looks like it wouldn't cool well.

Hrm, yeah I have no idea what to expect from that chassis's airflow

40 degrees C is the typical max recommended for hard drives, right?

 

[danb35]

Pools can be expanded with additional vdevs. Vdevs cannot be changed. (Exception: vdevs can be increased by replacing drives with larger capacity ones)

The other exception to this is that you can add disks as mirrors. So, if you have a single-disk vdev, you can attach a second disk as a mirror. If you have a two-disk mirror, you can attach a third disk and make it a three-way mirror. I'm not aware of any limitation on the number of disks you can do this with, though making more than a three-disk mirror is probably silly.

40 degrees C is the typical max recommended for hard drives, right?

Correct.

 

[ctag]

The other exception to this is that you can add disks as mirrors. So, if you have a single-disk vdev, you can attach a second disk as a mirror. If you have a two-disk mirror, you can attach a third disk and make it a three-way mirror. I'm not aware of any limitation on the number of disks you can do this with, though making more than a three-disk mirror is probably silly.

Correct.

Cool, I updated my little list of concepts.

I don't know how to address temperature issues other than to keep an eye on the situation and add/change fans if things are bad. If there are well-recommended desktop style chassis I'd gladly consider it over what I currently have listed. I'll look around and see if there's something better than just what Amazon dregs up.

 

[BigDave]

I'd like to replace the ReadyNAS with a FreeNAS box, and possibly consolidate some of the other systems into it as well. So I'm looking for long term file storage, media streaming, maybe website hosting, and maybe small services like IRC. To be worthwhile, I wouldn't want to build it for less than 8 terabytes usable space, but I'm leaning more toward 16 so there's plenty of room to grow into. I'd like to spend less than $2500, but my budget philosophy is that I'd rather splurge and do it right the first time than have to build it twice, or worse lose my data

In the case you picked out above (8 drive capacity) your goal of 16TB of usable space suggests the use of seven drives with 4TB capacity giving you a usable capacity in RAIDz2 of 15.74TB.
At the price you linked to in the OP, thats $938. Taking that cost off the top of your budget leaving $1,562.
This will certainly work (as others have already stated).
My suggestion is to begin with a smaller capacity pool and spend a larger portion of your budget on the motherboard, cpu and memory buying more hard drives later as your needs dictate.

 

[Chris Moore]

Here is an idea, if you were interested in purchasing something that would just work straight away. This is a workstation but it has the Xeon CPU and ECC memory you need. I use one of these as a development server at work and they are quiet and reliable. All you would need to do is add a good IT mode SAS controller and hard drives, you could buy something like this:
https://www.ebay.com/itm/Dell-Precision-T7500-Workstation-6-Core-2-40GHz-E5645-24GB-RAM/171961119957
and
https://www.ebay.com/itm/Dell-H310-...0-IT-Mode-for-ZFS-FreeNAS-unRAID/162834659601
You can put hard drives in the four front 5.25 bays and it has four internal 3.5 bays, so you can easily put 8 hard drives in it and it can be upgraded all the way to 196 GB of RAM if you add the second CPU. Only half that with only one CPU.

 

[ctag]

In the case you picked out above (8 drive capacity) your goal of 16TB of usable space suggests the use of seven drives with 4TB capacity giving you a usable capacity in RAIDz2 of 15.74TB.
At the price you linked to in the OP, thats $938. Taking that cost off the top of your budget leaving $1,562.
This will certainly work (as others have already stated).
My suggestion is to begin with a smaller capacity pool and spend a larger portion of your budget on the motherboard, cpu and memory buying more hard drives later as your needs dictate.

Thanks @BigDave.

The idea of expanding later on makes me anxious, the way I see it I'd want the two drives worth of redundancy spread out over as much of the pool as possible. E.G. if I have two 7-drive vdevs with RAIDz2, then I'm using four disks for redundancy, but losing just two disks can take the pool out. I'm not sure if that's accurate or even makes sense, but from the little reading I've done so far, most system expansions seem to be creating additional vdevs.

Here is an idea, if you were interested in purchasing something that would just work straight away. This is a workstation but it has the Xeon CPU and ECC memory you need. I use one of these as a development server at work and they are quiet and reliable. All you would need to do is add a good IT mode SAS controller and hard drives, you could buy something like this:
https://www.ebay.com/itm/Dell-Precision-T7500-Workstation-6-Core-2-40GHz-E5645-24GB-RAM/171961119957
and
https://www.ebay.com/itm/Dell-H310-...0-IT-Mode-for-ZFS-FreeNAS-unRAID/162834659601
You can put hard drives in the four front 5.25 bays and it has four internal 3.5 bays, so you can easily put 8 hard drives in it and it can be upgraded all the way to 196 GB of RAM if you add the second CPU. Only half that with only one CPU.

I actually like this idea a lot. Having something pre-built would mean I could focus more on getting used to FreeNAS administration without worrying that I bought and assembled everything correctly.. And it's a lot less expensive.

I understand very little of what's going on with the re-flashed SAS controller, but it looks like it'd need two cables like this to connect the drives?

 

[danb35]

if I have two 7-drive vdevs with RAIDz2, then I'm using four disks for redundancy, but losing just two disks can take the pool out.

No, you'd need to lose at least three disks (all in the same vdev) to take the pool out, and you could potentially survive losing up to four disks (two each in the two vdevs) without data loss.

 

[LTCM]

Thanks @BigDave.

The idea of expanding later on makes me anxious, the way I see it I'd want the two drives worth of redundancy spread out over as much of the pool as possible. E.G. if I have two 7-drive vdevs with RAIDz2, then I'm using four disks for redundancy, but losing just two disks can take the pool out.

Losing two disks in a raidz2 vdev leaves your data at a high risk of loss but it's not gone yet. If you replace the damaged disks you can save the entire array.

Edit: danb35 beat me by a min. My fingers arent as fast as they used to be.

 

[Chris Moore]

I'm not sure why you're talking about a 7 drive vdev. If you're going to go with that chassis I pointed out, I would go with 8 drives.
That system has a couple of SATA connections on-board.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Chris Moore]

Thanks @BigDave.

The idea of expanding later on makes me anxious, the way I see it I'd want the two drives worth of redundancy spread out over as much of the pool as possible. E.G. if I have two 7-drive vdevs with RAIDz2, then I'm using four disks for redundancy, but losing just two disks can take the pool out. I'm not sure if that's accurate or even makes sense, but from the little reading I've done so far, most system expansions seem to be creating additional vdevs.


I actually like this idea a lot. Having something pre-built would mean I could focus more on getting used to FreeNAS administration without worrying that I bought and assembled everything correctly.. And it's a lot less expensive.

I understand very little of what's going on with the re-flashed SAS controller, but it looks like it'd need two cables like this to connect the drives?

The SAS controller I pointed out is already flashed to IT mode, but it's pretty simple.

Edit: This is the kind of bracket I would use to mount the HDDs in the front.

https://www.ebay.com/itm/EverCool-D...o-Triple-3-5-in-HDD-Cooling-Box-/253129580230

 

[BigDave]

I'm not sure why you're talking about a 7 drive vdev. If you're going to go with that chassis I pointed out, I would go with 8 drives.
That system has a couple of SATA connections on-board.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

My thinking is about having a spare bay for burning in a replacement drive
for a single VDev pool resilver. If I have a two VDev pool, I would want two
extra bays (i.e. 16bay chassis with two 7 drive VDevs).
If you have another machine in which to carry out the burn-in process, then
this is a non-issue of course. I'd rather have it and not need it, than to need it and not have it.

 

[Chris Moore]

My thinking is about having a spare bay for burning in a replacement drive
for a single VDev pool resilver. If I have a two VDev pool, I would want two
extra bays (i.e. 16bay chassis with two 7 drive VDevs).
If you have another machine in which to carry out the burn-in process, then
this is a non-issue of course. I'd rather have it and not need it, than to need it and not have it.

I see... I have another system that I use for burning the drives in and to wipe them before disposal.
I do have open drive bays in the servers in case I need it for something, but I never thought of using the active server to burn-in new drives.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[ctag]

No, you'd need to lose at least three disks (all in the same vdev) to take the pool out, and you could potentially survive losing up to four disks (two each in the two vdevs) without data loss.

Losing two disks in a raidz2 vdev leaves your data at a high risk of loss but it's not gone yet. If you replace the damaged disks you can save the entire array.

Edit: danb35 beat me by a min. My fingers arent as fast as they used to be.

Oh right, whoops.

I guess what I'm trying to get at though is that having two RAIDz2 vdevs still leaves 2 disks worth of redundancy in the worst case (disks failing from the same vdev), but it uses four disks for redundancy between the two vdevs. It feels wasteful somehow.

The SAS controller I pointed out is already flashed to IT mode, but it's pretty simple.

Edit: This is the kind of bracket I would use to mount the HDDs in the front.

https://m.ebay.com/itm/EverCool-Dua...5-in-HDD-Cooling-Box-/253129580230?nav=SEARCH

Thanks. I think I'm just going to go ahead and pick up that used server and see how it does, and save building my own as an exercise for next time.

My thinking is about having a spare bay for burning in a replacement drive
for a single VDev pool resilver. If I have a two VDev pool, I would want two
extra bays (i.e. 16bay chassis with two 7 drive VDevs).
If you have another machine in which to carry out the burn-in process, then
this is a non-issue of course. I'd rather have it and not need it, than to need it and not have it.

I have a few computers around that could be used for burning in a drive.. But I like the idea of having a bay empty in the NAS to do it.

 

[Chris Moore]

Oh right, whoops.

I guess what I'm trying to get at though is that having two RAIDz2 vdevs still leaves 2 disks worth of redundancy in the worst case (disks failing from the same vdev), but it uses four disks for redundancy between the two vdevs. It feels wasteful somehow.

Two disks for redundancy in each vdev. The two in vdev-0 only serve vdev-0 and likewise for the two in vdev-1. I use this method with 6 drives in each vdev and two vdevs. It gives me 12 drives with 4 dedicated for redundancy and as long as you don't have three drives catastrophically fail in the same vdev at the same time, you can't lose any data. Also, due to the checksum data, you have to get a pretty serious error before you loose any data to a drive fault. I have not lost any data and I have replaced every drive in both of my home servers at least twice. You could make the vdev of 8 drives with 2 drives for redundancy but the more drives in the vdev, the more time it takes to resilver when you do need to replace a drive.