BUILD Yet another first time build, comments please

[annoyingnoob]

OK, I've had another external drive die (got my data off before it went completely, fortunately),
and it's time to go for a NAS.

I've been looking over brianmoses.net, pcpartpicker.com freenas box 2016, and the forums.freenas.org
hardware recommendations, and I think I've got most things, but I'd like some feedback from the
people here.

What I want (at least for now), is really an ultra-reliable external drive for long-term data storage.
No media server, just a big drive I can rely on. It will be doing absolutely nothing 99% of the
time, writing new data maybe 0.2% of the time, and reading current data 0.8% of the time. Really.
Processing of data will be done on local drives before being returned to the NAS. If the system was
off the other 99% of the time, I wouldn't notice or care.

As part of the ultra-reliable part, there will be (soon, not necessarily immediately) a 2nd copy
stored at another location. This makes me a bit more cost conscious.

I'm planning on a 6 x 4TB system in a raid-z2 configuration, which should be good enough. I'm not
(yet) paranoid enough for raid-z3.

What I'm planning on:

• Motherboard+CPU: I'm trying to decide between these two:

Asrock C2550D4I (built-in fanless Intel Atom board, quad core), ~$280
Asrock E3C226D2I + Intel G3260 3MB Haswell Dual-Core 3.3 GHz ~ $275​

About the same price, fanless might be nice but it's not a dealbreaker. Any suggestions?​

• RAM: Crucial 2 x 8GB ECC ~$90
• Case: Fractal Design R4 ~$95

While the Fractal Design Node 304 is popular, I'd rather have a larger case with better airflow.​

• Powersupply: SeaSonic S12G-550 ~$80
• Drives: 6 x WD 4TB Red Nas 6 x ~$150 = ~$900

Just about every thing I see talks about WD Red NAS drives, except for brianmoses.net. I'm perfectly happy with these, but wondering why Seagate is ignored?​

Is there any reason to think about other WD drives? Western Digital RE WD4000FYYZ datacenter drives can be found for $170, but they are 7200rpm drives and would run hotter. Would there be any significant advantage to them?​

One thing I haven't found (maybe not searching on the right terms) is behavior in power loss.
If the power suddenly goes out, what would happen to the NAS? Do I need a UPS, and if so how
do I have the UPS communicate to the NAS that it's time to shut down? In the event of power loss,
it's perfectly acceptable for the system to say off until manually restarted. Suggestions?

Thanks for all your help!

 

[Nick2253]

What I want (at least for now), is really an ultra-reliable external drive for long-term data storage.
No media server, just a big drive I can rely on. It will be doing absolutely nothing 99% of the
time, writing new data maybe 0.2% of the time, and reading current data 0.8% of the time. Really.
Processing of data will be done on local drives before being returned to the NAS. If the system was
off the other 99% of the time, I wouldn't notice or care.

Maybe I'm not understanding you correctly, but if you are looking for an "external drive", then FreeNAS (or any NAS) may not be what you want. If you're looking for the ability to directly connect your drive, then I'd suggest a USB or eSATA two disk enclosure with RAID1.

What I'm planning on:

• Motherboard+CPU: I'm trying to decide between these two:

Asrock C2550D4I (built-in fanless Intel Atom board, quad core), ~$280
Asrock E3C226D2I + Intel G3260 3MB Haswell Dual-Core 3.3 GHz ~ $275​

About the same price, fanless might be nice but it's not a dealbreaker. Any suggestions?​

• RAM: Crucial 2 x 8GB ECC ~$90
• Case: Fractal Design R4 ~$95

While the Fractal Design Node 304 is popular, I'd rather have a larger case with better airflow.​

• Powersupply: SeaSonic S12G-550 ~$80
• Drives: 6 x WD 4TB Red Nas 6 x ~$150 = ~$900

As a user of the E3C226D2I, I can tell you that I'm very happy with the board. The Atom setup is quite nice (and the ability to expand to 64GB of RAM is a huge plus), but if you may ever do anything more complicated on the server, you may find the Atom is outside of its league. The biggest thing I could think of is if you decided to encrypt your data, having the ability to grow to a Xeon will definitely be worth while.

Anecdotally, I have no sound problem with the stock heatsink from Intel. My case fans and hard drives are much louder.

Just about every thing I see talks about WD Red NAS drives, except for brianmoses.net. I'm perfectly happy with these, but wondering why Seagate is ignored?

Is there any reason to think about other WD drives? Western Digital RE WD4000FYYZ datacenter drives can be found for $170, but they are 7200rpm drives and would run hotter. Would there be any significant advantage to them?​

​

I am a user of the Seagate drives, and I have no problem with them. There were well reported and documented problems with some of Seagate's 2TB drives, and that, in my opinion, has been the big impetus towards WD drives in data-critical storage. There have, to the best of my knowledge, been no documented reliability difference between WD Reds and Seagate NAS drives 4TB+, but the stigma has stuck.

As far as enterprise drives are concerned, you won't likely see a benefit from the increased speed. 6x 5400RPM drives can easily saturate a 1Gbps connection. The main reason you might consider going with the higher RPM drives is if you have a need for faster random I/O (like VM storage), or are using a 10Gbps network.

One thing I haven't found (maybe not searching on the right terms) is behavior in power loss.
If the power suddenly goes out, what would happen to the NAS? Do I need a UPS, and if so how
do I have the UPS communicate to the NAS that it's time to shut down? In the event of power loss,
it's perfectly acceptable for the system to say off until manually restarted. Suggestions?

Power loss is bad, especially if we are talking critical data. If you are serious about your data's well being, a UPS should be considered mandatory. FreeNAS has built in support for UPS notification, if the UPS is supported by NUT. The FreeNAS documentation discusses that in more detail, and NUT has a list of supported UPSes. Your best bet is to search the forums, and find UPSes that users here have successfully used, since some common manufacturers/models don't work.

 

[annoyingnoob]

Maybe I'm not understanding you correctly, but if you are looking for an "external drive", then FreeNAS (or any NAS) may not be what you want. If you're looking for the ability to directly connect your drive, then I'd suggest a USB or eSATA two disk enclosure with RAID1.

Part of the problem is that the amount of data is getting pretty big. Right now, I've got at least 6TB of data, with more to come. That means I'd need mirrored 8TB drives. I'm currently using a pair of external 4TB drives, but splitting data across different drives (and thus filesystems) is unwieldy.

And, I'd feel better with the error detection / correction / rebuild properties of raid z2 instead of a simple mirror (if the mirrors differ, which one is correct?)

As a user of the E3C226D2I, I can tell you that I'm very happy with the board. The Atom setup is quite nice (and the ability to expand to 64GB of RAM is a huge plus), but if you may ever do anything more complicated on the server, you may find the Atom is outside of its league. The biggest thing I could think of is if you decided to encrypt your data, having the ability to grow to a Xeon will definitely be worth while.

Anecdotally, I have no sound problem with the stock heatsink from Intel. My case fans and hard drives are much louder.

OK, something to consider.

Power loss is bad, especially if we are talking critical data. If you are serious about your data's well being, a UPS should be considered mandatory. FreeNAS has built in support for UPS notification, if the UPS is supported by NUT. The FreeNAS documentation discusses that in more detail, and NUT has a list of supported UPSes. Your best bet is to search the forums, and find UPSes that users here have successfully used, since some common manufacturers/models don't work.

Sounds good. I'm searching, but haven't quite found what I need yet. The APC systems have USB and powerchute, but it's not clear if they are NUT compatible.

 

[joeschmuck]

(if the mirrors differ, which one is correct?)

The one that sumchecks properly.

What I want (at least for now), is really an ultra-reliable external drive for long-term data storage.

Yea, that confused me too. I guess you could consider a NAS a large external drive, in a way. The main thing is it's external, not very portable, or I wouldn't recommend it being used as portable.

The APC systems have USB and powerchute, but it's not clear if they are NUT compatible.

I have rarely seen an APC UPS with a USB port that doesn't support NUT. You need to size it properly for your environment so if you have frequent power outages then a larger one. If you rarely have power outages then one that will run at least 10 minutes will do fine. If you plan to turn it off most of the time and then on just to do backups, again, 10 minutes would be fine. I prefer a little overkill because I figure that if you buy decent hardware that it will last you longer. I have one APC UPS that I've had for almost 15 years and I've only replaced the batteries twice. It was a great investment. My signature line has the UPS I use but it's a 1000VA version but it's still nice and gets the job done.

And yes, you absolutely need an UPS if value your data. A worse case example is that you have your RAIDZ2 system with the six 4TB drives. Now you have a single failure and you replace the drive and it starts to resilver (estimated time is 14 hours, just ball parking it since it matters how much data you have on the drives). During the resilvering you have a second drive failure and honestly this type of stuff does happen, but thankfully it's rare. Now you cannot afford for the system to loose power and be shut down abruptly or you are more than likely to have data loss. It really is critical to have an UPS and if I could recommend the capability then I'd recommend a 1000VA each and every time, or larger. Mine will run for over 30 minutes depending on what's going on.

 

[annoyingnoob]

The one that sumchecks properly.

I'm a techie, but not a sysadmin. For a "simple" mirror, where is the checksum kept? The poster suggested using a USB/eSATA enclosure with RAID1, but does the hardware keep file integrity data, or must it be done externally?

Also, someone elsewhere has suggested using a ZFS mirror, which does have it's attractions.

Yea, that confused me too. I guess you could consider a NAS a large external drive, in a way. The main thing is it's external, not very portable, or I wouldn't recommend it being used as portable.

I don't consider my current external USB drives portable -- too worried about shock. With network access,
it would live under the desk and not move.

I have rarely seen an APC UPS with a USB port that doesn't support NUT. You need to size it properly for your environment so if you have frequent power outages then a larger one. If you rarely have power outages then one that will run at least 10 minutes will do fine. If you plan to turn it off most of the time and then on just to do backups, again, 10 minutes would be fine. I prefer a little overkill because I figure that if you buy decent hardware that it will last you longer. I have one APC UPS that I've had for almost 15 years and I've only replaced the batteries twice. It was a great investment. My signature line has the UPS I use but it's a 1000VA version but it's still nice and gets the job done.

And yes, you absolutely need an UPS if value your data. A worse case example is that you have your RAIDZ2 system with the six 4TB drives. Now you have a single failure and you replace the drive and it starts to resilver (estimated time is 14 hours, just ball parking it since it matters how much data you have on the drives). During the resilvering you have a second drive failure and honestly this type of stuff does happen, but thankfully it's rare. Now you cannot afford for the system to loose power and be shut down abruptly or you are more than likely to have data loss. It really is critical to have an UPS and if I could recommend the capability then I'd recommend a 1000VA each and every time, or larger. Mine will run for over 30 minutes depending on what's going on.

We rarely have power problems (less than once a year), but I've seen systems die when hit with a power loss. So, a UPS is a must. If I'm spending about $1400 or more (per system!) for data integrity, another $100 is cheap.

I'm torn about leaving the system running 24/7 (ease of access), and powering it up once a week for archiving. How long does it take to boot a FreeNAS system? Does FreeNAS allow power-down of drives when not accessed for a given period?

 

[Ericloewe]

For a "simple" mirror, where is the checksum kept?

All ZFS blocks are checksummed. No exceptions.

You should probably read cyberjock's guide (link in my sig).

 

[Stux]

Sounds like you're ready to step up to having a proper centralised data repository.

Build a nice FreeNAS box.

But how are you going to back it up?

 

[annoyingnoob]

Sounds like you're ready to step up to having a proper centralised data repository.

Build a nice FreeNAS box.

But how are you going to back it up?

With a second FreeNAS box at a different location.

 

[joeschmuck]

I'm torn about leaving the system running 24/7 (ease of access), and powering it up once a week for archiving. How long does it take to boot a FreeNAS system? Does FreeNAS allow power-down of drives when not accessed for a given period?

If you do enough planning you could have the drives sleep (spin down) when not in use but I'd recommend against it. If you are truly only plan to use the system once a week then you can just power it up and down however if you will be using it a few times a week, there are wear factors you should consider. Every time you power on a system there is a power surge and components can blow, although this is pretty rare. The main concerns are the hard drives failing and the primary cause there would be the drive motor or electronics powering the drive motor, and possible corruption of the USB flash boot drive, assuming you use a USB flash drive. These latter two items are very real. For the hard drives it is best to leave them spinning all the time because the bearings generally do not fail before something else fails. My drives have been spinning virtually constantly for over 3 years without a single issue. I power down my system to clean it out or do some hardware reconfiguration/upgrades but that is it. I expect at least 2 more years of life from these drives but at least I've got my moneys worth already.

 

[Nick2253]

possible corruption of the USB flash boot drive, assuming you use a USB flash drive. These latter two items are very real.

Just to add an anecdote, this happened to me last night. I was having some problems with AD integration not working, which caused me to reboot a number of times last night. Sure enough, suddenly my system stopped booting, and my USB drive is no longer recognized. It's been going strong with no errors for the last ~18 months, and now it's not working. Had I not needed to reboot my system, I'm certain my drive would still be OK.

Thank goodness for backups! :D

 

[Dice]

Thank goodness for backups!

And SSD's for boot device x)

 

[joeschmuck]

Just to add an anecdote

Thanks for adding this, it helps make a point. I hope you resolved your AD issues. Thankfully I don't need to use it at home.