Building a NAS in a mini-ITX

[ECWithana]

Hi,

I went through most of the posts and "so you want some hardware suggestions" sticky. But I hope the following is worth asking just to make sure.

I'm planning to build my FreeNAS based NAS with 4-6 drives in a mini ITX case (I already have WD red 3TB drives). I trying to stick with a $200 budget for motherboard, processor, RAM and PSU. I can go a bit further but I don't think I can spend $500 to go for expensive hardware.
Also, my best case would be to go for ZFS or RAID-10 but worst case, RAID-1 will be enough for me.

Hope someone could help me answer following questions.

1. was considering http://www.newegg.com/Product/Product.aspx?Item=N82E16813157367 as the motherboard simply because it has 4 SATA ports and PCIe 3.0 x16 slot for later expansion. Is that a good motherboard to pair with Intel Celeron G1610 Ivy Bridge 2.6GHz LGA 1155 55W Dual-Core ?

2. Or, I was also looking at Asus Intel H871-PLUS motherboard since it has 6 x 6.0 Gb/s slots. Will that be a better motherboard than #1 above for ZFS?

3. The motherboard mentioned in #2 above has hardware level RAID from intel and in this page Intel says linux kernel 2.6.27 supports hardware RAID with mdadm. What if I install CentOS or another stable linux distro with a kernel later than 2.6.27 and use hardware RAID? How is it compared to FreeNAS if I want to use RAID-10?

3. heard from a friend that I need at least 16G RAM and server level motherboards to go for ZFS, is that true (seems so to some extent in "so you want some hardware suggestions" page) ? But I read couple of times here that 8G ECC RAM is enough but didn't see recommendations for a motherboard in my price range.

4. Finally, really appreciate if I can get a good recommendation for a motherboard, processor, RAM and PSU for the budget and for the above given requirements. I'm fine with buying 8GB KVR16E11/8I as suggested in the "so you want some hardware suggestions" if its whats needed. But if ZFS is gonna be too expensive, I'm fine with settling with either RAID-10 or RAID-1 in that order.

Really appreciate your answers here.

 

[DJ9]

Neither one of the motherboards supports ECC memory. And at $200 total budget, your better off saving up a bit more first.

 

[cyberjock]

Its just as DJ9 said. Your budget is far too low. You need to wait until you have more cash or consider another solution. FreeNAS isn't meant to run on cheap hardware. It's an enterprise OS and as such has certain requirements that enterprises have no problem meeting. For home users, many will end up being left out due to hardware cost.

 

[underpickled]

Feel free to check out my build for a mini-ITX FreeNAS setup here: http://forums.freenas.org/threads/mini-itx-c226-haswell-build.15371/
If you're not doing media transcoding or full disk encryption, you can probably use a Pentium G3220. Unfortunately, ECC RAM and mobo + CPU for under $200 is just not really possible... honestly if you found that kind of price, I'd be wary of the component quality. Remember that you want to store a lot of data, reliably, for years. It's an investment.

 

[engmsf]

I can't comment on the ASRock H61MV-ITX LGA. However I have tested the Asus Intel H87I-PLUS and it works with FreeNAS 9.1. The recommendation on this forum is to use ECC RAM with FreeNAS and neither of these board support ECC.

Just remember that ZFS (RAID) is not backup. They guard against certain type of hardware failure. There are lots of failure modes that it doesn't guard against - fire, theft, human error and the list goes on.

 

[Richman]

How come I have heard of many on there using consumer MB's and non ECC memory and doing just fine? I am sure this person is not talking about running storage for a webserver or corporation with dozens or hundreds of clients logging on. Sounds like its a home solution for them.

 

[Michael Wulff Nielsen]

You heard about them because they were lucky. FreeNas (actually zfs) requires ecc memory to function reliably.

Your cheap consumer-hw might work well for years and suddenly blow up if some memory goes bad.

There are several threads on this forum where people have lost all their data due to memory corruption. Especially during scrubs/resilverings.

There are no recovery tools and nobody can help you if your pool dies because of bad memory.

But honestly if you wish to use the cheapest hw then use something with a non-check summing file system like ext3. Your data won't be safe, but you don't have the risk of mass suicide due to faulty memory.

 

[dalepres]

FreeNAS isn't meant to run on cheap hardware. It's an enterprise OS and as such has certain requirements that enterprises have no problem meeting.

That seems an odd statement. Enterprise NAS comes from IBM and other enterprise level hardware AND software with enterprise level support. I thought the whole idea of open source and free was to break the chains of big dollar corporate solutions. I work for a large enterprise. We would never consider using FreeNas; we spend millions of dollars per single NAS instance. I firmly believe that there's some middle ground between what we spend and FreeNas where we should be but those calling the shots for our enterprise require enterprise level support.

I think it would enhance FreeNas's presence if they were to support less expensive hardware that enables small and medium sized companies to get the benefits of a NAS solution.

Just curious, I see you're an admin here so I assume you have some significant personal ownership in the product. Is there a mission statement that I can read to better understand where FreeNas fits in the space?

 

[survive]

Hi dalepres,

Maybe a better way to think about FreeNAS is that its a free OS packed with enterprise features....sure you "break the chains of big dollar corporate solutions" but that doesn't disconnect you from the reality of what those enterprise-class feature need to run.

This middle ground you seek really only adds $150, maybe $200 to the cost of the system which gives support for ECC. You can put together a perfectly acceptable Supermicro system for $400.

Richman,

You hear about them when they say the system is working fine, then you hear from them later when their pool corrupts or doesn't mount. You don't generally here thes complaints form the guys who build their filers properly.

-Will

 

[Knowltey]

How come I have heard of many on there using consumer MB's and non ECC memory and doing just fine? I am sure this person is not talking about running storage for a webserver or corporation with dozens or hundreds of clients logging on. Sounds like its a home solution for them.

Doesn't matter if the server is going to be used in the home or in a business setting. The fact remains that ZFS and to an extent BSD were both designed with server grade hardware in mind. Running them on hardware that they were not designed to run on is simply asking for trouble and instability. With ZFS and non-ECC RAM it's as simple as a RAM bitflip causing a data corruption in the zpool's metadata section to not only trash your entire zpool but any backups that are created from it. And if you happen to run a scrub while you have bitflip you're also more likely than not going to corrupt a whole crapton of your data if not all of it.

 

[Richman]

This all being said, then I don't see that ZFS is what one would want to use if they are just building an At-Home NAS for personal data that is NOT mission critical like financial data or even web serving. I say this because, even though I haven't tested or analyzed or researched them, ReadyNAS, Drobo, HP's offering .... N41 or whatever the model is, or many of the other popular solutions for a home environment do not us ECC memory as far as I know or ZFS for that matter.

I also don't see the issue that a few posters here mention about having problems and a pool not mounting since RAID, ZFS or any other FS implemented, I have heard is said many times, RAID is NOT backup. In other words redundancy is not the same as backup. RAID CAN be your backup but a back up implies having more than one copy of your data which everyone should have. Then you just rebuild your NAS with your backup data.

Further, I did some research about a year ago or maybe two on bitflips. What I remember the 'take-away' from it was that while it can happen and does, it is not common as it may be realized on any particular harware about once every 7-10 years. If you happen have a bitflip then maybe it will corrupt all of your data on a ZFS with no ECC but on Non-ECC or a different FS it may mean the loss of a few pixels in a video file or image file. I have experienced lost pixles a few times in these files, possibly not from a bitflip but just from the system corrupting the data in other ways.

Now, here is where I backpeddle and recant what I said about not using ZFS because everyone hell-bent on enterprise hardware putting the fear of a bitflip corrupting ALL of your data in us , and the fact that everyone should have a backup and shouldn't be using their NAS as the sole copies of their data, it shouldn't matter if one uses ZFS on a consumer grade NAS.

I think maybe one of the main reasons to use enterprise hardware is that I have heard that building, re-building a ZFS pool is harder on hardware than other RAID implementations, or am I wrong?

 

[Knowltey]

Well yeah, FreeNAS may be overkill for what a home user needs, each NAS solution out there is different and has different needs. Just because some other independent NAS offering doesn't use ECC RAM doesn't make it any less needed for FreeNAS. The ECC is mainly because of how ZFS works and I don't believe those other solutions use ZFS.

With the bitflip it is just a game of chances. You could lose a video file if the flip causes the checksum on that file to fail, but if the checksum fails on pool metadata it'll corrupt that and then you're fucked. ZFS is wonderful for assuring data integrity when it can trust RAM but when that trust gets violated it jumps to the opposite end of he spectrum.

 

[Dusan]

You could lose a video file if the flip causes the checksum on that file to fail, but if the checksum fails on pool metadata it'll corrupt that and then you're fucked.

A single bit-flip will definitely not kill a pool. Every ZFS metadata block is stored in at least two copies (the more important in three copies and there are four copies of the vdev label). You will find lot of interesting information in this whitepaper: http://research.cs.wisc.edu/adsl/Publications/zfs-corruption-fast10.pdf

 

[engmsf]

Richman,

To me, ZFS or RAID is not backup. I have a separate backup storage and so should everyone. As such I don't consider ECC memory to be all that important and use consumer based hardware on the FreeNAS systems I have. To me it is no different than a motherboard or cpu failing, or even a fire or flood at home. I can't protect against everything and I can only do risk mitigation, if so I choose.

With that said, I plan to upgrade to a mini itx server board in the distant future.

 

[Knowltey]

A single bit-flip will definitely not kill a pool. Every ZFS metadata block is stored in at least two copies (the more important in three copies and there are four copies of the vdev label). You will find lot of interesting information in this whitepaper: http://research.cs.wisc.edu/adsl/Publications/zfs-corruption-fast10.pdf

Good to know, other threads I had read about ZFS and bitflips seemed to imply such was possible.

 

[Dusan]

Good to know, other threads I had read about ZFS and bitflips seemed to imply such was possible.

Ah, sorry, I did not read the entire thread. A single bit-flip on a no redundancy pool (no RAIDZ, nor mirror) should not kill the pool due to the metadata redundancy. However, I think it may be possible a for "lucky" memory bit-flip to kill a pool. But, that's probably true for any file system.

 

[Richman]

There are no recovery tools and nobody can help you if your pool dies because of bad memory.

But honestly if you wish to use the cheapest hw then use something with a non-check summing file system like ext3. Your data won't be safe, but you don't have the risk of mass suicide due to faulty memory.

Micheal made a valid point aboveabout recovery tools for ZFS pools. First of all, why not. why hasn't someone started coding recovery tools as I am sure that even with ECC, things can go wrong and thier may be a real need for it ..... or maybe I don't understand fully how ZFS operates. But then if you have a backup of your data, why would you need to recover it as apposed to use the backup to rebuild. Is recovery easier than starting your NAS over and rebuilding? I am getting the the idea that some are using NAS as their main data files.

This middle ground you seek really only adds $150, maybe $200 to the cost of the system which gives support for ECC. You can put together a perfectly acceptable Supermicro system for $400.
-Will

I see it as more than $150-200. Course I can use a spare MB laying around = $0, and spare RAM = $0 with just the const of the disks unless youahve spare disks laying around. I will highlight New vs Used and I am sure many will say, "buy new, you get a warranty and less likely the MB will pop a cap or something" So comparing apples to apples ........... I guess from what I figured , the $200 dif only hold true if comparing the purchase of used equipment.


Below are averages found.
If I bought Used:


Enterprise
SuperMicro or compatible MB = Min .... $160
CPU ................................................... $40
RAM 8GB ............................................$ 40 - 60
Controller card ....................................$75 - 100
Total .................................................. $315 - 360

Consumer
MB ...................................................... $30
CPU ..................................................... $15
RAM 8GB ............................................. $30
No controller card needed .................... $0
Total .................................................... $85-95

Difference ...........................................$230 - 265

If bought New


Enterprise
SuperMicro or compatible MB = Min .... $300
CPU ................................................... $230
RAM 8GB ............................................$ 105
Controller card ....................................$400-500
Total ...................................................$1035 - 1135

Consumer
MB ....................................................... $15 - 50
CPU .......................................................$60
RAM 8GB .............................................. $45-55
No controller card needed ..................... $0
Total .................................................... $120 - 165

Difference ......................................... $915 - 970

There appears to me to be at least 1,000 considerations, all of which have making my heard hurt for several months.

A single bit-flip will definitely not kill a pool. Every ZFS metadata block is stored in at least two copies (the more important in three copies and there are four copies of the vdev label). You will find lot of interesting information in this whitepaper: http://research.cs.wisc.edu/adsl/Publications/zfs-corruption-fast10.pdf

So I am assuming now that Knowltey and Michael Wulff are both dead wrong? This is just another thread that sends me on a full circle journey of leaning to enlightenment, to a measure of understanding and back to confused. We need a FreeNAS ans ZFS know-it-all to iron this out.

Good to know, other threads I had read about ZFS and bitflips seemed to imply such was possible.

Clarify .... you said other threads seamed to imply such was possible ........ Which? possible a bit flip could destroy a whole pool or that there are 3 or more copies of meta data that somehow protect this senario?

Richman,

To me, ZFS or RAID is not backup. I have a separate backup storage and so should everyone. As such I don't consider ECC memory to be all that important and use consumer based hardware on the FreeNAS systems I have. To me it is no different than a motherboard or cpu failing, or even a fire or flood at home. I can't protect against everything and I can only do risk mitigation, if so I choose.

With that said, I plan to upgrade to a mini itx server board in the distant future.

Good point as I was trying to make a similar point.
I find it impossible to use an ITX onless a controller card is used as not many have more than 4 SATA ports.

 

[Knowltey]

Clarify .... you said other threads seamed to imply such was possible ........ Which? possible a bit flip could destroy a whole pool or that there are 3 or more copies of meta data that somehow protect this senario?

That a possible bit flip could destroy a whole pool. There was an thread somewhere I read a while back that seemed to imply that and I also made a thread a while back as well asking for clarification on the matter and some of the people in there were saying it was indeed possible if the bitflip is unlucky enough to be in just the right place.

 

[jgreco]

Ah, sorry, I did not read the entire thread. A single bit-flip on a no redundancy pool (no RAIDZ, nor mirror) should not kill the pool due to the metadata redundancy. However, I think it may be possible a for "lucky" memory bit-flip to kill a pool. But, that's probably true for any file system.

Given a free hand to pick the bit, corruption of critical metadata could easily result in an allocated block being thought free; were that bit to represent a block that was holding critical metadata, one could convince the system to stomp on that metadata with a different valid block merely by writing file data until the block was allocated. Now you have a data block that is ALSO linked as a metadata block and is interpreted as such by part of the fs structure while ACTUALLY containing arbitrary file data. So if you get really unfortunate, now along comes a process reading the other directory or file whose metadata has been overwritten, ZFS reads the block without error, then panics the kernel on a supposedly impossible assertion.

 

[Knowltey]

So I am assuming now that Knowltey and Michael Wulff are both dead wrong? This is just another thread that sends me on a full circle journey of leaning to enlightenment, to a measure of understanding and back to confused. We need a FreeNAS ans ZFS know-it-all to iron this out.

Yeah, I keep getting answers regarding this possibility to both sides. Some saying that no flip can cause entire pool corruption with yet others saying that it can. The whitepaper that Dusan linked though seems to indicate that a bitflip can cause a system crash and possibly make the pool unmountable if the superblock in the page cache gets corrupted (at which point all is lost, so *technically* no it's not corrupting the entire pool, but the effect is still the same, you've lost all the data.) That is if I'm understanding the whitepaper correctly in "7 Beyond ZFS"

And @jgreco: http://i.imgur.com/TJstUAc.jpg