Building a NAS in a mini-ITX

Status
Not open for further replies.

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I think a single bit-flip could potentially trash your metadata despite there being 2 copies. If its a situation of "copy bit XYZ from RAM to disk 1 sector 1000 and copy bit XYZ from RAM to disk 1 sector 2000" then you just successfully paid full price for a bit flip despite there being 2 alleged backup copies. It stupid situations like this that make non-ECC RAM very undesirable.

For the record, I didn't read that link(not at home right now). But I definitely don't see them creating XYZ in separate code and then writing each copy to 2 different places. At that moment you've immediately created a need for double the RAM. And why deal with situations like that when ZFS was expecting ECC RAM(so doing 2 copies is stupid).

So either you are stupid for ignoring your assumptions or you are stupid for doubling the amount of RAM. Which flavor of stupid do you want to be?
 

Knowltey

Patron
Joined
Jul 21, 2013
Messages
430
So either you are stupid for ignoring your assumptions or you are stupid for doubling the amount of RAM. Which flavor of stupid do you want to be?

Well my assumption is that it is indeed possible and I'm not ignoring it. I'm just curious as to the actual mechanics involved of doing so I can answer people better in the future regarding the neccessity of ECC RAM. The whitepaper in question that Dusan links though seems to show at least one specific instance of memory bitflip that results in an unmountable pool.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Well my assumption is that it is indeed possible and I'm not ignoring it. I'm just curious as to the actual mechanics involved of doing so I can answer people better in the future regarding the neccessity of ECC RAM. The whitepaper in question that Dusan links though seems to show at least one specific instance of memory bitflip that results in an unmountable pool.

A single bitflip by itself is unlikely to result in an immediate problem. However, the reality is that if your FAT filesystem or UFS filesystem metadata develops an issue, the possibilities in the disk layout are limited and it is relatively easy to identify and even correct many errors, a process called chkdsk or fsck that anyone here should be familiar with. These often correct small errors without fuss, right? ZFS has nothing like this because the theory is that the pool is always correct and protected against errors through other mechanisms.

Once RAM-originated bit errors are introduced into a pool, it is left to chance and fate to correct them. Maybe it was file data. Maybe it was metadata for a short-lived file. If so ... no permanent damage to the pool. But since when has fate been relied on to be kind?
 

Richman

Patron
Joined
Dec 12, 2013
Messages
233
What happens in ECC if the data being checksumed is wrong or the checksum itself has a bitflip? I don't know if its possible .....just asking?
A single bit-flip will definitely not kill a pool. Every ZFS metadata block is stored in at least two copies (the more important in three copies and there are four copies of the vdev label). You will find lot of interesting information in this whitepaper:http://research.cs.wisc.edu/adsl/Publications/zfs-corruption-fast10.pdf
I read this article too and while I don't think anyone is refusing to acknowledge the possibility of it happening, and it looks like this in more than half the cases will probably cause a system crash and maybe it won't be able to come back up or be recovered. I can see in an Enterprise situation this could be a major problem even if you have the data backed up on another NAS RAID elsewhere. As for myself and I believe the OP, we are talking about personal files at home..... are we not? I may be not expecting anymore error correction or error safety than the disk in my personal desktop or my WD MyBook DAS.
So the question really is: Is it less safe than your typical HDD in a desktop or DAS attached to it or the same or slightly better?

To me, ZFS or RAID is not backup. I have a separate backup storage and so should everyone. As such I don't consider ECC memory to be all that important and use consumer based hardware on the FreeNAS systems I have. To me it is no different than a motherboard or cpu failing, or even a fire or flood at home. I can't protect against everything and I can only do risk mitigation, if so I choose.

With that said, I plan to upgrade to a mini itx server board in the distant future.
Exactly my point. I have 4.2 TB of data and none of it at the moment is protected against bitflips. So, big bitflippin la-di-da. Some of that data is backed up on another disk .... here at home. I have not yet implemented the 'take a disk to work or over to mom's house' scenario in case my house burns down, which is totally possible with an old house with 60 amp service and constantly finding circuits overheating ( and it scares the crap out of me) or threat of moletof cocktails being lobbed my house which is also totally possible. I don't run any manual check-sums to see if some of it on one copy or the other became corrupted. I think my plan was closer to what engmsf was doing and I think that cyber and some of the other are so geared to what FreeNAS is designed and intended for and geared to Enterprise Networks and maybe they work installing them in the industry, is that they forget that a HOME NAS solution possibly doesn't need as much protection as humanly or technologically possible at this point in time unless you can just afford the few hundred bucks additional. I am sure they are just wanting to warn us against the possible catastrophe because many that implement do not and think that redundancy is perfect backup. Sure it makes complete sense to use ZFS if you using FreeNAS and ECC if your using ZFS and I would love to do it myself and one day I will. Not sure if my 1st will be though. I am just looking for a central place to store and access files on the network and then I will back that up in case something catastrophic happens, like a flippin bitflip.
 

Dusan

Guru
Joined
Jan 29, 2013
Messages
1,165
What happens in ECC if the data being checksumed is wrong or the checksum itself has a bitflip
If there's one bitflip in the 72bit word (64 bits data, 8 bits ECC) it will correct it and the BIOS will log the event in the system event log. If more bits get flipped it will log the event and halt the system.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Read the ecc sticky. That should answer the question for you.

Sent from my DROID BIONIC using Tapatalk
 

Richman

Patron
Joined
Dec 12, 2013
Messages
233
Read the ecc sticky. That should answer the question for you.

Sent from my DROID BIONIC using Tapatalk

OK hmmmm. I googled Sticky because I have heard the term several times on other forums but never really understood it except to assume it was a link in someones signature line. The only thing on google specific was in an Ubuntu forum where someone asked the same question and the reply was:

Re: What is a sticky thread?
Stickies are stuck at the top of the forum pages - information that admin deem important.

So, since I don't see anything at the top or this page, your sig, or the forum HOME page, with the risk of getting flamed, WHAT is a STICKY and where do I find them? Anyone wanting to flame me can just 'sticky it' .... where is dark and gloomy
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
How dare you criticize the almighty sticky! Just kidding. If you look at the sections of the forum some at the top have a red push pin. Those are important topics....and stickied by the forum admins.

Sent from my DROID BIONIC using Tapatalk
 

Michael Wulff Nielsen

Contributor
Joined
Oct 3, 2013
Messages
182
I'm starting to think that many seriously underestimate what corrupt memory can do. Normally when memory goes bad it's not just 1 bit, it's several and most of the time in several places.

But it is a fact that people have lost their zfs pools due to faulty memory.

Also many of us who use FreeNas take our data very seriously and want it to be safe. If we didn't, why bother with freenas? Why not just buy a synology, qnap or other nas-box?

So consumer-grade hw might work, then again it might not, and you get no warning before the stuff hits the fan.
 

engmsf

Dabbler
Joined
May 26, 2013
Messages
41
So consumer-grade hw might work, then again it might not, and you get no warning before the stuff hits the fan.


Consumer grade hardware works with ZFS and there is no doubt about this. ECC memory is not a requirement, it is a highly recommended.

If ECC was a hard requirement (ie you can't even run the system), I would imagine the usage and adoption of ZFS would decline dramatically.
 

BrandonS

Dabbler
Joined
Dec 3, 2013
Messages
13
the_tox's thread has good builds if you want to use ECC memory. If you want more specific detail on the ASUS because you aren't going for ECC support, checkhttp://forums.freenas.org/threads/is-asus-h87i-plus-motherboard-working.15335/. You will see my experience of returning that board to get the C226. Additionally, I have a Pentium G3220 sitting here NIB :) As mentioned the Celeron listed doesn't support AES-NI so you wouldn't want to use encryption with that probably.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Consumer grade hardware works with ZFS and there is no doubt about this. ECC memory is not a requirement, it is a highly recommended.

If ECC was a hard requirement (ie you can't even run the system), I would imagine the usage and adoption of ZFS would decline dramatically.

There's been talk of a hard requirement. The catch, there is no sure-fire way to determine if you are using ECC RAM. Besides, we take the stance of "if you want to be dumb enough to use non-ECC.. well, you'll get what you deserve if things go bad". We don't do sympathy for that anymore and threads are typically ignored on sight by the experienced people. We have better things to do as volunteers than recommend ECC and after you ignore us and come crawling back ask us to not ignore you.
 

Richman

Patron
Joined
Dec 12, 2013
Messages
233
OK, for future reference, do you 'experienced people' ignore threads only based on issues where memory wend bad or do you blanket ignore any thread where anything less than enterprise hardware is mentioned as being used?

Ahhhhhh Stickies ........ I feel all warm and gooey now. Thanks for the sticky enlightenment. Don't know how I missed them. I am glad they aren't snakes.
 

Knowltey

Patron
Joined
Jul 21, 2013
Messages
430
If it is someone that we've already told them that they need to use ECC memory and they are now having issues because they ignored what they were told about minimum system requirements they are likely to be ignored, if they can't take advice regarding the basics why should we think study are capable of taking advice regarding more advanced stuff?

Anyhow, what is the point of making a NAS and selecting the OS focused on data integrity and then cutting out the hardware that allows that feature to operate properly? If you want a NAS but don't care about data integrity then use an OS that fits your needs.

It's like buying a car, if you only need to drive back and forth to the office each day and never need to go off-roading you don't go buy a Hummer H1. If all you need is a simple media file server for home use and don't care about data integrity then why are you buying the NAS designed around data integrity?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Actually it is a combination of getting burnt out doing someone else's homework for them, and being tired of seeing people convinced that the advice wasn't meant for them, etc.

Personally I tire of the use of the word "enterprise" said in such a way that it makes it sound like a euphemism for gold-plated... when in fact decent server boards typically cost less than prosumer grade gear.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
OK, for future reference, do you 'experienced people' ignore threads only based on issues where memory wend bad or do you blanket ignore any thread where anything less than enterprise hardware is mentioned as being used?

Ahhhhhh Stickies ........ I feel all warm and gooey now. Thanks for the sticky enlightenment. Don't know how I missed them. I am glad they aren't snakes.

Just to clarify, we "experienced people" don't have meetings in dark rooms where we decide who to ignore and when to answer and not answer. It's more of a situation where we all complain about the same stuff in emails to each other. That being said, its pretty easy to see what is ignored because if I want to ignore it there's a good chance everyone else will too. Been around these guys enough to see what they do and don't feel like answering to since we get the same darn questions every week(sometimes twice a day).

That being said, if you do stupid things that resulted in trashing your pool and we know there's little or no chance of recovery, we aren't likely to spend any time on you. Bad RAM with non-ECC is an excellent example. We'll tell you to run a RAM test and when it fails we won't go further. It wasn't our data, we don't have a commitment to fix your data(we are volunteers) and there's virtually no chance of success anyway. If you wanted to pay us you might get more support. But considering the time that someone will have to be invested in recovery, you probably aren't willing to drop $1000 to find out if you can get your data back as you couldn't be bothered with going with ECC RAM which was probably cheaper than the cost of data recovery. It's nothing more than having car insurance, and we all know that its smarter to have car insurance just from a financial perspective.

If your hardware doesn't match our recommendations and the problem doesn't appear to be related to your poor choice of hardware then we'll usually help you. But we also have limits. You're kind of off on a bad foot the second you tell us you are using hardware from 2005 with 2GB of RAM and ZFS. I'll often tell you to upgrade and ignore you until you actually have appropriate hardware. Won't take much for us to write you off and ignore you though. If you didn't care enough to spend money to protect your data why should we care enough to spend several hours trying to fix your data? You made your choice that your data wasn't worth a few hundred dollars to get good parts, so why would you pay a few thousand to get it back later?

In essence, you'll get out of us what you put into protecting your data. If you showed that you didn't care, we won't either. And if you want to have an attitude with us over it, feel free to. But you didn't have an attitude with yourself when you got stupid and did everything we told you not to do. And Ill basically put you on my "sh*t" list and ignore you, which won't do you any favors even if you go back and do everything right after losing your data.

I won't speak for everyone else here, but I will say this. I am a volunteer. I spend at least 6 hours a day, every day, on this forum and countless hours every day on IRC. I'm here in the mornings. I'm here in the evenings. I'm often here even in the middle of thenight. And if I don't want to answer I won't. I'm tired of certain questions(plugin updates, transmission problems, permission problems, crappy hardware choices and now painted in a corner, bad RAM, no UPS, etc). I'll close my browser tab and won't feel guilty about it. We've got stickies for most of that stuff. There's Google, the FreeNAS manual and learning for yourself for the rest. I don't get paid to teach you a new operating system. Just like if I told you to teach me how to setup a windows server in all aspects you wouldn't do it for free, so why should you expect the same from me. You could spend weeks teaching me how to use Windows Server and use all of the features it offers. There are schools that teach people how to use Windows and charge $20k+ for it over a course of months. If you want to pay that kind of money, I'd put together a lecture and lessons for you on FreeNAS. But you and I both know you aren't about to pay me that kind of money as you could easily pay someone to build your FreeNAS server for less. Also, someone is already paid to maintain the FreeNAS documentation. Can't beat free documentation, but you can't complain if it doesn't teach you every aspect of using the OS either. Microsoft's manual that comes with Windows doesn't tell you how to do everything either. So when people complain about poor documentation, they can eat a bag of whatever fecal matter they can find because they don't complain to Microsoft over the same situation. In fact, most people paid for MCSE certifications(and maybe even training) but aren't willing to pay for FreeNAS training and knowledge.
 

Richman

Patron
Joined
Dec 12, 2013
Messages
233
Hi,

2. Or, I was also looking at Asus Intel H871-PLUS motherboard since it has 6 x 6.0 Gb/s slots. Will that be a better motherboard than #1 above for ZFS?

I wouldn't get caught up on 6 Gb/s SATA ports as has been said many time on other threads that mathematically you can not saturate a Gb LAN with 3Gb/s SATA ports. You only consideration for 6Gb/s would be local I/O if you were gong to have more than one array and were planning on transferring a lot of data back and forth between disks or you were going to have fiber channel network which nobody utilizes at home or in a small-medium office networks anyway.

3. heard from a friend that I need at least 16G RAM and server level motherboards to go for ZFS, is that true (seems so to some extent in "so you want some hardware suggestions" page) ? But I read couple of times here that 8G ECC RAM is enough but didn't see recommendations for a motherboard in my price range.

The FreeNAS hardware wiki is recommending 8GB and if you haven't seen it it would be a good read. But then as I find it depends on what you plan to do with your NAS and how you use it and what type of data, FS and so on.
 

Richman

Patron
Joined
Dec 12, 2013
Messages
233
If it is someone that we've already told them that they need to use ECC memory and they are now having issues because they ignored what they were told about minimum system requirements they are likely to be ignored, if they can't take advice regarding the basics why should we think study are capable of taking advice regarding more advanced stuff?

Anyhow, what is the point of making a NAS and selecting the OS focused on data integrity and then cutting out the hardware that allows that feature to operate properly? If you want a NAS but don't care about data integrity then use an OS that fits your needs.

It's like buying a car, if you only need to drive back and forth to the office each day and never need to go off-roading you don't go buy a Hummer H1. If all you need is a simple media file server for home use and don't care about data integrity then why are you buying the NAS designed around data integrity?

Because .......... some of us are caught in the middle and plan to upgrade hardware in the near future and starting with FreeNAS allows us to migrate out RAID config file to the new hardware when we get it and are not willing to wait another 8 more months till we get another $3-400 of hardware.

Its like buying a Hummer H1 when you know you plan to go off road and will need it but can't afford to buy two vehicles and the Honda Accord you can afford now will not provide the needed off-road capability when you go that route. I guess you never thought of this scenario?

Actually it is a combination of getting burnt out doing someone else's homework for them, and being tired of seeing people convinced that the advice wasn't meant for them, etc.

I think that is tiring for all of us

Personally I tire of the use of the word "enterprise" said in such a way that it makes it sound like a euphemism for gold-plated... when in fact decent server boards typically cost less than prosumer grade gear.

Really? Find me a descent for server board for less than $50 and I will buy one. You must have ome connections I havn't run across yet and I am constantly looking.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
I wouldn't get caught up on 6 Gb/s SATA ports as has been said many time on other threads that mathematically you can not saturate a Gb LAN with 3Gb/s SATA ports. You only consideration for 6Gb/s would be local I/O if you were gong to have more than one array and were planning on transferring a lot of data back and forth between disks or you were going to have fiber channel network which nobody utilizes at home or in a small-medium office networks anyway.

Put more bluntly, there's a simpler math argument: A typical hard drive only manages to transfer data at around 1.5Gbps, fast ones around 2Gbps. Only SSD's generally exceed 3Gbps. This is a hard limit. There is little point in a 6Gbps port wasted on a 1.5Gbps disk.

But you're also limited by the speed of the network connection for many operations, too.

3. heard from a friend that I need at least 16G RAM and server level motherboards to go for ZFS, is that true (seems so to some extent in "so you want some hardware suggestions" page) ? But I read couple of times here that 8G ECC RAM is enough but didn't see recommendations for a motherboard in my price range.

8GB is sufficent for a modest pool. As for a board in your "price range", that isn't really a good way to go about it. You should find what you need, then figure out how to acquire it. We typically find that there are nice prosumer boards in the $200-$300 range that may offer most of the desirable features, but that the low end $50 ones typically have Realtek ethernets and other severe limitations. And all those boards are designed for desktops, gaming, or other uses that are only somewhat compatible with the featureset you desire. The real secret is that a quality server motherboard for a whitebox server is actually pretty cheap (often ~$150), has EVERY feature that is desirable, including some like remote management that even the prosumer boards lack, and so are ideal candidates. Throwing around the words "server level" as though they're the most expensive option is a bit of a distraction. They're actually the best deal, especially if you can get a nice one with integrated controllers of the types you happen to need...
 
Status
Not open for further replies.
Top