Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.
Hardware Recommendations Guide

Hardware Recommendations Guide Discussion Thread Rev 1e) 2017-05-06

kdragon75

FreeNAS Expert
Joined
Aug 7, 2016
Messages
2,429
Last edited:

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,076
I understand it's still recommended but do we all agree that there is no MORE danger in using normal RMA with ZFS than any other file system like NTFS or ext4?
Probably, but that's a dangerous attitude to emphasize, because it implies "Oh, I've lived without ECC all these years, why should I bother now...". It's better to emphasize "You want ECC because the added cost is small and your data is significantly safer".
 

Chris Moore

Super Moderator
Moderator
Joined
May 2, 2015
Messages
9,552
Did we finally quash the "bad memory will cause a scrub of death" rumor?
FreeNAS.org Hardware recommendations
Will ZFS and non-ECC-RAM kill your data?

I understand it's still recommended but do we all agree that there is no MORE danger in using normal RMA with ZFS than any other file system like NTFS or ext4?
I did a comparison (using prices on eBay) and the cost of 32GB of non-ECC memory was a whopping $7.60 less than 32GB of ECC memory (Using the same brand and speed for fairness), so tell me again why you are worried about spending extra for some level of assurance that the system will be more reliable?

Reference:
Crucial kit of 32GB - non-ECC memory - $304.00:
https://www.ebay.com/itm/Crucial-32GB-4x8GB-PC3-12800-DDR3-1600-240-PIN-DIMM-Memory-NON-ECC-CL11-Module/232594687993
Crucial kit of 32GB - ECC memory - $311.60:
https://www.ebay.com/itm/Crucial-32GB-4x8GB-PC3L-12800E-DDR3L-1600-240pin-DIMM-ECC-Unbufferted-RAM-1-35V/232579056047

Note, if you are willing to buy Kingston, this kit of ECC memory (32GB) appears to be only $200:
https://www.ebay.com/itm/32GB-Kit-Kingston-PC3-12800-8-GB-DDR3-1600MHz-CL11-ECC-Memory-KVR16LR11D8-8/302646037192

Also, I made this comparison last week and the prices were lower, but the difference between the price of ECC vs non-ECC has become less.
 

kdragon75

FreeNAS Expert
Joined
Aug 7, 2016
Messages
2,429
I fully agree that ECC should be used in ANY system of ANY importance. I just want to be sure we are not spreading misinformation for the sake pushing best practices.
Also there may be cases where someone HAS ample hardware that not ECC capable. Why buy $300+ on an ECC capable CPU, motherboard, and RAM if its just for storing easily replaceable movies?o_O

Again the price difference only matters if you buying something you don't already have.
 

MatthewSteinhoff

FreeNAS Guru
Joined
Feb 2, 2016
Messages
574
if its just for storing easily replaceable movies?
The hardware recommendations guide is for people who want to protect their data on a network-attached storage device. It is not a 'it might work fine if you are only pirating movies' guide.

Best practice still includes ECC RAM and I don't ever see that changing.

(My first car didn't have seat belts or air bags. I'm buying a car today and am unsure if I need those safety features given that I'm still alive today and have driven seven thousand miles without those features. I'm a Uber driver so, most of the time, the people in my car are unimportant to me. What does everyone recommend with regards to seat belts and air bags? Worth the extra expense?)

Cheers,
Matt
 

Chris Moore

Super Moderator
Moderator
Joined
May 2, 2015
Messages
9,552
if its just for storing easily replaceable movies
My movies are not easily replaceable. I have three copies of the files because it took THOUSANDS of hours of my precious time to make those files and I don't want to do again what I already did before. That would be like saying it is all right to burn down my house because I can just build another.
Then there are the absolutely irreplaceable family photos. I have my own server and a backup server because I am not willing to pay a monthly fee to a cloud storage provider to backup my data. I don't trust them enough to let them have my data for one, but I am not willing to shell out that monthly fee either. So I have a reliable server to store my own data.
If you don't have any data you care about, it doesn't matter what you use to store it. Just burn it to CD and throw it in a shoe box. When you have data you care about, you use proper hardware to store it.
 

kdragon75

FreeNAS Expert
Joined
Aug 7, 2016
Messages
2,429
What you store is irrelevant, only the value. You all saying that ALL DATA IS PRECIOUS... It's up to the owner of the data.

it might work fine if you are only
You are kidding right? It WILL work. You face more risk using a SLOG without its own battery backup/super cap but nobody makes a big fuss about that.

(My first car didn't have seat belts or air bags. I'm buying a car today and am unsure if I need those safety features given that I'm still alive today and have driven seven thousand miles without those features. I'm a Uber driver so, most of the time, the people in my car are unimportant to me. What does everyone recommend with regards to seat belts and air bags? Worth the extra expense?)
Makes me thing of using a RAM drive for important files...

Again I agree its best practice. Nobody here is debating that. I'm talking about immutable facts. ZFS with non-ECC is still safer than NTFS, ext3/4, UFS, etc... WITH ECC. This is why FreeNAS uses ZFS in the first place.

Back to my original point; bad DIMMs will NOT cause a scrub of death.

I don't understand the hostility here...
 
Last edited by a moderator:

Chris Moore

Super Moderator
Moderator
Joined
May 2, 2015
Messages
9,552
What you store is erlevent, only the value.
That is not even a word. If you are going to disagree, you should have some basis for it.
You all saying that ALL DATA IS PRECIOUS... It's up to the owner of the data.
Sure, it is up to the owner, not only the value of the data but how much they are willing to spend to safeguard the data. Saying that the labor required to create the data doesn't factor into the value of the data is what doesn't make any sense. That makes your argument void.
Your kidding right? It WILL work. You face more risk using a SLOG without its own battery backup/super cap but nobody makes a big fuss about that.
Sure we do. Every time someone suggests it, it is pointed out to them that it is a bad idea and if they do it they risk losing their data.
Again I agree its best practice. Nobody here is debating that.
Sure you are. If you are not, why are you saying anything at all.
bad DIMMs will NOT cause a scrub of death.
Maybe not, but that is not why ECC memory is the recommended choice and you arguing against their use doesn't make sense.
I don't understand the hostility here...
You are the one that started the ball rolling. What don't you understand?
 
Last edited:

kdragon75

FreeNAS Expert
Joined
Aug 7, 2016
Messages
2,429
That is not even a word. If you are going to disagree, you should have some basis for it.
This is an unnecessary attack on a person not an idea or concept being presented or disused an is irrelevant. (also corrected in my post)
Sure, it is up to the owner, not only the value of the data but how much they are willing to spend to safeguard the data. Saying that the labor required to create the data doesn't factor into the value of the data is what doesn't make any sense. That makes your argument void.
The labor in not inherently relevant. That's entirely up to the user.
Sure we do. Every time someone suggests it, it is pointed out to them that it is a bad idea and if they do it they risk losing their data.
This is not pushed anywhere nearly as much as the ECC/scrub of death concern was. This was brought up to point out the disproportionality of risk to emphasys in the two cases.
Sure you are. If you are not, why are you saying anything at all.
I am ensuring readers of this thread will have vetted and correct facts (on this subject at least). Best practice is thoroughly documented elsewhere on FreeNAS.org (remember my quote and link in the beginning?)
Maybe not, but that is not why ECC memory is the recommended choice and you arguing against their use doesn't make sense.
I'm not debating why ECC is recommended NOW, only the rumor perpetuated by a number of users and moderators on this forum.
You are the one that started the ball rolling. What don't you understand?
I was attempting to share factual information in an effort to dispel incorrect rumors about ZFSs behavior when used in conjunction with non-ECC memory. I am happy to discuss the technical details of why the use of ECC is still ideal.

I hope this clears up any confusion about my desire to share facts and clear up some long standing misinformation.
EDIT: removed unneeded whitespace & emphasis.
 
Last edited:

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,076
<Moderator hat>

Calm down, everyone, there is nothing here to get worked up about. As always, don't immediately jump at other people's throats (well, don't do it later, either) and give everyone the benefit of the doubt if something that they wrote seems off.

</Moderator hat>

Now, as the author of the document, let me make my position clear:

I'm not planning on changing what the document says. The Scrub of Death is certainly very rare and opinions diverge on whether its frequency actually is zero. That is, however, irrelevant as far as the document is concerned, because ECC is a must when best practices are considered.

For one, the document is long enough as it is without a detailed discussion of ECC RAM. Then there's the fact that I'd rather not imply in any way that not running ECC is a good idea. People have interpreted stuff on the forums in the craziest ways, so I try to stay clear of telling them not to stick beans up their noses.
 

Arwen

FreeNAS Expert
Joined
May 17, 2014
Messages
1,126
@Ericloewe, agreed.

I would like to make one more point about memory that does not come up often, (or at all). Size. We are now seeing 16GB DIMMs come into common usage. And 64GB being used in higher end servers. That amount of density is massive, so much so that I would not trust it not to drop a bit or 2 every now and then.

I think it's such a high risk that when DDR5 standard comes out, they should support 80 bit modules, (with 16 bits of ECC). That would allow for higher density modules, like 256GB to 1TB modules. I personnally don't kmow if 16 bits of ECC would allow detect and correct of 2 bits of errors, and detect only on 3 or 4 bits of errors. But, in my opinion, something has to be done. With that in place, lower priced systems should use 72 bit wide memory with ECC.
 

kdragon75

FreeNAS Expert
Joined
Aug 7, 2016
Messages
2,429
@Ericloewe, agreed.

I would like to make one more point about memory that does not come up often, (or at all). Size. We are now seeing 16GB DIMMs come into common usage. And 64GB being used in higher end servers. That amount of density is massive, so much so that I would not trust it not to drop a bit or 2 every now and then.

I think it's such a high risk that when DDR5 standard comes out, they should support 80 bit modules, (with 16 bits of ECC). That would allow for higher density modules, like 256GB to 1TB modules. I personally don't know if 16 bits of ECC would allow detect and correct of 2 bits of errors, and detect only on 3 or 4 bits of errors. But, in my opinion, something has to be done. With that in place, lower priced systems should use 72 bit wide memory with ECC.
I have read about this concern in the past. You are right, we are reaching some incredible data densities not just in SSD but also DRAM, that coupled with higher data rates will greatly increase the chance of bit flips in a given period of time. That said I don't worry to much with my 4GB DIMMs ;)
 

danb35

FreeNAS Wizard
Joined
Aug 16, 2011
Messages
10,572
I did a comparison (using prices on eBay) and the cost of 32GB of non-ECC memory was a whopping $7.60 less than 32GB of ECC memory
...but that comparison is nearly meaningless. The cost delta isn't so much in the RAM itself as in the rest of the hardware, specifically the motherboard. Proper server-grade motherboards are just more expensive than cheap desktop crap. There are good reasons for the price delta, but whatever the reason, it's still there. Some of the smaller pre-built servers erase that issue (the ML10 certainly did for a while, for example), but if you're piecing together a server from new components, you're going to not going to be able to get a proper server board as cheap as a desktop board.

But that aside, nobody here is arguing against recommending ECC--@kdragon75 sure isn't. What he's arguing against is the use of FUD to support that recommendation--and the Scrub of Death (tm) is FUD, or at best a close relative.
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,076

Stux

FreeNAS Wizard
Joined
Jun 2, 2016
Messages
4,166
I still haven’t worked out why a Xeon Scalable Processor is any better than Broadwell processor from the perspective of a FreeNAS user who is interested in hard disks.

Meanwhile, AMD zen based processors should be on the avoid list until FreeBSD can actually run in them with out stalling after a few days/weeks.

Atoms c3000 systems should be something people consider when looking for hardware. I think the reports are coming in or general good results these days

New Xeon D 2100 is also fairly promising. Good compatibility too since it’s basically a sensible/affordable way to get into Xeon Scalable

There has also been a lot of movement in the high end PCIe NVME with PLP SLOG device space.

Meanwhile intel keeps on iterating on skylake and calling it a new generation...
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,076
Atoms c3000 systems should be something people consider when looking for hardware. I think the reports are coming in or general good results these days
Yeah, though the really nice stuff (lots of SATA/SAS + 10GbE) is nowhere to be found.
New Xeon D 2100 is also fairly promising. Good compatibility too since it’s basically a sensible/affordable way to get into Xeon Scalable
They're still around Xeon E5-2xxx pricing, which limits enthusiasm. The whole BGA situation also makes it very difficult to keep up and recommend specific motherboards.
 

Dice

FreeNAS Expert
Joined
Dec 11, 2015
Messages
1,186
The header for the resource did not change (yet?).
upload_2018-4-7_9-4-50.png
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,076
Yeah, I'd have to upload a new file, which I didn't really want to. It's temporary, anyway.
 

MrToddsFriends

Documentation Browser
Joined
Jan 12, 2015
Messages
1,339
Yeah, though the really nice stuff (lots of SATA/SAS + 10GbE) is nowhere to be found.
Are you expecting more than 12x SATA and/or 2x 10GBaseT? While not exactly cheap, the Supermicro A2SDI-H-TF (8 Cores C3758 Denverton) is available, at least in Europe (sorry, German price comparison link).
https://www.supermicro.nl/products/motherboard/atom/A2SDi-H-TF.cfm
https://www.heise.de/preisvergleich/supermicro-a2sdi-h-tf-retail-mbd-a2sdi-h-tf-o-a1712566.html?hloc=at&hloc=de
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,076
Hmm, I must've missed that one. C3758 is just about perfect for FreeNAS, too.
 
Top