ECC vs non-ECC

[MR. T.]

Hi all,

I have been running a freenas machine for a while, and had so many problems that a lesser man would have given up, but it's so pleasing to see things when they start to fall in line.

I have slowly fixed up all the problems, and spent ridiculous amounts of money doing so but now i come to a choice i have to do.

I really, really, REALLY want ECC ram.

My problem comes down to cost. My motherboard will hold 4 dimms to up to 64gb (4x 16gb), but those are quite expensive.
Buying 4 right now is completely impossible.
I am running on normal corsair ram (non-ECC), with 32 gb (4x8gb).

At this point i can keep running as it is till i have the money to buy at least 2 ECC sticks (to keep the amount of ram), or buy one stick and reduce the ram to 16gb.

Normally i would go with the smaller but better ram, but i have 70tb in my pools... running that of 16gb makes me nervous.
In fact just 32gb feels quite short given the 1gb ram to 1tb of storage recommended.

As timings... i expect to buy one ram stick at the end of January... the second one probably only on late march or april.
I would love to be able to get them sooner, but i am pretty sure my wallet wont allow for it.

Given this limitation... what should i do?
Should i wait? or should i reduce the amount of ram, so i can run ECC?

Best regards

 

[Mr_N]

in an ideal world your spending of ridiculous amounts of money would have included a motherboard capable of more than 64GB of ram max ;)
unless your really worried about performance or have alot of other stuff going on apart from just storing files I dont think you'll notice the 16GB ram and would be switching to ECC as soon as possible to avoid this all being for nothing ;)

 

[wblock]

Don't get hung up on the "1GB per TB" thing. It's a guideline, not a hard rule. Instead, look at performance. Is the system performing well? Is it swapping?

You could test this with the existing system by removing 16GB. As always, back up first.

 

[Dice]

My problem comes down to cost.

i have 70tb in my pools...

Given this limitation... what should i do?

You need to decide if you care enough for your data to handle the cost of running FreeNAS.
Having 70TB data and not being able to afford ECC RAM - then priorities have gone severely wrong somewhere in the chain.

 

[hwong]

Just get the ECC RAM - if you get data loss due to ram.. that would cost you more than the price of ECC RAM itself.

 

[mattbbpl]

Don't get hung up on the "1GB per TB" thing. It's a guideline, not a hard rule. Instead, look at performance. Is the system performing well? Is it swapping?

You could test this with the existing system by removing 16GB. As always, back up first.

I concur on the first part. I'm not so sure about the 16GB recommendation. I'm running 60TB on 32GB of RAM, and it experiences slowdown under heavy load. I would be hesitant to halve that amount of RAM on an even bigger pool (albeit marginally bigger).

Regardless, you want ECC RAM as soon as you can get it. 32GB will likely be fine under normal household use.

 

[wblock]

It's not a recommendation to stick with 16GB, just that 16GB might not be as bad as feared, at least temporarily.

 

[ChriZ]

What motherboard do you have?

 

[joel3452]

Maybe it just me, but the most expensive part of my server build was hard drives. If your willing to spend the amount on hard drives, then spending some money on ECC doesn't seem like a stretch.

I know for my build the ~230 bucks for 32 GB of ECC memory seems like nothing compared to the ~1,800 bucks on hard drives.

 

[joeschmuck]

@MR. T.

You put us at an unfair advantage, you never specified your usage of your FreeNAS server so all the previous advice would really apply to most people who are running a small home system with Plex. Small has nothing to do with the capacity of your pool.

So if you are running a Plex media server and maybe running transmission in a jail, dropping to 16GB RAM would be fine and you could probably stay there forever. However if your demands are more then you likely will need more RAM.

What happens if you install the single 16GB ECC RRAM and you really need a bit more? Well that Swap Space you have on each data drive (by default) will provide a place to swap memory out. While FreeNAS should not crash when doing this, it will slow things down. With the 16GB RAM installed you can examine the SWAP Utilization (Under Reporting -> Memory) and see how much is being used. Look now while you have 32GB RAM, it should be a really small value. I have attached mine as an example and I also run only 16GB RAM for my system. I run Plex and backup files, nothing significant and you can see that my used value is zero.

So I would buy the 16GB ECC RAM, install it and burn it in. Then see how your system operates. If you think it's running slower, check out the Swap Utilization and this will tell you how bad it is or if you are just loosing your mind. 70TB of data (I'm sure it's all media) is a lot and if it becomes corrupt, well it will suck to restore it all.

 

[mattbbpl]

Thanks, Joe. I'm really interested in a test case of the 16GB RAM for a > 50TB pool. I've never seen anything on that level, so if anyone gives this a shot I'd appreciate it if you could share the results.

 

[joeschmuck]

Thanks, Joe. I'm really interested in a test case of the 16GB RAM for a > 50TB pool. I've never seen anything on that level, so if anyone gives this a shot I'd appreciate it if you could share the results.

It really depends on what the system is being used for. I'm certain that if I had 90TB of storage, 16GB RAM would be fine for a Media Server and a few light jails. The risk comes with how tolerant your pool is to failures and how fast it can rebuild a new drive. If someone has > 50TB and is only using a RAIDZ2 setup, well I don't think that is a smart idea. Lets say I have ten 8TB hard drives in a RAIDZ2 format and would give me about 58TB usable space, and almost 50TB recommended usable space. If you had 50TB of data, oh my it would take a week or longer to rebuild a single hard drive. That is too much room for error. So my point is, when dealing with capacities this large it's not just about RAM. You would need to build your pool to be very fault tolerant or be willing to take that kind of risk. If I had to store that much important data, I'd have a second FreeNAS server as a backup for my main server, after I shot myself in the head for somehow collecting that much data in the first place. But since I suspect anyone with a home system using that much storage would likely be copying BluRay media, not truly important data in my book. I'm not criticizing, I have a few movies too but I'm in the 280GB range right now. Mine goes up and down, there is no reason to keep a movie that I'll never watch again.

Okay, I'm off my soap box.

 

[MR. T.]

Hi all,

sorry for taking so long to come back to this... i was engaged with other things and didn't have the time.
I came back to the fight flashing an ibm m1015, and so far is a brilliant piece of kit... well worth the money.
The HP 24 port SAS expander it should work with... seems to be dead. :(

My motherboard is an asrock C2550D4I. i bought it because i assumed it would be cheap to run, and my needs weren't that big, just huge storage to keep plex running for 1 user... performance could be sacrificed for storage.

Unfortunately, as with everything, things changed. One day i'll have to write up my endless stream of woes as a cautionary tale to those jumping in to FreeNAS without understanding the basics.

It seems that a system that has all that storage ends up being more useful that just to store a few rarely used files.... and I ended up using it a lot more to the point that performance is starting to be a problem.

Also... the motherboard uses DDR3 UDIMMs... a 16gb stick at this moment costs an arm, a leg and another bodypart at your choice.

I am going for the lower RAM with ECC option as it seems the consensus and the most sane option.

My problems aren't the time to rebuild a disk (as I can wait a few weeks without using the NAS at all, so to not reduce performance or increase the possibility of error), but the fact that my disks are simply disconnecting.
At random times, random disks, of random pools, with random sizes and settings, connected to random controllers (have 4) simply drop off and the pools end up running degraded or go offline completely.

If it was always the same 2 or 3 disks i would think they are damaged, but it has happened to all the disks so far at least once and there are 26 of them.

My memory usage on the NAS is not huge... it always uses all of it for the ARC, but never swaps. (i think it only swapped twice in its lifetime just before it committed harakiri and rebooting itself).

I hate having the swap all over the disks... I'm pretty sure i found something explaining how to set up a dedicated swap disk but can't find it anymore. anybody can post a link for an easy (for a windows user) tutorial on how to achieve this?

 

[Arwen]

...
My problems aren't the time to rebuild a disk (as I can wait a few weeks without using the NAS at all, so to not reduce performance or increase the possibility of error), but the fact that my disks are simply disconnecting.
At random times, random disks, of random pools, with random sizes and settings, connected to random controllers (have 4) simply drop off and the pools end up running degraded or go offline completely.
...

@MR. T.

Please show your disk manufacturer, models and pool layout, in code tags.

Some consumer type disks, (like Western Digital Blues and Greens), have settings that make them less
suitable for RAID. Specifically quick idle parking and no TLER, (Time Limited Error Recovery). The
first can be worked around by using "WDIDLE3.EXE", (spelling?). The second can sometimes be enabled
or changed from the default to 7 seconds.

That said, ZFS is less prone to problems with either of those 2 than hardware RAID.

 

[joeschmuck]

@MR. T.

Also list all of your hardware and how the hard drives are physically connected to the system. I'm thinking it could be a timing issue or just a bad part.

That is a lot of storage.

 

[MR. T.]

Please show your disk manufacturer, models and pool layout, in code tags.

Hi, i don't have the exact information at this time, but it's 11 disks seagate 8tb archive. connected to the 3 motherboard controllers (Intel® C2750,Marvell SE9172,Marvell SE9230). These are as a single pool in raid z2.

I have a bunch of random 500gb disks their majority from seagate. 4 of them are in a raid z1 pool.

the rest of the 500gb disks and a 3tb disk (seagate) and two 2tb disks (toshiba and seagate) are as loose 1 disks stripes.
I needed the space and don't care for the data in them... if it disappears it can easily be rebuilt from what's on the big raidz2 pool.

Also list all of your hardware and how the hard drives are physically connected to the system. I'm thinking it could be a timing issue or just a bad part.

i changed this recently. Originally all 8tb disks were connected via SATA cables to the motherboard and everything else to a 16disk HBA from LSI with sas to sata breakout cables.
The problem is the HBA i bought was old and was limited to 2tb disks.

So i have upgraded and replaced the hba with an IBM M1015 flashed to the IT firmware. The plan is to use an sas expander, but didn't manage to get that to work yet (i have replaced the power cable to the sas expander, and the sas expander itself...). The expander is the 24 port one from HP.

so at this moment i have 4 8tb disks connected to the IBM controller with a SAS to sata cable and 7 connected to the motherboard. everything else id disconnected till i get the expander to run.

The disks dropped off freenas before both from the ones attached to the motherboard and the ones on the HBA. Now they drop off both the motherboard and the IBM controller.

I used to have a very aggressive spindown set on those disks, but have them always running to avoid problems (but they drop off anyway).

At this moment i suspect that one of the 500gb disks that was dropping off to be faulty but i see no reasons for the rest of them to be having that behaviour.

I am going to triage the cables because i have brand new ones to use with the SAS expander. so... as soon as i can get the damned thing to work i'll be sure.
I also want to get it to run so i can connect the 8tb disks to SAS cables. I really don't trust SATA cables anymore

That is a lot of storage.

That's what i thought when i bought my first 8gb disk... and then at the first 40gb disk... and again at 250gb and 500gb.
This time i hope it lasts me for a few years :D

 

[joeschmuck]

That's what i thought when i bought my first 8gb disk... and then at the first 40gb disk... and again at 250gb and 500gb.
This time i hope it lasts me for a few years :D

I can relate. My first hard drive was a used 5MB ST-506 Seagate MFM drive. I got a second identical used drive a few months later. That was a lot for a 80286 computer. I can recall performing head alignments on those drives to keep them operational and then converted them to RLL format which increased the capacity (can't recall what that was), but it was 5MB, that was big and more importantly it was FAST compared to booting from a floppy disk.

There has been some significant advancements and now it's just crazy about the recording density and speed we have now. I cna't imagine what it could look like in 40 year. Hopefully I'll be alive to see it.

 

[Arwen]

Hi, i don't have the exact information at this time, but it's 11 disks seagate 8tb archive. connected to the 3 motherboard controllers (Intel® C2750,Marvell SE9172,Marvell SE9230). These are as a single pool in raid z2.
...

Let me get this straight.

You are using 11 x 8TB Seagate SMR Archive disks in a RAID-Z2?

I have one of those disks and they are not very fast for writting. One problem with them is that they can become
busy in a way that RAID does not like. Specifically during writes, they can stop accepting new commands until
the prior write is complete. Some writes take a long time to process due to the need to read prior tracks before
over-writing the next track. I do use mine in a single disk pool for backups, so it's not a problem for me.

That said, ZFS is less prone to problems with slower disks. It's possible that you have a larger block size than the
default of 128KB. Thus, if you used 1MB block size, (the current maximum), it's possible you ran across an issue
where the disk becomes too busy for too long for ZFS's comfort.

Seagate SMR, (Shingled Magentic Recording), Archive disks have an un-shingled disk write cache, which for the
8TB model I belive is 20GBs. Whence that is full, the disk HAS to flush the data to shingled space. Slower, and
potentially making the drive un-available for seconds at a time.

...

It should not mater if the SATA disks are on SATA or SAS controllers.

If a SATA disk is on a SAS port without a SAS expander, the host SAS port turns it's self into a SATA port, same
cable limitation as normal SATA.

If a SATA disk is on a SAS expander, the SAS expander does the electrical part of the SATA protocol, thus the
SATA disk's cable to the SAS expander is limited to the SATA length. BUT, the SAS expander to SAS host bus
adapter, (aka controller), runs at SAS speed, (upto 12Gbps for SAS 3), and SAS electrical specs, 10 meter cables.
The SAS expander does the SATA to SAS tunneling protocol so that all communication between SAS expander
and SAS controller is SAS.

 

[MR. T.]

Seagate SMR, (Shingled Magentic Recording), Archive disks have an un-shingled disk write cache, which for the
8TB model I belive is 20GBs. Whence that is full, the disk HAS to flush the data to shingled space. Slower, and
potentially making the drive un-available for seconds at a time.

Yes... the disks are quite slow but for my usage (long term storage) they suffice.
I found myself with read speed bottlenecks but not write speed on those drives, probably because i write once... and read a lot of times.

I started to buy these disks when nobody thought they any good, but the "little" drives have been exceeding expectations all over the place.

It should not mater if the SATA disks are on SATA or SAS controllers.

I know it should just work, but since the disks that drop the most are connected to the SATA ports, i wanted to connect them to the SAS expander to make sure the problem isn't the sata controllers or bad quality cables.


Yesterday i managed to get the SAS expander to work.
The powered riser cable doesn't seem to work at all, and i even replaced it with another one.
Connecting to a spare motherboard made all disks come to life.

I'l wait a few days to see what drops off and what runs fine.

 

[Arwen]

Yes... the disks are quite slow but for my usage (long term storage) they suffice.
I found myself with read speed bottlenecks but not write speed on those drives, probably because i write once... and read a lot of times.
...

The read speed can be slow due to the fact that related data may be anywhere on the disks. Meaning ZFS has
one concept of fragmentation, but SMR disks take that to a whole new dimension. This could potentially increase
read time dramatically, due to lots of disk head seeking.

Good luck.

Edit: Fixed "disk head seeking"