ZFS or Hardware RAID Non-ECC RAM

[Charles Hurst]

Afternoon,

Apologies if you feel this is a duplicated question, I have looked around and done huge amounts of reading but wanted to just ask the question for my particular setup. This is a new field for me as most my clients/previous work, etc. has been completed using SAN/NAS kit brought for huge sums of money (mainly NetApp).

Sadly I do not have huge sums of money to buy one of these what I do have a machine which is your basic home hardware desktop no Server grade kit except for a Network card (Intel dual NIC Gigabit which I'm hoping to Bond). It has a RAID controller for the built-in SATA ports which will be hosting 2x2Tb SATA drives.

This will be for a Lab environment so not critical data however would prefer it not to loose data continuously as I might not get too much work done in my Lab. It will be used for hosting XenServer VM's via an ISCSI target so I'm guessing corruption would mean death to any VHD's almost instantly.

So what I already know is:

Do not use hardware RAID and ZFS together.
Do not use ZFS without ECC Ram due to almost certain risk of corruption.

What I'm unsure of is:

What is FreeNAS using if its not ZFS, in other words what file type does its ISCSI LUN use?
If I'm not using ZFS should I RAID 1 my drives using the hardware RAID and create the LUNS on just the one present drive which would still be protected by the hardware RAID?
As the drives are only 7,200 SATA drives should I forget about redundancy as its a Lab environment and just RAID them in Striped 0 to gain a speed increase and if it goes bad then just rebuild my lab?

Thanks in advance,

Charles

 

[danb35]

Do not use ZFS without ECC Ram due to almost certain risk of corruption.

Fixed that for you--there's a non-zero risk, and the way that ZFS operates increases that risk compared to some other filesystems, but it's far from "almost certain".

What is FreeNAS using if its not ZFS, in other words what file type does its ISCSI LUN use?

On FreeNAS 9.3 and up, you are using ZFS. You then create a zvol, and that's what's presented as the iSCSI LUN.

If I'm not using ZFS should I RAID 1 my drives using the hardware RAID and create the LUNS on just the one present drive which would still be protected by the hardware RAID?

Not if you're using FreeNAS.

As the drives are only 7,200 SATA drives should I forget about redundancy as its a Lab environment and just RAID them in Striped 0 to gain a speed increase and if it goes bad then just rebuild my lab?

That's really up to you. Striping the disks gives you double the capacity, less-than-zero redundancy, and higher performance. Only you can prioritize these issues for your use case.

 

[Charles Hurst]

Thanks danb35 very interesting answers.

I can see from what your running you have RAIDZ2, therefore I think I'm using the terminologies incorrect.

I thought ZFS was the name of FreeNAS RAID system but it isn't it's the whole file system and FreeNAS raid is RAIDZ2 correct?

So with or without RAID in FreeNAS I'm at risk of data corruption using non-ECC ram?

I thought the risk of data corruption came with RAID not the filesystem as a whole.

Ok that kind of makes my questions bit redundant then.

 

[cyberjock]

Keep in mind if you have non-ECC RAM and it fails, you likely won't know how bad the corruption is until you've lost your data and any backups that weren't corrupted have expired. That's happened to something like 99% of users that have had non-ECC RAM with a good backup routine.

 

[Charles Hurst]

Thanks cyberjock that actually just answered a question I had in my head.

I'm reading and digesting https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-ram-and-zfs.15449/

Does the post I've pasted and what your saying mean that if the non-ECC RAM does not 'go bad' as in if the RAM is healthy and stays healthy then corruption will not appear? In other words does your RAM have to be dead/dying to cause file corruption, would file corruption not occur on fully working healthy RAM?

 

[danb35]

ZFS is both a filesystem and a volume manager, which has gotten some folks' panties in a twist due to the allegedly blatant layering violations. That allows it to do a lot of cool things, though (to give just one example, resyncing (aka resilvering) a RAID array only copies the blocks that are used, rather than every block on the device). ZFS will allow you to create single- or multi-disk volumes with or without redundancy. Because of the self-healing nature of ZFS, it's a bit more susceptible to corruption due to bad RAM. This risk can be nearly eliminated with the use of ECC RAM, which is why ECC is so strongly recommended.

FreeNAS 9.2 and prior gave the option of using UFS, which is less resource-intensive and somewhat less susceptible to corruption due to bad RAM. Even in the versions where it was present, though, support wasn't the greatest--at least since 8.x, FreeNAS has really been about putting a nice GUI on a ZFS-based NAS.

 

[Charles Hurst]

Thanks again guys been a massive help.

By the way since when did ECC RAM become so cheap?? Last time I purchased ECC RAM it was about £20 a GB just found 8GB for £60 at that price I might as well just order some :)

 

[HoneyBadger]

The RAM will only cause corruption if it's having issues; the problem with non-ECC RAM is that it won't tell you.

ZFS is the filesystem; RAIDZ2 is a configuration of a vdev. To use an analogy, it's the number after "RAID X" - RAIDZ2 is the ZFS partner to RAID 6, RAIDZ1 is the partner to RAID 5. A mirror is a RAID 1, a stripe is a RAID 0.

In your situation I'd run a 2x2 configuration - two 2-drive vdev mirrors. Can you post the rest of the configuration you're planning to use? Running VM targets is a fairly demanding use-case.

 

[Charles Hurst]

Hey HoneyBadger thanks for your reply.

It's:
Quad Core AMD PileDriver FX-4 (unsure which CPU model exactly)
16Gb of non-ECC DDR3 RAM
64GB SSD HD (which I thought I would install FreeNAS onto)
2 2TB WD Red SATA 3 HDD's
2 Dual NIC PCI-E Intel Gigabit Cards (was hoping to Bond either 2 and 2 or all 4 for 4Gbps)
I also have a layer 3 smart switch to do all the clever networking bits with LACP bonded NIC's, etc.

I was going to run XenServer then virtualise FreeNAS but from what I read putting a hypervisor between FreeNAS and its hardware is not recommended so I will install it directly on to the SSD.

I guess performance is more important than redundancy as I will be running quite a intense Lab so if RAID 0 (a stripe) is the way to go then I will do it for the speed.

 

[Tywin]

Keep in mind if you have non-ECC RAM and it fails, you likely won't know how bad the corruption is until you've lost your data and any backups that weren't corrupted have expired. That's happened to something like 99% of users that have had non-ECC RAM that failed with a good backup routine.

Fixed that for you. Just because you have non-ECC RAM doesn't mean it's going to go bad.

 

[HoneyBadger]

Hey HoneyBadger thanks for your reply.

It's:
Quad Core AMD PileDriver FX-4 (unsure which CPU model exactly)
16Gb of non-ECC DDR3 RAM
64GB SSD HD (which I thought I would install FreeNAS onto)
2 2TB WD Red SATA 3 HDD's
2 Dual NIC PCI-E Intel Gigabit Cards (was hoping to Bond either 2 and 2 or all 4 for 4Gbps)
I also have a layer 3 smart switch to do all the clever networking bits with LACP bonded NIC's, etc.

I was going to run XenServer then virtualise FreeNAS but from what I read putting a hypervisor between FreeNAS and its hardware is not recommended so I will install it directly on to the SSD.

I guess performance is more important than redundancy as I will be running quite a intense Lab so if RAID 0 (a stripe) is the way to go then I will do it for the speed.

If you plan to run an "intense" lab (how many VMs? What workloads? SQL? Mail services?) then you're going to need more RAM, at least 32GB.

Don't use the SSD for boot - FreeNAS uses only 4GB or so of that, and the rest will be wasted. Get a decent 8GB thumbdrive (Sandisk) and use that instead, leaving the SSD for another system or possibly L2ARC - but you'll have to go to 32GB first before L2ARC won't hurt more than it helps. Maybe use it as a separate standalone zpool for small high-performance VMs.

With only two 7200rpm HDDs you're probably going to have to stripe them to get any kind of high performance, which means you're full YOLO on the redundancy level already, so stick with iSCSI and sync=standard. Also regarding iSCSI, you'll want to leave the interfaces unteamed and use MPIO rather than LACP. You'll want to put them in separate VLAN at the least to help ensure your paths stay distinct.

Fixed that for you. Just because you have non-ECC RAM doesn't mean it's going to go bad.

Again - the problem with non-ECC RAM is that it doesn't tell you if it's going bad. If you have a non-ECC system you have no way of knowing there isn't corruption already existing until it's too late.

 

[Charles Hurst]

Thanks HoneyBadger that is all great advise.

I will be running probably around 16 VM's per Server in HA so 32 VM's total (they will be Microsoft and Citrix so a mix of everything including a SQL Server for SCCM) however if need be I'll build another FreeNAS box for the 2nd XenServer to keep them separated so actually maybe only 16 VM's.

I have 32GB of RAM in my main machine DDR3 which is probably a waste to be honest as I barely use it except logging on to Servers and working/studying so maybe I'll swap the 16GB and 32GB to give the FreeNAS box 32GB of RAM.

I just quickly read about L2ARC (first I've ever heard of it) sounds really good, definitely worth getting the extra RAM and using from what I read.

OK very interesting on the MPIO over LACP advise, I will have to do some reading up on MPIO as although I've heard of it not touched it at all. LACP I've used a lot so quite comfortable with it. Happy to separate VLAN's got about 10 running already so no issues there :)

Thanks again, I'm sure on Friday when I sit down to build this I will be back lol

 

[Tywin]

Again - the problem with non-ECC RAM is that it doesn't tell you if it's going bad. If you have a non-ECC system you have no way of knowing there isn't corruption already existing until it's too late.

Sure, never disagreed with that; if I had a FreeNAS server it would be running ECC RAM. The point remains, though -- using non-ECC RAM is not a guarantee of data loss, it is merely an increased risk of data loss.

 

[HoneyBadger]

32 VMs including SQL? All I can say is that you'd better start loading that server up slowly and keep a close eye on your latency numbers.

You'll probably find yourself in need of more back-end vdevs.

 

[Charles Hurst]

You'll probably find yourself in need of more back-end vdevs.

Sorry I maybe being really stupid but how would more vdevs help against a bottleneck? Also in that regard what in your opinion would be my biggest bottleneck, in terms of Drive Read/Write speed, SATA port speed, System resources, network throughput (NIC speed), etc?

Maybe I don't fully understand what a vdev is but I though a vdev was like a block device like another partition, is that not the case?

Thanks for the help

 

[HoneyBadger]

A vdev is a component of your zpool. You're thinking of a zvol, which you'll access as a block device over iSCSI.

While ZFS does wonders with buffers and cache, at some point it's got to either flush that buffer to stable storage (writes) or it will miss the cache (reads) and have to go to your pool. With a small number of vdevs your random I/O performance will be low when this happens, and especially in a blended, heavily random I/O scenario (VMs) it's going to enact a serious performance penalty against your zpool when that happens.

This may never happen if the usage is fairly light and your working set of data is small enough to be cached entirely in ARC/L2ARC, because you'll be able to fulfill your reads entirely from RAM or SSD while ZFS is furiously spooling the in-RAM transaction group down to disk. But if you have a cache miss during that time, it's going to be a long time, relatively speaking, before that data will be able to be read from some very busy disks and returned back up the data path.

I can see it being a server that "dies under heavy load" in that when a large volume of data is written to it, the data gets ingested very quickly initially (into RAM) and then there's a hard stall for reads and writes as all the underlying vdevs are going "sorry, I'm busy writing this transaction group" which could cause iSCSI timeouts and subsequent VM hangs.

 

[cyberjock]

32 VMs including SQL? All I can say is that you'd better start loading that server up slowly and keep a close eye on your latency numbers.

You'll probably find yourself in need of more back-end vdevs.

Yeah, it won't last. He'll see. Just talked to someone today that has 196GB of RAM, 800GB of L2ARC, less than 30 VMs, but there's a few databases in there and they are having problems preventing the VMs from crashing due to high latency issues.... 32GB of RAM is gonna be hilarious to watch. It won't last long before he'll figure out that ZFS is laughing at his hardware.

 

[Charles Hurst]

Hey thanks for the further updates...

OK so I'm kind of in four minds at the moment, one part of me is saying FreeNAS and ZFS is just too overkill for my Lab Environment from this post and what I've read is. Also as I'm not studying/working with Storage it would be crazy to spend copious amounts of time learning and setting this up, so should I look for something 'lighter'.

Another part of me is saying even if I look for something 'lighter' will I still be bottlenecked by my hardware just because I'm expecting too much from what I have....

Then a third little voice is saying just buy a cheapish NAS that supports ISCSI like the Netgear RN104...

Then the fourth part of me is saying just use one of my Servers (I have two huge 16 core 196GB or ECC Ram beasts) for storage however two issues I have with that: 1. I would have to compromise by loosing one of my virtual hosts 2. I didn't want to leave these Servers on all the time due to the amount of electricity they eat up...

Ahh decisions decisions... Thanks again for your time on this post I know it's a bit below your level of knowledge!

 

[cyberjock]

I will tell you that the "file server expert" employment opportunities are doing very well lately (especially ZFS experience). Not enough of them around. ;)

 

[HoneyBadger]

I will tell you that the "file server expert" employment opportunities are doing very well lately (especially ZFS experience). Not enough of them around. ;)

I need to find one I can pull off as a fly-in consultant.