Lower RAM requirement for RAID1?

[danb35]

There is no way of mirroring vdevs

Correct; you mirror disks, not vdevs. Multiple vdevs are always striped.

 

[John Nas]

Note that ZFS has a feature to split off a mirror from a pool made of Mirrored vDevs.

Does that mean Arwen too was wrong earlier in this thread?

 

[danb35]

Does that mean Arwen too was wrong earlier in this thread?

As a matter of terminology, yes. Substitute "disks" for "vdevs" in her statement and it's correct.

 

[John Nas]

As a matter of terminology, yes. Substitute "disks" for "vdevs" in her statement and it's correct.

Ahh, I was under the impression vdevs effectively cannot be altered once created. Does this mean you can in fact remove and reattach a drive from a mirror vdev without affecting the pool the vdev belongs to? Would this be a more or less advisable strategy to periodically syncing a drive to be stored in a secondary location compared to something like a simple file copy via rsync?

 

[danb35]

Ahh, I was under the impression vdevs effectively cannot be altered once created.

That is mostly correct. Adding a disk to create (or add to) a mirror (i.e., turn a single disk into a mirror, or a 2-way mirror into a 3-way mirror), or removing a disk from a mirror to reduce redundancy, are the exceptions.

Does this mean you can in fact remove and reattach a drive from a mirror vdev without affecting the pool the vdev belongs to?

Yes, but it's really only practical if you only have one vdev. If you have, say, four disks in striped mirrors (which would be two vdevs), the data is going to be striped across the two vdevs, and you won't then be able to take out one disk and have anything usable on it.

Would this be a more or less advisable strategy to periodically syncing a drive to be stored in a secondary location compared to something like a simple file copy via rsync?

My preference would be to create a separate backup pool, and use ZFS replication to keep it in sync with your main pool.

 

[Turgin]

But there's nothing about ZFS that makes it more risky than any other filesystem with non-ECC RAM.

I've done a far bit of reading on these forums and unless I'm guilty of the same internet FUD you mentioned earlier this doesn't fit with what I believe based on my research.

Based on this thread https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-ram-and-zfs.15449/ due to the checksumming aspect, especially during scrubs, I don't think this is true.

Now, real world reliability of RAM in general and its affects on the actual risk involved may be a different matter.

 

[danb35]

due to the checksumming aspect, especially during scrubs, I don't think this is true.

I'm aware of that thread. Here's why it isn't true:

In order for a scrub to damage data as a result of bad RAM, a number of things have to happen:

1. The pool in question must have redundancy of some sort
2. The scrub reads a block on disk along with its checksum. Due to bad RAM, either the data or the checksum is read incorrectly, such that they don't match.
3. The scrub then, noting the error, goes to {parity|the mirror} for a clean copy of the data. Also due to the bad RAM, it reads/computes the data incorrectly. Further due to the bad RAM, it reads the checksum incorrectly. And by a sheer stroke of bad luck, the bad data matches the bad checksum.
4. Having now read bad data that ZFS thinks is good, the scrub writes the bad data over the good data read in step 2 (which it thought was bad).

As you can see, this is a highly unlikely series of events. If you want to change "highly unlikely" to "astronomically unlikely", you can tell ZFS to use crypto-grade checksums, which makes step 3 pretty much impossible. For more information, see http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/

Edit: Another post from the same author lays out some good reasons for using mirrors rather than RAIDZn. I don't agree with all his reasons as absolutes, but many of them are valid considerations.

 

[Turgin]

@danb35 Fair enough. Just trying to be as (correctly) informed as I can be.

 

[John Nas]

@danb35

That second post you linked to was very helpful thank you, especially the details on upgradability. Having already purchased two 6TB drives, it seems most economical to purchase a third for a 3 drive mirror than two more for the same level of redundancy in a four drive RaidZ2. Any fourth drive I can get my hands on would provide more data security as an offsite backup at this point, and trying to achieve 12TB of storage is perhaps a problem best deferred until necessary (at which point perhaps prices will have changed for such capacities).

I assume there's no direct migration path that let's you add a fourth drive to a 3 drive mirror, and convert it into a four drive RaidZ2, even in the proposed vdev expansion update?

As you can see, this is a highly unlikely series of events.

Edit: I'll add that many of the pro-ECC arguments, especially the ones about the "scrub of death", also border on FUD.

ECC RAM should be used, IMO, in any server, under any OS, with any filesystem--if you care about your data, you should be using ECC RAM as much as possible

So I understand your position clearly, you think that the particular risks of a ZFS scrub with non-ECC ram have ben exaggerated, but the general risks of using non-ECC ram are significant enough to warrant people investing in ECC chips, ram and boards?

 

[danb35]

I assume there's no direct migration path that let's you add a fourth drive to a 3 drive mirror, and convert it into a four drive RaidZ2, even in the proposed vdev expansion update?

There is no such current path, and I'm not aware of anything under consideration that would do this either.

So I understand your position clearly, you think that the particular risks of a ZFS scrub with non-ECC ram have ben exaggerated, but the general risks of using non-ECC ram are significant enough to warrant people investing in ECC chips, ram and boards?

That's an accurate summary. If you're willing to consider used server hardware, the investment need not be great.

Edit: There's always a tradeoff among cost, capacity, and redundancy. Three-way mirrors are generally considered on the paranoid side of that tradeoff--there's nothing wrong with them as such, but you do lose 2/3 of your total capacity to redundancy. One more disk in RAIDZ2 gives you the same redundancy, but twice the capacity, and is a more common configuration.

Though I think that post gives many valid considerations, I disagree with the author's position that wanting to use more than 50% of your disk capacity is "greedy", and certainly with his conclusion that everyone but wizard-level sysadmins should use mirrors. But the pros and cons presented are, broadly speaking, valid.

 

[gpsguy]

One thing you could do, is populate your RAIDz2 vdev with different sized disks. Say you had 2x4TB + 2x6TB disks in a RAIDz2 vdev. ZFS would effectively see the 6TB disks as 4TB disks. But, once you were able to replace the 4TB disks with 6TB disks, your storage would automatically expand. It's one way to work around the need to backup/rebuild your pool at a later date. But, the cost, is that the pool suze is based on the size of the smallest disk.

and convert it into a four drive RaidZ2, even in the proposed vdev expansion update?

 

[Arwen]

Note that ZFS has a feature to split off a mirror from a pool made of Mirrored vDevs.
...

Does that mean Arwen too was wrong earlier in this thread?

As a matter of terminology, yes. Substitute "disks" for "vdevs" in her statement and it's correct.

Okay, okay I give in. Less than ideal wording.
Perhaps something like this;

Note that ZFS has a feature to split off disk(s) from a pool made of Mirror vDev(s). This creates a new striped pool from the split off disk(s). Of course if you have multiple Mirrored vDevs, this requires each vDev be made of more than 1 disk.

 

[Arwen]

Now on the subject of ECC, I like the approach;

If you care about your data, make an effort to use ECC memory.

This really applies to any file system, not just ZFS.
There is also the corollary;

If you care enough about your data to use ZFS, why aren't you using ECC memory?

Both those said, I do use ZFS exclusively on my Linux boxes, none of which have ECC memory. (Desktop, laptop & miniture media server.) I hope to replace the 2 older ones with ones that support ECC memory. Mostly because I also want to increase the amount of memory, (8GB -> 16GB for the desktop, and 2GB -> 8/16GB for the laptop). So, more memory means more chance of memory errors...

But don't let me start another ECC verses non-ECC debate. As far as I am concerned, the 2 statements above sum it up. And the most common answers for the second one are availablility and cost. We can't get computers with ECC memory as easily as we can non-ECC memory. And in many cases, the cost is prohibitive. Those answers are perfectly reasonable and understandable.

 

[rvassar]

I've always felt the "scrub of death" scenario is a bit far fetched. But before everyone rushes off to build non-ECC FreeNAS boxes... Consider that the likely outcome of an uncorrected memory error is a kernel panic. As a survivor of the UltraSPARC II e-cache parity error issue of 2000, let me assure you... Having your server panic and reboot every 18 - 30 days like clockwork is no picnic.

 

[danb35]

If you care about your data, make an effort to use ECC memory.

If you care enough about your data to use ZFS, why aren't you using ECC memory?

I think this sums it up about perfectly.

 

[Chris Moore]

I guess I am late to see and respond to this...

are the memory requirements dictated only by drive size and the ZFS file system, and thus have less to do with the complexity of parity calculations?

But I could not help saying ZFS uses memory for cache (Adaptive Replacement Cache or ARC) and it has nothing to do with the way the pool is configured (drive layout) it has to do with the amount of data, the number of users, the kinds of transactions (large files / small files) and if the same data is being handled repeatedly or if it is always handling different data. A lot goes into it, which is why the general rule is 1GB of memory per 1TB of storage. It gets you in the ball park of where you should be, but you should never even try to use less than 8GB of memory with ZFS and ZFS is the file system for FreeNAS.

 

[John Nas]

If you care enough about your data to use ZFS, why aren't you using ECC memory?

I would say mostly because ZFS is free, while ECC ram is not, especially considering that it is not usually just a case of buying the ram, but also the motherboard and CPU that support it. I think most recognise FreeNAS and ZFS in general are enterprise technologies and not like solutions such as unRaid which are built specifically to support enthuiast hardware recycling. Obviously data security/integrity is of different priority to different users, but I doubt anyone who goes to the trouble of setting up centralized storage, doesn't care if they lose what they're storing, and so systems like ZFS entice.

With regards to creating backup copies of a mirror, is there any data on which technique is more stable/less prone to error? Rsync vs, disk splitting vs ZFS replication?

 

[John Nas]

it has to do with the amount of data, the number of users, the kinds of transactions (large files / small files) and if the same data is being handled repeatedly or if it is always handling different data.

Are there simple correlations for these factors? Less users, smaller files, repeated data = generally lower memory requirements? (With the accepted caveat that 8GB is the minimum).

I'm also curious how that 8GB threshold is reached, as it is often mentioned against the 1GB for 1TB rule. How can the minimum recommendation be a fixed value when it has a correlation to the pool capacities? Is it something to do with the particular version of ZFS? I ask since years ago I tinkered with a 2 drive nas4free box on a little low powered machine. At the time FreeNAS still had the 8GB recommendation but the 2GB I had was well within nas4free's recommendations. When I recently considered unmothballing the build I went to update to the latest nas4free and noted their requirements has now jumped up to the same 8GB.

 

[Chris Moore]

With regards to creating backup copies of a mirror, is there any data on which technique is more stable/less prone to error? Rsync vs, disk splitting vs ZFS replication?

Rsync does a comparison of the source and destination to see if the file is the same or has changed and only copies the file if there is a difference. ZFS replication looks at the time markers in the file system and if there is new data, it copies over the new data. Both of those systems are reasonably reliable and have been used in the enterprise for years and they can be automated to keep two servers or even two pools inside the same server synchronized. Splitting a disk out of a vdev is a manual process. I would not suggest that as a way of making a backup, but it does have it's purposes and I have used it before. As @danb35 already said, it would only work in a pool with a single vdev. Once you go to a larger storage pool with striped vdevs, you can't do that any more.

 

[Chris Moore]

I'm also curious how that 8GB threshold is reached, as it is often mentioned against the 1GB for 1TB rule. How can the minimum recommendation be a fixed value when it has a correlation to the pool capacities?

Because ZFS is software and it is designed to expect a certain amount of memory to be available. My understanding is, ZFS can work with less than 8GB of memory, but it turns caching off. I read that in a post on the forum a couple years ago and I can't cite the source.

Is it something to do with the particular version of ZFS?

There was a time before when ZFS did not required 8GB as a minimum but I am not aware of when this changed.

I ask since years ago I tinkered with a 2 drive nas4free box on a little low powered machine. At the time FreeNAS still had the 8GB recommendation but the 2GB I had was well within nas4free's recommendations.

There was a time when NAS4Free was using UFS instead of ZFS. Do you know that it was ZFS? If it has been 'years ago' it is very likely an earlier version of ZFS. If I recall correctly, there have been at least 3 upgrades of the ZFS features since I have been using it with another coming in FreeNAS 11.2

When I recently considered unmothballing the build I went to update to the latest nas4free and noted their requirements has now jumped up to the same 8GB.

Probably because of the ZFS upgrades I mentioned.