First Server

Status
Not open for further replies.

Stux

MVP
Joined
Jun 2, 2016
Messages
4,358
In my experience, a resilver happens surprisingly quickly...

Code:
resilver in progress since Mon Oct 31 21:40:10 2016
		6.57T scanned out of 18.5T at 2.19G/s, 1h33m to go
		13.8M resilvered, 35.43% done


Now, that just a drive I pulled for a few minutes to test hot swap... 8 way RaidZ2.

(I'm testing a new HBA)

Essentially... the bottleneck is how fast you can read from n-1 drives and write to 1 drive.

Actually... seems like it'd be more or less the same for mirrors vs raidz2.

Maybe it would make a difference if you were CPU/IO bound, but my system certainly isn't.
 

Dice

Wizard
Joined
Dec 11, 2015
Messages
1,410
you should be happy because that provides you with the opportunity to argue against what is stated in the article and provide individuals like myself insight on the matter
That is true, but it gets old after some a while... do a search on the forum to see how many times that link has been argued against.
Beyond that - take the habit of searching the forums too.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Ugh, why do people keep digging that crap up?

The author is an ass. Performance is simply not a concern for a properly-designed RAIDZ2 vdev. Rebuild times are perfectly manageable - and the pool is actually protected by some redundancy during the process, unlike what happens with simple mirrors.

RAIDZ2 is unequivocably safer than a set of simple mirrors.
I have to agree. RAID-Z2 is the better option for many reasons. The idea that mirrors are better would only be applicable under a narrow range of conditions and even then, for reliability, you would need 3-way mirrors which would burn three disks for the storage capacity of one disk.
I am able to saturate a 1oGB network interface with the transfer speed of my storage array. It is fast at rebuild too. I can resilver the array on a drive replacement in around 3 hours.

By the way, RAID-Z1 is not recomended for use with any drive sized 1TB or larger.
I have used RAID-Z3, but ended up destroying the set and rebuilding it as Z2 because the performance of Z3 was very poor, at least with my hardware. My thought is that all the extra reading and writing for the additional parity data slows everything down.
 
Last edited:

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Probably because it gets a high placement in search results and us noobs find the logic to be sound. If they're bringing it up though, you should be happy because that provides you with the opportunity to argue against what is stated in the article and provide individuals like myself insight on the matter instead of walking down a potentially misleading path. :)

Anyway, one of the things that captivated me the most about that article is wanting to expand the size of your storage and swaping existing drives out with larger ones. He argued that in a Z2 of 6x 2TB devices, you could force one to fail, replace it with a larger one, and wait upwards of 12 (if I recall correctly) hours for it to rebuild. With that logic, a full rebuild of a single vdev could take a week to complete if you did one a day. Granted, this isn't something you'd want to do often but... humor me... is it accurate? To contrast, how long would it take in a mirrored setup? I understand the speed at which one can increase the size of their pool is irrelevant if the pool fails before you even need to, I'm not saying it alone is reason to pick mirrored over Z2+, I'm merely trying fish for all of the facts to base a decision on.

I plan to do a lot more research and even experimenting a bit when I get my NAS built before making my final decision. That said, please don't take my skepticism (for lack of a better word) personally; I just like to have all of the information before making a decision.
I certainly understand your desire to have all the answers. I have gone through a lot of the steps to learn the answers the hard way, so I am happy to share my experience with you to help save you the time and expense of involved.

You mentioned my exact configuration (Z2 vdev of 6 drives, 2TB each) so I can give my direct experience. I also went and looked at the article and see that it is dated from February of 2015. Not quite yet, but soon that will be two years ago. The speeds in the article were probably based on results with SATA-I or SATA-II drives. When I was using those older, slower drives and a controller that maxed out at 150MB/s, I saw rebuilds take around 12 hours. Now that I am using a better controller with new SATA-III drives (600MB/s), my rebuild times are much faster. Typically around 3 hours. Unless you are going to connect via a 10GB NIC, the performance of the array is going to be greater than you will be able to utilize, even with RAID-Z3 (which is slower), so the article talking about performance being a deciding factor is using outdated and/or flawed logic. The more important factor is reliability. With RAID-Z2 (what I use) you can have 2 drives fail without any data loss. In a mirror, unless it is a 3-way mirror, you can only loose 1 drive out of any pair of drives. Which is bigger, 1 or 2?

Here is a calculator that will allow you to plug in some numbers: https://jsfiddle.net/Biduleohm/paq5u7z5/1/embedded/result/

The other thing to think about is the likelihood of replacing all the drives in your vdev with larger drives. What I have done, I attached a whole new set of drives to my server (a whole new pool) and moved my data from one pool to the other to get my data from a pool of 5 x 1TB drives to a pool of 5 x 2TB drives. Later, I did the similar process again to get from the pool of 5 drives to my current pool of 12 drives. I still have the 5 x 2TB drives sitting on standby as spares because, why not, and I only used them for around a year. My most recent build has been online for under six months and the one before that for almost a year. (These are the twins in my signature)

It is recommended to keep a vdev under 10 drives, so I could have put more drives in each vdev, but one of the reasons that I kept my vdevs to 6 drives is so that I can expand an existing vdev more quickly by putting in larger drives. Also because 6 x 4 = 24 giving me the potential of 4 vdevs in a 24 bay chassis. My theory is, by the time I need to do that, the larger drives will have come down in price. I am currently using 2TB drives and it is in my long term plan to replace them with 4TB drives. My drive replacement plan, that won't begin until my existing storage is near 70% full, involves buying and installing a 4TB drive once a month or perhaps every other month until all the 2TB drives in a vdev have been replaced. So, I don't plan to change them all at once, and I won't be sitting waiting for the drives to rebuild (resilver is the ZFS term). I did this before too, when I changed from Western Digital drives to Seagate drives. I was using Western Digital Re (enterprise capacity) drives and went to a mix of Seagate Desktop, Constellation, Barracuda and NAS drives. The Seagate drives run about ten degrees cooler and I like the SMART data from the Desktop/Barracuda drives better than what the NAS drives give. I have used every kind of hard drive I can think of and compared the SMART data they provide and the temperatures in my SuperMicro chassis and found that (in my opinion) the Seagate drives are preferable, but I digress.

There is a wealth of information available on this forum that is specific to FreeNAS and the version of ZFS that FreeNAS implements and there are people that are currently using the software on a variety of hardware. If you can't get your question answered here, there may not be an answer.
 
Last edited:

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Here's the proper place to read the argument in favor of mirrors:
http://constantin.glez.de/blog/2010/01/home-server-raid-greed-and-why-mirroring-still-best
Mostly, for a home user with modest storage needs, the big win is flexibility. RAIDZ2 is clearly more reliable.
Please, that article is from 2010. The reasons for mirrors were all about speed and that speed or more can be had with RAID-Z2 which gives distributed parity so that more than one drive can fail in a zpool without causing data loss. Your pointing outside the forum at outdated information.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,358
Raidz2 writes faster than mirrors. And reads nearly as fast.

The comparison is between z2 and pairs of mirrors. So yeah, article is obselete.

But the point about flexibility is true.

A 3 way mirror is in a way a form of double parity array. Like z2.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
Mirrors are needed in IOPS-limited scenarios. Otherwise, I find that they make little sense in most cases.
 

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
The reasons for mirrors were all about speed and that speed or more can be had with RAID-Z2 which gives distributed parity so that more than one drive can fail in a zpool without causing data loss.
It's true, the author does talk a lot about performance, but also devotes a significant portion of his argument to flexibility. I'm drawn to the flexibility argument and tend to forget about the performance part. People often show up in the forums asking whether they can add drives to FreeNAS piecemeal, a la unRAID. Mirrors are what get you closest to that, i.e. add a pair of drives of any size to your array and immediately grow your storage. And I point to this old article because the new one that gets linked to so often adds nothing but misinformation, which annoys me.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,358
The flexibility argument is good. Buy what you need today, and add what you need tomorrow.

I suspect the parity 'loss' may actually be made up over time in more efficient upgrading and reducing future cost of storage and net present value of the cash
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
It's true, the author does talk a lot about performance, but also devotes a significant portion of his argument to flexibility. I'm drawn to the flexibility argument and tend to forget about the performance part. People often show up in the forums asking whether they can add drives to FreeNAS piecemeal, a la unRAID. Mirrors are what get you closest to that, i.e. add a pair of drives of any size to your array and immediately grow your storage. And I point to this old article because the new one that gets linked to so often adds nothing but misinformation, which annoys me.
True, if flexibility is your goal, adding mirrors may be the way to go. I am just too paranoid about having a hard drive fail and loose data. You can setup a FreeNAS vdev on a single drive, then add a second drive to that vdev as a mirror, then add a third drive to the mirror to make it a 3-way mirror (for redundancy). There is a lot of flexibility in ZFS.

The main reason that RAID-Z2 is preferred is because of the cost/benefit ratio. You just pick what works for you and keep in mind that backups still need to be made.
 
Status
Not open for further replies.
Top