Conflicting info 9.3.1

[sonnyinaz6]

Yes, it is a killer, really overkill server, 'at this point' in time. The bump I've seen in my electric bill after firing it up the first time and letting it run was about 22.00 But it's worth it to me. I'll be updating it however over time. I already have a 16 bay sata storage shelf for it that I'll eventually be adding to. This is because I can buy 3.5" sata brand new for 65.00. The sas drives are minimum triple that. Eventually though they'll come down in price as sizes increases and I'll migrate over to those. Eventually I'm going to want about 16tb and I fully plan to have that at about 75% utilized, so the memory, yea, gonna need it.

Here's where we differ momentarily... After working in data centers for over 10 years now, I can without any doubt, none, state from experience, that I have replaced over 1000 hard drives in servers, raid 1, raid 10, raid 5, and raid 6, that had a failed, or predictive failure, that generally gets replaced same day, to 8 months (I've had 2 companies that didn't replace drives in raid 1 that ran a year). Not 1 system.. that's 0, ever failed it's array during this 10 year period of time. Not 1. So with all the expertise I've seen, all the articles I've read about how important it is to have minimum 2 drive failure support (which is why raid 6 came into play) (most data centers rely on heavily protcted dasd now anyways) I can refute every word of it. I've worked in 4 massive data centers now without a fail, and no, after 10 years and over a 1000 failed drives replaced, can you say "you've been lucky". I don't care what anyone, or any company says, I have my facts, and that's what I rely on. Anyone can debate this point with me till their blue in the face, I have real world experience and believe in it.
However, I do agree with having more than 1 drive and eventually I will have. For that much data I'll eventually have 2 hot spares with 2 drive fail capabality, and I think zfs has room for 3?. For the moment though, even running a z1 I've managed to lose damn near 2tb from the 6tb I have, which is about 1tb more than I lose with a hardware raid 5. Can't afford to give up that much space at the moment.

 

[SweetAndLow]

even running a z1 I've managed to lose damn near 2tb from the 6tb I have, which is about 1tb more than I lose with a hardware raid 5.

You lose about exactly the same amount. Blocks on disk are the same no matter if you choose hardware raid or software raid.

Also no enterprise storage solution will let you run a protection level lower than 2 disk redundancy. This is because it's way to risky when rebuilding an array that can take days. This is why everyone suggest having multiple drive protection.

 

[Scharbag]

Yes, it is a killer, really overkill server, 'at this point' in time. The bump I've seen in my electric bill after firing it up the first time and letting it run was about 22.00 But it's worth it to me. I'll be updating it however over time. I already have a 16 bay sata storage shelf for it that I'll eventually be adding to. This is because I can buy 3.5" sata brand new for 65.00. The sas drives are minimum triple that. Eventually though they'll come down in price as sizes increases and I'll migrate over to those. Eventually I'm going to want about 16tb and I fully plan to have that at about 75% utilized, so the memory, yea, gonna need it.

Here's where we differ momentarily... After working in data centers for over 10 years now, I can without any doubt, none, state from experience, that I have replaced over 1000 hard drives in servers, raid 1, raid 10, raid 5, and raid 6, that had a failed, or predictive failure, that generally gets replaced same day, to 8 months (I've had 2 companies that didn't replace drives in raid 1 that ran a year). Not 1 system.. that's 0, ever failed it's array during this 10 year period of time. Not 1. So with all the expertise I've seen, all the articles I've read about how important it is to have minimum 2 drive failure support (which is why raid 6 came into play) (most data centers rely on heavily protcted dasd now anyways) I can refute every word of it. I've worked in 4 massive data centers now without a fail, and no, after 10 years and over a 1000 failed drives replaced, can you say "you've been lucky". I don't care what anyone, or any company says, I have my facts, and that's what I rely on. Anyone can debate this point with me till their blue in the face, I have real world experience and believe in it.
However, I do agree with having more than 1 drive and eventually I will have. For that much data I'll eventually have 2 hot spares with 2 drive fail capabality, and I think zfs has room for 3?. For the moment though, even running a z1 I've managed to lose damn near 2tb from the 6tb I have, which is about 1tb more than I lose with a hardware raid 5. Can't afford to give up that much space at the moment.

Statistics suck. The odds of winning the lottery are terrible, but someone always wins the damn thing. You got lucky. Raid 5 and Z1 waste the same space. Both allow for 1 failed drive, and only 1 failed drive. I have not worked in massive data centres and I have seen a single parity array fail a rebuild. Luckily, the daily backup was OK and not much was lost, except for about 100 work days of labour because that is how many people had to repeat their day due to the loss. Sucked. And the cost of a second parity drive is covered in less than 1 person's daily wages.

So, "your" facts are just that, yours. Statistics, while not always accurate, do allow for us to make educated decisions. Cell phones and fiber optics only work because of statistics. I trust them, and so do most storage architects. If someone told me to put in a single parity array for critical data today, I would laugh them off the site. Use cheap drives, lots of parity and trust statistics. And to beat the dead horse - BACK UP YOUR DATA!!!

 

[sonnyinaz6]

I agree, and I understand what you're saying about it. However, in my case, they don't hold water. statistics do suck, which is why I rely on the information I have.

However, twice now I've seen posts stating z1 takes only as much as raid 5. Can someone tell me why I only have 4 tb then?

 

[sonnyinaz6]

forgot pics

 

[Scharbag]

Just remember that TB and TiB are different. Regardless of what number gets reported to you, Z1 uses 1 drive equivalent for parity, plus a bit of ZFS overhead.

My 12 drive (2x6x3TB) array reports 21 TiB of total capacity. That is 2.625TiB per drive given 8 drives are available for data out of the 12. That is equal to 2.88TB per drive, implying that ~4% of each drive is ZFS overhead IF seagate did not lie about their capacity :) And looking at the ST3000DM001 POS drives I have, they are actually a 2.977 TB drive, so that info reduces the ZFS overhead to 3.3%.

Cheers,

 

[sonnyinaz6]

I do want to clarify something here. I'm not in any way advocating the use of a raid 5 (or the z1 equivalent). this is simply a matter of economics for me at this point time time. Right now I can't really afford to build out and lose too much data to the protection. I would much prefer have the z2 setup, but for me it eats up too much of the data I need. Eventually I will have z2 or if z3 is inherently better I'll run with that. But that's down the road. For now I need every bit of this 4tb I have... nothing more. I'm in total agreement with everyone on the recommenced levels of protection.

 

[Nick2253]

Here's where we differ momentarily... After working in data centers for over 10 years now, I can without any doubt, none, state from experience, that I have replaced over 1000 hard drives in servers, raid 1, raid 10, raid 5, and raid 6, that had a failed, or predictive failure, that generally gets replaced same day, to 8 months (I've had 2 companies that didn't replace drives in raid 1 that ran a year). Not 1 system.. that's 0, ever failed it's array during this 10 year period of time. Not 1. So with all the expertise I've seen, all the articles I've read about how important it is to have minimum 2 drive failure support (which is why raid 6 came into play) (most data centers rely on heavily protcted dasd now anyways) I can refute every word of it. I've worked in 4 massive data centers now without a fail, and no, after 10 years and over a 1000 failed drives replaced, can you say "you've been lucky". I don't care what anyone, or any company says, I have my facts, and that's what I rely on. Anyone can debate this point with me till their blue in the face, I have real world experience and believe in it.

I do want to clarify something here. I'm not in any way advocating the use of a raid 5 (or the z1 equivalent). this is simply a matter of economics for me at this point time time. Right now I can't really afford to build out and lose too much data to the protection. I would much prefer have the z2 setup, but for me it eats up too much of the data I need. Eventually I will have z2 or if z3 is inherently better I'll run with that. But that's down the road. For now I need every bit of this 4tb I have... nothing more. I'm in total agreement with everyone on the recommenced levels of protection.

But see, your earlier statement entirely contradicts what you just said here. How else are we supposed to interpret "Anyone can debate this point with me till their [sic] blue in the face, I have real world experience and believe in it."?

And, really, there's no evidence that would convince you otherwise? You are so dead-set on the value of your own anecdotal evidence that you won't even consider other evidence? If I literally showed you that every other RAID5 array failed, you'd still say "I have my facts, and that's what I rely on"?

The issue of RAID 5 failures isn't just a theoretical one; it's a practical one. RAID 5 array rebuilds fail all the time. Go anywhere on the Internet; it's not an uncommon problem. It's also not an old problem; it's a new one. Experience with RAID 5 ten years ago doesn't tell you squat about the reliability of RAID 5 today given that drive sizes are two orders of magnitude larger, and therefore significantly more susceptible to UREs.

Based on the best empirical study I can find (Microsoft Research Study), URE events occur at rates around once per 3.7E15 bits. For a four 4TB drive array with RAIDZ1, that gives you a AFR of just under 5%. For the same array with 1TB drives, the AFR is under 3%. You can see that drive size plays a large role in that. Even 5 years ago, you were likely dealing with, at best, 500GB drives. Or if you were dealing with high-speed SAS drives, drives that were likely smaller. And if that's the case, it's not entirely surprising you haven't seen a failure.

But we're not talking about small enterprise drives here (which, as enterprise drives, are also an order of magnitude less likely to experience a URE); we're talking about massive high-end consumer drives, which are less reliable, less hardened, and by virtue of their size, much more susceptible to a URE.

It's your data; do with it what you want. Only you can set your priorities. But don't go claiming single-parity arrays are safe, when not only the theory, but the evidence, is against you.

 

[cyberjock]

My apologies, I didn't mean to bail on you earlier, I work a 12 hr graveyard and was up 4 hours after being home already and just died about that time.

Pun not intended... right? :P

 

[sonnyinaz6]

But see, your earlier statement entirely contradicts what you just said here.

Not really. I do believe in this version of software raid with zf filesystems. Something I would not have done in a windows environment. I do fully believe in what is being said here as far as it's better (now) to use this software raid, than the earlier industry standard hardware raid 5. It kinda disturbs me that so much hate is placed upon a standard that has saved who knows how many systems because it was used and worked, well. It's older tech now of course. However as I've stated before in my previous posts, I've had no such failures as preached here in 10 years. None. So to hear this being touted as a worthless technology is pretty disturbing when I know what I know. It's still in use believe it or not in many data centers, although there is less and less and times goes by because there is better technology. My point being, it was a great technology at the time. Now, comparatively, not as great as tech continues to move forward. If not for raid 5, many losses over the years would have taken place. Most businesses have raid 5 to thank for that, and then 6 as it came about. Now there's zf2. Awesome. I'm glad it's here. But I'm not gonna hate all over 5 because zf is better...

Agreeing that it's old hat now, and much better is available is easy to agree with, and I do agree. I'm just a bit perterbed that there are so many (seems like everyone) that's hating on a "proven" technology that did it's job. Time and time again. I don't know what it is about this group of people that doesn't realize what 5 did FOR this industry.
You can look at anything that's old, and say the same thing can you not? I have a 1983 Honda V45 Sabre. It was a muscle bike back in the day. Can't possible compare to what's out there today, but you know what... in it's day, it was awesome...... And in my opinion, so was 5.

 

[Nick2253]

Not really. I do believe in this version of software raid with zf filesystems. Something I would not have done in a windows environment. I do fully believe in what is being said here as far as it's better (now) to use this software raid, than the earlier industry standard hardware raid 5.

It kinda disturbs me that so much hate is placed upon a standard that has saved who knows how many systems because it was used and worked, well. It's older tech now of course. However as I've stated before in my previous posts, I've had no such failures as preached here in 10 years. None. So to hear this being touted as a worthless technology is pretty disturbing when I know what I know. It's still in use believe it or not in many data centers, although there is less and less and times goes by because there is better technology. My point being, it was a great technology at the time. Now, comparatively, not as great as tech continues to move forward. If not for raid 5, many losses over the years would have taken place. Most businesses have raid 5 to thank for that, and then 6 as it came about. Now there's zf2. Awesome. I'm glad it's here. But I'm not gonna hate all over 5 because zf is better...

Agreeing that it's old hat now, and much better is available is easy to agree with, and I do agree. I'm just a bit perterbed that there are so many (seems like everyone) that's hating on a "proven" technology that did it's job. Time and time again. I don't know what it is about this group of people that doesn't realize what 5 did FOR this industry.
You can look at anything that's old, and say the same thing can you not? I have a 1983 Honda V45 Sabre. It was a muscle bike back in the day. Can't possible compare to what's out there today, but you know what... in it's day, it was awesome...... And in my opinion, so was 5.

You have created one heck of a straw man here. Not once did anyone say they "hated" RAID5. Not once did anyone say it is "worthless". There's no "hate" of RAID5 here. It's just absolutely inappropriate for the job today. And that's just the facts.

Since you went with the car analogy, your argument is the equivalent of defending a covered wagon for trans-continental transportation. Sure, it was once the greatest technology, and it was used by hundreds of thousands of pioneers in the US to migrate westward. It's an absolutely essential part of our history, and no one "hates" the covered wagon. Honestly, if it weren't for the covered wagon, automotive technology would never have been the same (much less our national history). But that doesn't mean you defend the covered wagon for transcontinental travel today. And saying that the covered wagon is not appropriate for transcontinental travel today is not tantamount to "hating" it.

Just like RAID5, which was a pioneering technology which brought RAID to the masses, we look back on it with respect. But just because something was once useful, does not continue to make it useful. RAID 5 is no longer appropriate for reliable data protection.

Please, if you've never had the opportunity, visit the Computer History Museum in Mountain View, CA. It has an incredible section on the history of RAID. It's fascinating stuff. But that's where RAID 5 belongs: in a museum. Not "protecting" your data.

 

[sonnyinaz6]

Well, let's keep our facts straight here, I never said anyone stated "they hated raid 5". I said there's 'hating' going on... people not saying anything good about it, at all... there's a difference.... "RAID 5 is no longer appropriate for reliable data protection." This kind of statement... just sends me because it is a reliable form of data protection, and has been for years.... I dunno.. It is old, and it is time to move on, however, I'm currently supporting about 600 of about 5000 servers which still have it until their replaced.
Guess it's just me... forget I said anything....

 

[danb35]

I don't know what it is about this group of people that doesn't realize what 5 did FOR this industry.

Perhaps you can explain further, but I don't see this as having anything to do with the issue under discussion. The claim being made is that RAID5/Z1 does not provide adequate protection today for the data stored on arrays using consumer drives in capacities that are typical today. This claim has nothing to do with the usefulness, validity, or importance of RAID5/Z1 historically (which nobody is disputing or denying), but even so, I think it's exaggerated a bit.

There's no dispute that RAIDZ2 provides more protection for your data than RAIDZ1 (nor Z3 than Z2), but some folks are saying things like "if you have RAIDZ1 with > 1 TB disks, resilvering will fail", which is simply wrong (this absolute statement can be disproven with a single counterexample). Some are saying "will probably fail", which is still wrong--"probably" indicates a >50% probability, which is simply incorrect. The truth is that the statistical chance of encountering an unrecoverable read error during resilvering is higher than some find acceptable. The effects of such an error could be anywhere from catastrophic (if it comes up in, say, the partition table) to minor (if it's in a data block, it would result in a file being corrupted) to negligible (an error in the metadata would result in reading from one of the extra metadata copies).

 

[solarisguy]

@sonnyinaz6, if the servers you take care of are anything like I am familiar with, you are perfectly right in saying that RAID5 is fine.

However, the other guys in the thread are not concerned about the servers running for around ten years ago with 72GB disks (or 142GB ones, you get the idea). They concentrate on the ones being deployed today with 6TB or 8TB disks. It is that 100 times greater capacity that makes RAID5 (or RAIDZ1!) a sure recipe for a data loss.

The data loss would not occur immediately. But, if you have 1000 servers running for 10 years...

They want to protect the forum readers from disasters some time in the future. Some of those who come here say outright that they need capacity, but do not have money for unnecessary protection. They only know that a single disk offers no protection, but they are often misguided in thinking that the 11 disk RAIDZ1 (RAID5) with 8TB disks offers all the protection they ever need.

Also, if we think ahead, it becomes obvious that in a couple of years, this forum would be filled with a discussion on RAIDZ3 versus RAIDZ2! Why? Disks capacities keep increasing...

 

[sonnyinaz6]

@sonnyinaz6, if the servers you take care of are anything like I am familiar with, you are perfectly right in saying that RAID5 is fine.

Well, here I have to surrender... and you're right. I work with a mixmash of new and old servers, gen 3 4 5 HP, mostly gen 4 5 dell and a few sun boxes t2000's and the like.. These mostly have between 73gb and 300gb. A couple at 1tb. However even with the newer gen7 gen8 servers there's only 2 drives, raid 10 anymore as the company I work for now is replacing all the older systems many of them booting to nas. I get it.. thanks for pointing that shortcoming out.

I have the LSI Controller installed, and of course a new error, however found a post for that, controller firmware 5 not matching 16... gotta get busy.

Thanks to everyone for helping out, and yea, I'll try n think a little more thoroughly...