Kickstarter for Block Pointer Rewrite?

Status
Not open for further replies.

Sasayaki

Explorer
Joined
Apr 20, 2014
Messages
86
Is there any hope for Block Pointer Rewrite?

If the answer is: "iXsystems is unwilling to pay for it, you are welcome to contribute your own fix"...

Would this be something suitable for a Kickstarter? Who would be a good person(s) to hire for this task, and would anyone hazard a guess towards a good goal?

Suggested stretch goals could include:

- Assuming BPR works, enable functionality to expand, contract, and change RAID levels (convert RaidZ1 to RaidZ2 by adding another disk, or a 2xmirror, 3x stripe->RaidZ3. Etc)
- FSCK/recovery tool (I know they're not "necessary" but many users will find them useful).
- Anything else?

Thoughts?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
So here's a little history in case you don't know the history..

Sun owned ZFS from v1 to v28. At v28 Oracle bought Sun and starting with v29 close-sourced ZFS. The open source community wanted to continue development so they went to v5000 and created feature flags. Since 2009 when Sun was purchased the open source community has had some pain. There was no funding for ZFS officially and there wasn't a group of developers devoted to ZFS. Each ZFS version(be it FreeBSD, illumos, etc) had their own developer(s) that did feature flags. Some were OS specific, some were ported to make compatible elsewhere. Overall it was a nightmare for ZFS. Then in Q3 2013 the OpenZFS initiative came about. Many of the original ZFS developers now work on the OpenZFS initiative and are pushing out new features and such. Some of the developers of ZFS have moved onto other projects. Naturally as the original core team disperses that knowledge will be lost.

BPRW was something that was a wet dream around v10, but was also conceived of before v1 hit the market from what I've read. It was something that was going to take considerable work, but many people felt it was very important and very necessary long term to the survival of ZFS. It sat on the proverbial "drawing board" until Sun was sold.

Now, however, things are different. There isn't the massive funding that existed at Sun to partake such a huge product. All those features that have been released since v1 have made BPRW an even bigger and harder monster to slay. The end of 2013 one of the lead OpenZFS developers said that BPRW is probably impossible to implement now and isn't very likely to ever exit for the open source version of ZFS. It's still on the proverbial "drawing board" but implementation is so difficult because of everything you have to consider and factor in. The other thing is that it was expected that fragmentation would make BPRW a necessary project that would be absolutely necessary for good ZFS performance in the distant future. Well, the "distant future" is now. And right now it doesn't look particularly important if you're a good zfs admin and set reasonable expectations and don't do things that are stupid with your zpool.

So when am I expecting BPRW? When hell freezes over or someone coughs up a few million dollars(or more) to pay developers to create it. To be honest, I'm expecting hell to freeze over first. And as time goes on and developers get older and start retiring a whole new generation of developers will have to pick up the proverbial flag and carry it on. A lot of experience will be lost as those developers retire, so the window of opportunity is slowly closing.

Overall this isn't an iX thing. This is a ZFS thing. iX is not likely to be important enough to provide the necessary financial investment to get the work done. If its any consolation, Oracle hasn't done BPRW either and allegedly it was being worked on when Oracle bought Sun and the original ZFS team began dispersing.
 

Sasayaki

Explorer
Joined
Apr 20, 2014
Messages
86
Really interesting Cyberjock. Thanks for that. I didn't know most of it.

What about an offline tool? That'd be a lot simpler, I'd imagine.
 

fracai

Guru
Joined
Aug 22, 2012
Messages
1,212
To add to this, I saw a post at some point that indicated that BPRW wasn't even necessary to achieve the dynamic RAID level changes that you're suggesting. Admitedly, I don't have the post in front of my so I may have this wrong, or I might be applying the BPRW comment to a different context than you're refering to.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Online or offline doesn't really solve the primary problem. You have to keep consistency of the file system at all times.
 

solarisguy

Guru
Joined
Apr 4, 2014
Messages
1,125
I was not looking for it, but just found the original statement about BPR from the 11th of October 2013, the one cyberjock had summarized for you earlier
OpenZFS Office Hours with Matt Ahrens
Device Removal & Block Pointer Rewrite topic starts around 0:44:46.

You may also want to take a look at a presentation from the 18th of December 2013 (posted to YouTube last month)
OpenZFS (part 3) by Andrew Gabriel
that describes, starting at 6:40, what is currently being implemented with OpenZFS, with the list of features worked upon at 8:46. No BPR...
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Thanks solarisguy! Very helpful!
 
Status
Not open for further replies.
Top