Compatibility with Mac ZFS

[jgreco]

Geez write a frickin' book.

I am also with jgreco that I'm not going to discuss this further.

Think I lied because you made a few points I now want to comment on.

4. This thread was about Mac ZFS. I never used it, but people that have talked about it in IRC in the past said it was about the worst piece of software they've ever used.

I don't really trust the judgment of Macheads on this sort of thing, because many of them are non-techie sorts and expect that "quality software" is tied to "having a great GUI" and "easy to use" and all that. I'm guessing that it was much more difficult to use than they were expecting because, well, CLI, ZFS, UNIX, y'know.

I dare you to install FreeBSD and try to setup SSH with keys, a zpool with proper partition alignment and 4k sector size, create some datasets and share them with NFS and Samba with AD integration, make the permissions work properly in NFS and Samba, and then actually mount that stuff elsewhere and use it to transfer files at LAN saturating speeds. If you've never done it, you'd spend weeks just trying to figure out some of the basics. Nevermind try to handle updates to the OS and the whole other host of very real problems and maintenance that you have to do on the server. I've heard people claim that what they can do in 20 minutes in FreeNAS takes them a week or so to truly set up properly in FreeBSD. I'd wager that 99.99% of us in the forum are probably incapable of setting up a FreeBSD server with the same kind of quality that FreeNAS provides if we were given 3 months to do it.

It's actually not quite as difficult as you make it sound. It isn't a beginner-level project, and it can take several hours of work, and the real problem is that stuff like Samba and the AD trusswork usually require some workarounds and debugging to get everything working correctly, and this changes from Samba release to release. The usual lifespan of one of our filers is probably four to six years, though, so it isn't a major investment in time. The question is whether or not it is worth the investment in time to bother, if there's someone already out there focused on doing a good job of maintaining a dedicated filer platform. We still do non-FreeNAS filers for things such as NFS, because FreeNAS has made some engineering choices that fit poorly into our environ, such as ZFS and a massively bloated middleware, which makes the overall resource requirements in a virtual environment truly onerous. I have a difficult time reconciling "setting up a FreeBSD server with the same kind of quality that FreeNAS provides" with the realities here. I can show you a fast, high quality NFS server that exports an encrypted filesystem and remote synchronization capabilities, optimized in a number of ways to work within a virtualization environment, and that can live in 128MB of RAM (probably less). Much less bloat.

The *only* thing that various ZFS implementations have in common is the file system structure itself should be compatible. Not always, but *should* (despite feature flags and everything else). If you read around in FreeBSD there are lots of ex-solaris guys that rebuilt their pools in FreeBSD because they were aware of (and often ran into) problems when going cross-platform.

Ok, so pretend a pool is corrupted that came from Solaris and worked in FreeNAS for 2 months. What went wrong?

And we get to the heart of the matter here, "should" be compatible. But in reality until someone funds a bunch of developers on EACH platform to be experimentally importing pools and doing ongoing validation from all the different platforms, and EACH feature revision of EACH other platform, there's going to be risk. You know how FreeBSD can mount NTFS and read and write that, right? But there are caveats, limitations, things not supported.

I pretty much groaned in despair when ZFS implemented feature flags, because while there are some upsides to that strategy, it also means that there's more potential for a fragmented featureset and the compatibility matrix really becomes very much more complicated than merely comparing a version number. Ultimately you start battling lots of levels of choices made by developers that may or may not be valid on a different platform, and portability becomes more questionable.

In the end, it should be possible to import pools on different platforms, but the safest course of action is to do that only as an interim step while migrating from one platform to another. It is always going to be safest to use a pool natively created on the target platform. Other things could work, in theory, but the number of potential gotchas is too high. Anyone who doesn't like that is of course welcome to fund a bunch of developers to fix that, but we're talking a hell of a lot of work (many man-years, probably).

 

[RoboKaren]

As long as zfs send/receive work across pools, what's the big deal? Just zfs send from the old pool to a new pool and have a happy life.

Or am I missing something critical here?

 

[jgreco]

As long as zfs send/receive work across pools, what's the big deal? Just zfs send from the old pool to a new pool and have a happy life.

Or am I missing something critical here?

Just that people would like it to be MORE compatible than merely that, I think.

I am fine with what you describe though...

 

[Tywin]

As long as zfs send/receive work across pools, what's the big deal? Just zfs send from the old pool to a new pool and have a happy life.

Or am I missing something critical here?

Who says I have two pools?

 

[jgreco]

Who says I have two pools?

I do. Make a new pool. It's the only sane and safe thing to do.

 

[Tywin]

I do. Make a new pool. It's the only sane and safe thing to do.

Sadly, you stating that I have two pools doesn't make it true. If it did, I would ask for a new one each day and have infinite storage :) Here in reality though, with my single pool*, FreeNAS sits in limbo as being a member of OpenZFS and yet having "experts" on the forums claiming it is non-compliant. Chalk up one more in the column for "don't trust FreeNAS".

*To be clear, I have precisely zero pools, partially as a result of conversations like these.

 

[RoboKaren]

*To be clear, I have precisely zero pools, partially as a result of conversations like these.

Uhh... then why are you still here?

Even within the Sun universe, the safest way to transport zfs pools between different zfs systems was to use the zfs send/receive mechanism to transfer data from the old pool to a new pool on the new system. Given that 4tb drives are $130 on NewEgg, I really see no reason that you shouldn't just create a new zfs-2 pool and call it a day.

If you want data portability as your primary goal, you should stick with FAT or ufs.


(I have MacZFS on one of my machine at works. It's a total mess and not recommended. Just waiting for a new dedicated box to get my data off of it)

 

[Tywin]

Uhh... then why are you still here?

Because FreeNAS is the best candidate to become my next storage solution, and I wish to remain current with its functionality and issues. I believe there are areas where FreeNAS is lacking from a systems design perspective, and I am contributing to the community in an attempt to improve it and to make it a product that I would be comfortable running.

 

[jgreco]

Uhh... then why are you still here?

Even within the Sun universe, the safest way to transport zfs pools between different zfs systems was to use the zfs send/receive mechanism to transfer data from the old pool to a new pool on the new system. Given that 4tb drives are $130 on NewEgg, I really see no reason that you shouldn't just create a new zfs-2 pool and call it a day.

Pretty much exactly that. The reality of it will be that until someone once again funds ZFS development with millions of dollars, it is going to be wisest to use a native pool. That's what gets tested and that's what is expected to work. No amount of whining about how things should work changes this. You may indeed be able to port a pool between systems, and it may "work just fine", but this is really in the same class of issues that the ECC/non-ECC argument lives in. We expect that people are storing data with ZFS because they want their data to be safe and secure.

From the FreeNAS appliance point of view, a pool created on Solaris is going to look somewhat different than a pool created on FreeNAS, for example because of the swap partition strategy and device naming strategies. Will the import work? Yeah, it's supposed to and as far as I know it does. But what happens when a disk fails and you need to replace it? Will the FreeNAS GUI handle the Solaris pool design correctly? We don't really know.

We preach what we expect to work. We'll try to help you with what we expect to work. Outside that envelope, you're on your own, and that's a lonely place to be.

 

[Tywin]

Even within the Sun universe, the safest way to transport zfs pools between different zfs systems was to use the zfs send/receive mechanism to transfer data from the old pool to a new pool on the new system. Given that 4tb drives are $130 on NewEgg, I really see no reason that you shouldn't just create a new zfs-2 pool and call it a day.

If you want data portability as your primary goal, you should stick with FAT or ufs.

Data portability is not my primary goal. My primary goal is understanding the limitations and assumptions of FreeNAS that make it incompatible with other OpenZFS implementations. No amount of "just ignore the problem" will help with that understanding. I can't even imagine what system design flaws in FreeNAS might be lurking beneath the waters, but knowing why it is considered not safe to run a MacZFS pool from FreeNAS will help narrow that solution space.

 

[Tywin]

From the FreeNAS appliance point of view, a pool created on Solaris is going to look somewhat different than a pool created on FreeNAS, for example because of the swap partition strategy and device naming strategies. Will the import work? Yeah, it's supposed to and as far as I know it does. But what happens when a disk fails and you need to replace it? Will the FreeNAS GUI handle the Solaris pool design correctly? We don't really know..

That last part is what disquiets me about this and other issues mentioned about FreeNAS. They are avoidable by using Apple's mantra of "don't use it that way", but at least with the iPhone I understand that it was a flaw in the antenna design. I'm not trying to say FreeNAS must support MacZFS or Solaris pools or I'll never use it; I'm saying it's really worth understanding why we shouldn't mix these pool types, instead of just saying "oh geez I dunno....".

Edit: With regard to the ECC comparison, the issues with using non-ECC RAM are well understood and documented. Bit errors in your cache will cause the ZFS pool to believe the data on disk is incorrect, and it will happily go off and re-write that data to "fix" it for you. Incidentally, one way that this could be improved is that if a checksum error is detected in cache, read the data back to a different region and memory and check the checksum again before assuming what's in cache is correct. I am over-simplifying the problem of course, but there are additional mitigation strategies that could be employed that would increase the robustness of FreeNAS to faulty non-ECC RAM without impacting the performance of a nominal system.

 

[jgreco]

Because FreeNAS is the best candidate to become my next storage solution, and I wish to remain current with its functionality and issues. I believe there are areas where FreeNAS is lacking from a systems design perspective, and I am contributing to the community in an attempt to improve it and to make it a product that I would be comfortable running.

Then by all means feel free to point out a design shortcoming. It's designed to be an appliance. When you get a NetApp storage appliance, you cannot move your EMC disks to it. This is an appliance-class product that can be managed by a GUI and middleware. This necessarily precludes certain things, including a number of "clever" ZFS tricks like multiple partitions on a larger disk in order to maximize available space on an array of variously sized disks. The fact that a smart admin can potentially make such tricks work is nice, but making such things work correctly in an appliance through a GUI is basically impossible to do reliably.

 

[jgreco]

That last part is what disquiets me about this and other issues mentioned about FreeNAS. They are avoidable by using Apple's mantra of "don't use it that way", but at least with the iPhone I understand that it was a flaw in the antenna design. I'm not trying to say FreeNAS must support MacZFS or Solaris pools or I'll never use it; I'm saying it's really worth understanding why we shouldn't mix these pool types, instead of just saying "oh geez I dunno....".

Then for chrissakes please reread this thread, because I've explained some very basic issues.

This thread has turned illiterate and I am not interested in re-explaining things over and over. We've given you some hard data on real issues, and there are also unknown potential issues. Use a native pool and it's expected to be rock solid, and fully supported. Importing a foreign pool could be problematic, do it at your own peril. End of frickin' discussion, at least for me.

 

[cyberjock]

I'm locking this thread. Not going to continue with some useless circular argument over this yet again. Go read the other threads about this topic that have taken place over the years too. Nothing has changed and nothing will.

Good day to everyone!