APFS - New Apple Filesystem - Thoughts From a ZFS Developer

[mattbbpl]

Hello everyone,

I ran across the article below this morning, and I wanted to get a discussion going with some of the members here.

http://arstechnica.com/apple/2016/0...-good-and-bad-in-apples-new-apfs-file-system/

In short, Apple is developing a new ZFS-like filesystem, and the article discusses it's intended feature set.

A few key points that sprang to my attention were as follows:

* No RAID as it's intended for single hard drive devices
* Checksums on metadata but not on user data
* Inclusion of fsck
-This portion is really interesting. The ZFS dev said, "WTH is this here?" and the APFS Dev responded, "WTH isn't it there in ZFS?"
* Hubris?
- Explicitly not checksumming user data is a little more interesting. The APFS engineers I talked to cited strong ECC protection within Apple storage devices [implicitly citing that this made checksumming unnecessary]
- Apple engineers I spoke with claimed that bit rot was not a problem for users of their devices
* Scrubs essentially impossible as we think of them
- As data ages you might occasionally want to check for bit rot. Likely fsck_apfs can accomplish this; though as noted there's no data redundancy and no checksums for user data, so scrub would only help to find problems and likely wouldn't help to correct them.

From a ZFS perspective, this looks disappointingly insufficient, but as the article states this is intended for Single Drive devices (Macs to watches).

Are we looking at something useful here, or is it already overshadowed by the likes of ZFS/BTRFS/etc.? Would those be better alternatives, even, in single drive devices?

 

[Bidule0hm]

One more apple' shit technology in response to something that already exists and is better...

 

[philhu]

Apple and Microsoft both do this. They either buy out a product or try to build 'their own version' which is slower, buggier and lacks features.

Why they wouldn't just embrace btrfs and zfs, makes no sense. Alot of wasted developer hours

 

[mattbbpl]

Why they wouldn't just embrace btrfs and zfs, makes no sense.

Could it be a licensing issue? I believe, IIRC jgreco's comments on the matter, that OpenZFS's license is much better for use/modification of open technologies than the typical Linux license, but perhaps there's still something with the license that would cause headache's for Apple?

I'm not really sure what to think on the technical side. Between the comments about the main engineer "not wanting to dig too deep into existing filesystems for fear of being tainted" and the implication in several statements that he didn't seem to really understand what ZFS does in some cases or why it does it, it almost sounds like the thing is only half-researched nevertheless half-baked. That would be pretty shocking for something at this stage of it's lifecycle.

On the other hand, the author also makes it clear that they made some significant ground in other areas in which integrity is not the focal point (such as encryption). I'm skeptical that Apple can trust their hardware as much as the engineer thinks you can, but even so perhaps that's OK in a laptop/phone/tablet/watch world? Maybe Apple's use case is that everything you want saved should be in the cloud thus reducing the need for data integrity.

I'm more of a data integrity/bigger box guy, so maybe I'm just having a hard time wrapping my head around the end goals and my brain is throwing red flags in all the wrong places.

 

[anodos]

Why they wouldn't just embrace btrfs and zfs, makes no sense. Alot of wasted developer hours

Mobile devices with a single storage device are a bit outside of the area where ZFS / btrfs really shine. Apple wants to have a single filesystem that spans their entire product range. I think the only apple product at the moment that would maybe be a good candidate for ZFS is the Mac Pro.

 

[Jailer]

perhaps there's still something with the license that would cause headache's for Apple?

Since when has that ever stopped them?

 

[anodos]

Since when has that ever stopped them?

That part of me that always wants to make snarky comments wonders how much of this boils down to "Not Invented Here".

 

[HoneyBadger]

The Apple engineers contend that Apple devices basically don't return bogus data.

Took a break from reading here to laugh loudly for some time.

Apple engineers I spoke with claimed that bit rot was not a problem for users of their devices, but if your software can't detect errors then you have no idea how your devices really perform in the field.

Pretty sure this what the young people these days call a "mic drop" moment.

 

[Bidule0hm]

Apple and Microsoft both do this. They either buy out a product or try to build 'their own version' which is slower, buggier and lacks features.

When you see what they've done to the poor FreeBSD... they took the most stable OS I know and modified it to the point it's the most unstable OS I know (and yes, it's even more unstable than Windows, and a few months ago I'd say the contrary...).

Mobile devices with a single storage device are a bit outside of the area where ZFS / btrfs really shine. Apple wants to have a single filesystem that spans their entire product range. I think the only apple product at the moment that would maybe be a good candidate for ZFS is the Mac Pro.

As it doesn't checksum the user data then it's pretty useless, it's just a bit better than any other current FS.

Apple engineers I spoke with claimed that bit rot was not a problem for users of their devices

Lawl; you must be kidding me...

 

[Ericloewe]

Took a break from reading here to laugh loudly for some time.

They'd just stare at you blankly and offer to adjust your iPhone to share the great joke you just read. Or they might use the "Call conversion team" app on their iPads to dispatch a Customer Education Team to properly assimilate you.

The reality distortion field is strong.

The really big missed opportunity here is a filesystem designed for flash memory. Apple is the only company realistically capable of embarking on such an endeavor. I'm kinda glad they didn't, otherwise we'd end up with another Apple "standard".

Apple and Microsoft both do this. They either buy out a product or try to build 'their own version' which is slower, buggier and lacks features.

Why they wouldn't just embrace btrfs and zfs, makes no sense. Alot of wasted developer hours

Well, anyone with half a brain in a company like Microsoft would be seriously advised by legal to not even look at anything vaguely related to Oracle. btrfs' license is outright incompatible with Windows, by design. So Microsoft needs something in-house.

Though it may seem kind of wasteful, I don't mind at all if they document it properly and allow people to use it. Yeah, very big "if" there, but one can hope.

What's interesting is that Microsoft would only need a ZFS-grade filesystem, with all that entails. NTFS is fine for phones and smaller, though checksums would be nice, so they can stick with NTFS for smaller devices and ReFS for the big stuff.

 

[HoneyBadger]

The reality distortion field is strong.

Apple: It Just Works, Until It Doesn't, Then IDK WTF LOL buy a new one for only $1599?

The really big missed opportunity here is a filesystem designed for flash memory.

F2FS is a thing for Linux, but it's a Samsung product and they likely won't be sharing that too widely or freely.

Random thought, if you say "HFS" as a word rather than an acronym, it sounds like "half-ass."

 

[anodos]

Apple: It Just Works, Until It Doesn't, Then IDK WTF LOL buy a new one for only $1599?



F2FS is a thing for Linux, but it's a Samsung product and they likely won't be sharing that too widely or freely.

Random thought, if you say "HFS" as a word rather than an acronym, it sounds like "half-ass."

So that means HFS2 is "half-ass too"?

 

[Borja Marcos]

In short, Apple is developing a new ZFS-like filesystem, and the article discusses it's intended feature set.

Going to play Devil's Advocate here :)

* No RAID as it's intended for single hard drive devices

It's not a bad decision. Do you have redundant storage in your laptop? Phone?

* Checksums on metadata but not on user data

This one sounds controversial, but give it a thought or two.

ZFS checksum is necessary and good because of two known limitations of traditional storage:

1. Traditional hard disks have a simple error correction for disk blocks
2. Traditional storage uses complex and error prone interfaces and cabling (ATA, SCSI, SATA, SAS...)

ZFS checksum is useful when redundancy allows you to heal the corrupted data. With no redundancy the system can warn you about
data corruption. But the more elaborate error detection/correction methods in flash drives should detect that as well.

They aren't using ECC memory in their systems, except for the Mac Pros.

* Inclusion of fsck
-This portion is really interesting. The ZFS dev said, "WTH is this here?" and the APFS Dev responded, "WTH isn't it there in ZFS?"

Someone could argue that a scrub is a fsck of sorts. You can heat the argument all you want, but it's not without merit.

Are we looking at something useful here, or is it already overshadowed by the likes of ZFS/BTRFS/etc.? Would those be better alternatives, even, in single drive devices?

From what I've read so far (which is the same you have read) I think they're doing a great job. ZFS is a behemoth. An awesome behemoth (I can't live without it since 2007!) but still a behemoth.