Corrupted or truncated afp extended attributes (netatalk)

[giox069]

Hi, I'm using FreeNAS 11.3-U5 to serve 5TB of data in a couple of millions of files via afp (netatalk) to some MacOS.

I'm seeing a strange problem in a subset of 2000 files: extended attributes cannot be read or appears to be corrupted when seen from the MacOS (client) side.

Take the following two examples:

On the server side, FreeNAS everithing seems to be correct:

Code:

root@st-graf:~ # lsextattr user "Zhopper 2020_1_3.tif"
Zhopper 2020_1_3.tif   org.netatalk.Metadata   com.apple.metadata:_kMDItemUserTags
root@st-graf:~ # lsextattr user "ZAMAHA_481.pdf"
ZAMAHA_481.pdf       org.netatalk.Metadata   com.apple.lastuseddate#PS

On the client side (MacOS, recently updated, connected via afp):

Code:

MY-MAC-MINI:tmp user1$ xattr Zhopper 2020_1_3.tif
xattr: [Errno 22] Invalid argument: 'Zhopper 2020_1_3.tif'
MY-MAC-MINI:tmp user1$ xattr "ZAMAHA_481.pdf"
com.apple.FinderInfo
o
g.netatalk.Metadata
com.apple.lastuseddate#PS

As you can see, there are two problems: the first file (Zhopper 2020_1_3.tif) exists, but the client cannot read xattrs while FreeNAS server can.
In the second case (ZAMAHA_481.pdf), a random character in the attribute name read by the Mac is changed with a newline or end of string, cutting "org.netatalk.Metadata" into "o" and "g.netatalk.Metatata". The exact position of this truncation is different for different files. But the position of truncation is fixed for a single file, even after umount and remount.

sometimes, suddenly, it starts or stop working (usually it starts or stop working after some minutes from mount):

Code:

MY-MAC-MINI:tmp user1$ xattr "ZAMAHA_481.pdf"
com.apple.FinderInfo
com.apple.lastuseddate#PS

The problem only appears for a subset of files, and this subset does not change.

Randomly removing EA from the FreeNAS file with "rmextattr user ..." make the problem disappear for that file forever. But I can't randomly remove EA from the fileserver.. . users will complain a lot ;)

Any ideas? I Would like to get correct EAs from this afp connection.

 

[anodos]

Hi, I'm using FreeNAS 11.3-U5 to serve 5TB of data in a couple of millions of files via afp (netatalk) to some MacOS.

I'm seeing a strange problem in a subset of 2000 files: extended attributes cannot be read or appears to be corrupted when seen from the MacOS (client) side.

Thake the following two examples:

On the server side, FreeNAS everithing seems to be correct:

Code:

root@st-graf:~ # lsextattr user "Zhopper 2020_1_3.tif"
Zhopper 2020_1_3.tif   org.netatalk.Metadata   com.apple.metadata:_kMDItemUserTags
root@st-graf:~ # lsextattr user "ZAMAHA_481.pdf"
ZAMAHA_481.pdf       org.netatalk.Metadata   com.apple.lastuseddate#PS

On the client side (MacOS, recently updated, connected via afp):

Code:

MY-MAC-MINI:tmp user1$ xattr Zhopper 2020_1_3.tif
xattr: [Errno 22] Invalid argument: 'Zhopper 2020_1_3.tif'
MY-MAC-MINI:tmp user1$ xattr "ZAMAHA_481.pdf"
com.apple.FinderInfo
o
g.netatalk.Metadata
com.apple.lastuseddate#PS

As you can see, there are two problems: the first file (Zhopper 2020_1_3.tif) exists, but the client cannot read xattrs while FreeNAS server can.
In the second case (ZAMAHA_481.pdf), a random character in the attribute name read by the Mac is changed with a newline or end of string, cutting "org.netatalk.Metadata" into "o" and "g.netatalk.Metatata". The exact position of this truncation is different for different files. But the position of truncation it's fixed for a single file, even after umount and remount.

sometimes, suddenly, it starts or stop working (usually it starts or stop working after some minutes from mount):

Code:

MY-MAC-MINI:tmp user1$ xattr "ZAMAHA_481.pdf"
com.apple.FinderInfo
com.apple.lastuseddate#PS

The problem only appears for a subset of files, and this subset does not change.

Randomly removing EA from the FreeNAS file with "rmextattr user ..." make the problem disappear for that file forever. But I can't randomly remove EA from the fileserver.. . users will complain a lot ;)

Any ideas? I Would like to get correct EAs from this afp connection.

Is this also an SMB share?

 

[giox069]

No, this is an afp only share. I have the same problem in other afp shares, and one share which is both afp and smb. But this share is afp only.

I also ran dbd -f /topdir (it took more than 8 hours to complete), but nothing changed.

 

[anodos]

Can you generate a pcap of the client's traffic (try to limit to only AFP protocol from this particular client -- no other traffic to the NAS from this client). tcpdump -i <interface> -w /tmp/afp.pcap host <ip of Mac>

Then send it to me via private message.

 

[giox069]

Thank you for the suggestion. I can send you the full .cap file via private message if you need.

In any case, see in the following screenshot what I discovered capturing on freenas and then opening with wireshark:

1st xattr request to .pdf file

And then, after a stupid ls on topdir, it starts working. This is the same xattr command as the 1st attempt, but now works.

I suppose org.netatalk.Metadata is hidden by netatalk and not returned on the wire to MacOS when recognized, right?

 

[anodos]

I do not believe org.netatalk.Metadata should be visible when listing xattrs. It's an internal xattr used by netatalk to store certain AFP (FinderInfo, AFP Flags, Comment, CNID). Do the affected files perhaps have an appledouble file associated with them (as well as the xattr)?

 

[giox069]

No appledouble file, only a .DS_Store file si not a standard file in that folder.

 

[anodos]

If you can get a time when server is not busy (end of day perhaps), try setting "log level = maxdebug" and reproduce the issue, then send me the afp.log. This should show how the xattr list is passing through netatalk's vfs.