Slow directory listing and hickups

[no_connection]

Fine.

Why does dos attributes = no change the performance of ZFS then? As obviously ZFS is to blame here as nothing else can be wrong.

 

[cyberjock]

Uhh.. dos attributes = no applies to Samba, not to ZFS. ZFS will be impacted, but not significantly.

 

[no_connection]

Then why does the system go from usable to unusable with a huge impact?

 

[cyberjock]

I really am done with this conversation after this... ok? Do not expect another response from me as I have already explained that you need more RAM and then invalidated your own "explanation" for why you think I'm wrong.

Once you run out of resources for ZFS, changing things can have catastrophic consequences for ZFS. It's like the cliff I keep telling everyone about. Once you go off the cliff it's too late. You end up at the bottom and there's no way to get back up. I've seen this with a bunch of customers and the fix is either serious ZFS tuning (assuming it can be tuned enough for your limited resources) or add more resources.

Good luck!

Edit: For the record I'm getting out of this conversation because I have been in enough of these conversations to know that I'm not interested in spending 2 pages of back and forth trying to explain it. I've explained it many times and the forum search is there if you'd like to avail yourself of my prior posts.

 

[no_connection]

Alright, I will try to test more RAM if I can get any working in stock.

Problem is that if it does indeed solve the problem, the current implementation is inferior to what it was years ago and points to a serious design flaw.
I have a hard time to accept that ZFS would be designed and implemented this poorly, especially as I have used earlier version with great result.

So excuse me for believing there is something wrong rather than acceptable behavior.
The amount if successful users with less RAM and hardware does not help ether.

 

[no_connection]

Thought I should update my experiences a bit.

I tried 28GB RAM for a short while when I got the parts for my ESXi box. System was over all more snappy, but the initial problem of loading directories did not get completely resolved. A bit better yes but not the level of performance you could expect.

I have now been running 16GB RAM for quite a while and it have worked OK. Directories now get cached enough to still work fine most of the time. But the initial problem still remains.

So while adding RAM was beneficial, and I can now run the system without "dos attributes = no" there still remains some hiccups. And if I stream a movie or audio and browse a directory, the stream gets interrupted until it is loaded. Which might be 10 sec.
Once a large directory is loaded it is snappy to access as long as it remains in cache (it seems).

So I still say there is a SMB(or other) problem, even if you won't believe me.

 

[anodos]

Thought I should update my experiences a bit.

I tried 28GB RAM for a short while when I got the parts for my ESXi box. System was over all more snappy, but the initial problem of loading directories did not get completely resolved. A bit better yes but not the level of performance you could expect.

I have now been running 16GB RAM for quite a while and it have worked OK. Directories now get cached enough to still work fine most of the time. But the initial problem still remains.

So while adding RAM was beneficial, and I can now run the system without "dos attributes = no" there still remains some hiccups. And if I stream a movie or audio and browse a directory, the stream gets interrupted until it is loaded. Which might be 10 sec.
Once a large directory is loaded it is snappy to access as long as it remains in cache (it seems).

So I still say there is a SMB(or other) problem, even if you won't believe me.

The problem is more specifically related to how samba stores DOS modes ( Hidden | System | Read-Only | Archive ). The older default method of storing this information was by manipulating the execute bit of Unix (Owner | Group | Other) using the smb.conf parameters "map hidden | map system | map readonly | map archive". Unfortunately, these parameters do not work properly with "nt acl support" enabled, which is a requirement for properly mapping ACLs with the zfsacl VFS module (setting map* parameters causes ACLs to start going bonkers).

To get around these problems the samba team has transitioned to storing DOS modes as file system extended attributes (as well as other information such as NTFS alternate data streams). Unfortunately, for some reason xattr lookups under FreeBSD appear to be slow. You will have to determine for yourself whether you can safely disable DOS modes (some software requires setting and unsetting the 'archive' bit). If you decide you want to disable them, then you need to add the following auxiliary parameters to your global CIFS config:

• store dos attributes = no
• ea support = no
• map hidden = no
• map system = no
• map readonly = no
• map archive = no

If you don't turn off the 'map *' parameters then you may encounter permissions-related oddness. This means you will not be able to flag files or directories as " hidden | system | readonly | archive"

 

[no_connection]

Thanks, for the info.
I'm glad that it works adequate without resorting to disabling anything now. Although I am only using guest access fairly simple shares, so I probably won't need the above attributes.
Maybe readonly, but I have that set to shares when needed.

Still odd that many systems and users seems unaffected, storing files and folders is a very common use for a NAS after all.

 

[anodos]

Thanks, for the info.
I'm glad that it works adequate without resorting to disabling anything now. Although I am only using guest access fairly simple shares, so I probably won't need the above attributes.
Maybe readonly, but I have that set to shares when needed.

Still odd that many systems and users seems unaffected, storing files and folders is a very common use for a NAS after all.

The parameters I mentioned before are 'global' parameters. This means all your shares have them or none of your shares have them. Samba only reads the user.DOSATTRIB extended attribute of files in the folder where you are located as well as the current path in the file tree. Less files in folder or simpler file tree eliminates the problem in many cases. More ARC also improves ZFS's ability to cache the extended attributes.

That being said, there are many issues that combine together to make browsing slow. DOS modes are only part of the equation (in some cases it may be the primary cause). Using SMB2 instead of SMB1 can improve performance (SMB2 is much more efficient than SMB1 - if I remember correctly smbclient defaults to SMB1). Disabling thumbnail generation can help. Changing antivirus solution can help.

If you're using some sort of linux box to watch movies over CIFS / browse your file tree, you can modify your client's (not your FreeNAS server) smb.conf file by adding the following global parameter:
client max protocol = SMB3

This requires that your client have a minimum of samba 4.1.0.

You may experience some bugs as unix extensions and a few other samba features are not presently supported under smbclient.

 

[no_connection]

Client is win7 for the most part. AV(also on AV free machine) and thumbnails(don't use them) makes no difference.
I usually use total commander but explorer is slow too.
I checked with wireshark and SMB2 is used.

My only form of access control is read only shares and normal shares, with some hidden shares as well(don't really need more for my use). So I probably won't need the advanced stuff right?

And just to rule out any in-optimal configurations.
What should the dataset and share options be for my above usage?
Oh, I also need to be able to always execute file from the NAS as I use it for several portable apps (had a bit of a nightmare with that when I first set up the server)

 

[anodos]

Client is win7 for the most part. AV(also on AV free machine) and thumbnails(don't use them) makes no difference.
I usually use total commander but explorer is slow too.
I checked with wireshark and SMB2 is used.

My only form of access control is read only shares and normal shares, with some hidden shares as well(don't really need more for my use). So I probably won't need the advanced stuff right?

And just to rule out any in-optimal configurations.
What should the dataset and share options be for my above usage?
Oh, I also need to be able to always execute file from the NAS as I use it for several portable apps (had a bit of a nightmare with that when I first set up the server)

How many clients do you have? What type of network (wired or wireless, 10/100 or gigabit, real switch or ISP provided crap)? Is FreeNAS virtualized?

Changing the "store dos attributes" and associated parameters I mentioned above should not affect access to your shares. Access to shares is controlled by either (1) share definition parameters or (2) access control entries.

Post your smb4.conf file (/usr/local/etc/smb4.conf) enclosed in code tags.

 

[no_connection]

Dell T20 16GB 4x WD RED 4TB Z2 (bare metal)
Switch is HP 1810-8G (gigabit) Neat little bugger and you can turn of LEDs if you want.
I use netgear GB switches between my normal workstation, but it happens with ESXi VMs as well and that is on HP switch. It's a T20 too with 16GB.
*edit single human user for the most part. Although I run a few VMs that are connected to the NAS and uses it.
*edit2 share type for dataset is windows and atime is off.

I used FileZilla and SSH to get the file:

Code:

[global]
    server max protocol = SMB3
    encrypt passwords = yes
    dns proxy = no
    strict locking = no
    oplocks = yes
    deadtime = 15
    max log size = 51200
    max open files = 11070
    load printers = no
    printing = bsd
    printcap name = /dev/null
    disable spoolss = yes
    getwd cache = yes
    guest account = nobody
    map to guest = Bad User
    obey pam restrictions = Yes
    directory name cache size = 0
    kernel change notify = no
    panic action = /usr/local/libexec/samba/samba-backtrace
    server string = FreeNAS Server
    ea support = yes
    store dos attributes = yes
    null passwords = yes
    acl allow execute always = true
    local master = yes
    idmap config *:backend = tdb
    idmap config *:range = 90000000-100000000
    server role = standalone
    netbios name = CHRYSALIS
    workgroup = MSHOME
    security = user
    pid directory = /var/run/samba
    smb passwd file = /var/etc/private/smbpasswd
    private dir = /var/etc/private
    create mask = 0666
    directory mask = 0777
    client ntlmv2 auth = yes
    dos charset = CP437
    unix charset = UTF-8
    log level = 1
    

[MGMNT]
    path = /mnt/Chry8TB/CIFSshare
    printable = no
    veto files = /.snap/.windows/.zfs/
    comment = management
    writeable = yes
    browseable = no
    recycle:repository = .recycle/%U
    recycle:keeptree = yes
    recycle:versions = yes
    recycle:touch = yes
    recycle:directory_mode = 0777
    recycle:subdir_mode = 0700
    vfs objects = recycle zfsacl streams_xattr aio_pthread
    hide dot files = yes
    guest ok = yes
    nfs4:mode = special
    nfs4:acedup = merge
    nfs4:chown = yes
    zfsacl:acesort = dontcare
    

[RO]
    path = /mnt/Chry8TB/CIFSshare
    printable = no
    veto files = /.snap/.windows/.zfs/
    comment = read only
    writeable = no
    browseable = no
    recycle:repository = .recycle/%U
    recycle:keeptree = yes
    recycle:versions = yes
    recycle:touch = yes
    recycle:directory_mode = 0777
    recycle:subdir_mode = 0700
    vfs objects = zfsacl streams_xattr aio_pthread
    hide dot files = yes
    guest ok = yes
    guest only = yes
    nfs4:mode = special
    nfs4:acedup = merge
    nfs4:chown = yes
    zfsacl:acesort = dontcare
    

[Video]
    path = /mnt/Chry8TB/CIFSshare/Video
    printable = no
    veto files = /.snap/.windows/.zfs/
    comment = Video_Chrysalis
    writeable = no
    browseable = yes
    recycle:repository = .recycle/%U
    recycle:keeptree = yes
    recycle:versions = yes
    recycle:touch = yes
    recycle:directory_mode = 0777
    recycle:subdir_mode = 0700
    vfs objects = zfsacl streams_xattr aio_pthread
    hide dot files = yes
    guest ok = yes
    guest only = yes
    nfs4:mode = special
    nfs4:acedup = merge
    nfs4:chown = yes
    zfsacl:acesort = dontcare 

 

[anodos]

Well, it looks like you haven't really departed much from the defaults. You can try setting the following parameters under "Services" -> "CIFS" -> "Auxiliary Parameters"

Code:

store dos attributes = no
ea support = no
map hidden = no
map system = no
map readonly = no
map archive = no

You should be able to safely enable and disable these parameters without affecting your data. If the problem still presents after setting things as I listed above, then you will need to do quite a bit more rigorous testing.

 

[no_connection]

Works great now. Thanks!

I hope a solution can be found to have proper attributes while keeping performance.
At least for me I can use this workaround being a fairly simple infrastructure.

I would not mind a special cache residing on a SSD to handle this for example. (as L2ARC is not an option)

 

[anodos]

Works great now. Thanks!

I hope a solution can be found to have proper attributes while keeping performance.
At least for me I can use this workaround being a fairly simple infrastructure.

I would not mind a special cache residing on a SSD to handle this for example. (as L2ARC is not an option)

I believe there is some sort of fix or work-around in progress for this issue. An L2ARC may be able to fix the performance issue (once it's warmed up), but adding one with only 16GB RAM may hurt your overall performance. I don't think you can create a "special cache SSD" for extended attributes.

 

[AlainD]

I believe there is some sort of fix or work-around in progress for this issue. An L2ARC may be able to fix the performance issue (once it's warmed up), but adding one with only 16GB RAM may hurt your overall performance. I don't think you can create a "special cache SSD" for extended attributes.

Any info on the fix (timeframe?)?

 

[anodos]

Any info on the fix (timeframe?)?

I am not an insider. I assume a fix or workaround is not a simple matter since it may be an inefficiency with FreeBSD / zfs / extended attributes.

I know on Linux they worked around the xattr problem by allowing sa based xattrs, but the problem was worse on linux because xattrs were being written to a tdb file (I think). If that's the solution I certainly wouldn't hold my breath for a quick fix.

If you are experiencing this problem and need dos attributes, then do the following
1) max out RAM (try to get at least 48gb)
2) add l2arc once you've done (1) and if problem persists.

Alternatively, if you have a test machine you can see if it's fixed in the 9.3 beta.

 

[no_connection]

I guess that users like me don't really need much file cache from ARC at all. So it would be nice to be able to dedicate 50% or 75% of RAM to metadata (default is 25%?).
So if that is possible then that might be a usable fix.

 

[anodos]

I guess that users like me don't really need much file cache from ARC at all. So it would be nice to be able to dedicate 50% or 75% of RAM to metadata (default is 25%?).
So if that is possible then that might be a usable fix.

Whether it's usable depends on how much metadata needs to be stored in ARC. I've seen performance still suck on a system with large amounts of RAM and "vfs.zfs.arc_meta_limit" set to 50% of the total RAM. You can experiment with the aforementioned tunable, but I'd probably not go above 50% on your system. The annoying part of this problem is that the lookups are slow, which means you will see performance tank once you can't cache the extended attributes.

FreeBSD 10 has made some major changes in how locking is handled by the kernel, which may eliminate the problem (assuming it is caused by lock contention), but unfortunately FreNAS 10 is quite a long ways off.