SMBD process goes 100% and copy speed collapse, ARC request demand metadata issue ?

[Nicolas_Studiokgb]

Samba 4.18 isn't even in nightlies for core yet (and won't be until we branch out for 13.1. I'd have to build a custom ISO for you, which would distract from what work can be put into 13.0-U5 to help out. FWIW, I have another round of improvements that appear to reliably give 20% speedup in my test cases for a "plain" Windows-style directory listing (which are part of what Macs do). Will try to get that into U5 before code freeze.

Hello @Anados
Good news to read
Don't worry I didn't mean having some custom beta. Everything will come in time.
Let me know if you want me to send you some sample files to check if there is anything different in those that could impact performance.

 

[anodos]

TrueNAS-13.0-CUSTOM_RESOLVE_NO_SYMLINK_10-manual-update-unsigned.tar

drive.google.com

This is current state of 13.0-stable (nighlies) with support for additional parameter ixnas:dirent_optimization = true at share level. I'll take another look at maybe caching fdhandle structs in an RB tree to make cost of repeated dir listing by clients less so perhaps one more custom build.

 

[Nicolas_Studiokgb]

TrueNAS-13.0-CUSTOM_RESOLVE_NO_SYMLINK_10-manual-update-unsigned.tar

drive.google.com

This is current state of 13.0-stable (nighlies) with support for additional parameter ixnas:dirent_optimization = true at share level. I'll take another look at maybe caching fdhandle structs in an RB tree to make cost of repeated dir listing by clients less so perhaps one more custom build.

Hello @anodos

I test asap and I tell you
Thanks :)

 

[simonj]

Hi @anodos, hi @Nicolas_Studiokgb

I'm super excited about this thread as we are plagued by the same problems.

I did some semi-systematic testing also trying the custom build.

I started testing with a share that is in real-world use. Halfway through I realized that as it is a mixed AFP*/SMB share the parameters that are automatically set (eg. fruit:metadata = netatalk) are quite different from a freshly created SMB default.

Here is a WiP document with the results:

tests_SMB.docx

8horses Cloud

cloud.8horses.ch

Findings:​

Alternate data streams = on. Having them off leads to problems and performance degradation
Apple Extensions = off - seems to always improve performance. Most dramatically on mixed SMB/AFP shares, only slightly on SMB only shares
Is there any clear disadvantage of keeping them off?
Directory listing was always near instant in the systematic tests regardless of settings. This might have to do with the data just having been copied and the metadata being in ARC
Deleting the directories - Apple extensions = off helped mostly on the mixed AFP/SMB share and gave a slight improvement on fresh SMB only share
.DS stores = off - seems to improve performance very slightly
fruit:streamname_optimization = true - no clear improvement but also no deterioration (maybe not tested systematically enough)
Installing Custom Build & setting ixnas:dirent_optimization = true - seems to not directly improve performance. But gets rid of the problem of single cores going to 100% and locking up the connection between Mac and server!!

Other non-systematic tests with existing directories:​

Custom Build installed & setting ixnas:dirent_optimization = true
Listing of directories with many files (~150K) that are not in ARC is still taking very long: 33 minutes in Finder, 18 minutes in Terminal
Here again I didn't see the problem of the smbd process going to 100% anymore and the connection between Mac and server didn't lock up and lead to spinning wheel.
Listing of a similar directory served from a Mac over SMB takes only 5 minutes

Best practice / conclusions:​

Apple extensions = off
Alternate data streams = on!!
Try to avoid having mixed AFP/SMB shares
Write .DS stores to network shares = off ("defaults write com.apple.desktopservices DSDontWriteNetworkStores -bool TRUE")

Listing directories that are on a TrueNAS server still takes around 6 times longer than listing similar directories that are served from a Mac. From my memory the crappy QNAP server that we had before was also fast with listing directories over SMB.

IMO this should be the main area of research. If slow directory listing could be brought to similar performance like with Mac servers and others TrueNAS would be the perfect system for professional media workflows.

*AFP we have another problem when first accessing directories with many files (loading them into the media pool) with Davinci Resolve, the server disconnects with some symlink error in the logs. Workaround is to connect over slow but reliable AFP for those operations.

 

[Nicolas_Studiokgb]

Hi @anodos , @simonj

I've tested this morning. First tests with default settings on the mac (no nsmb.conf and leaving the usage of DS store over network enabled)
the new version on Truenas and
fruit:streamname_optimization=true ixnas:dirent_optimization = true
107GB 16k items
I get almost the same performance on copying (1) and browsing (2) (folder open) 5min to copy, 2/3 second to open a folder between 2500and 8000 items.
Delete (3) is closer to 5 minutes instead of 6 so it's better.
Smbd processes still goes to over 80% in spikes during delete

@simonj
Disabling DS stores make browsing very long. It's why I let it on. The price is deleting is longer (I have to test with the last version sent by anodos if it change anything)

I ll do some more test asap

 

[simonj]

Hi @anodos , @simonj

I've tested this morning. First tests with default settings on the mac (no nsmb.conf and leaving the usage of DS store over network enabled)
the new version on Truenas and
fruit:streamname_optimization=true ixnas:dirent_optimization = true
107GB 16k items
I get almost the same performance on copying (1) and browsing (2) (folder open) 5min to copy, 2/3 second to open a folder between 2500and 8000 items.
Delete (3) is closer to 5 minutes instead of 6 so it's better.
Smbd processes still goes to over 80% in spikes during delete
View attachment 65950
View attachment 65951

@simonj
Disabling DS stores make browsing very long. It's why I let it on. The price is deleting is longer (I have to test with the last version sent by anodos if it change anything)

I ll do some more test asap

Disabling .ds store was only a minimal or even random improvement for me.
But did you try disabling the Apple Extensions?

 

[Nicolas_Studiokgb]

I ve tried a few weeks ago but not recently.
I'm not sure it's a good idea for usage with Macs even if it give perf improvements. @anodos, do you have idea about enabling or disabling Apple extensions?

 

[simonj]

Good Morning

Just did a few more tests:

I launched a Ubuntu VM on our second TrueNAS Scale server
- Listing a folder with 150K files on the Ubuntu VM from Mac client over SMB takes around 38 minutes - same problem like with TrueNAS smb shares
- Listing a folder with 150K files on TrueNAS core server from Ubuntu VM over SMB takes around 3 minutes only!

File operations with directories containing ten-thousands of files:
Mac Client > Mac Server : Good
Mac Client > Customized SMB Server (like QNAP) : Good (by memory. Not recently tested)
Mac Client > TrueNAS Server : Barely usable
Mac Client > Generic Ubuntu SMB Server : Barely usable
Linux Client > TrueNAS Server : Good

I still suspect that Mac Os is spamming the SMB server with metadata requests that the server cannot handle.
This seems to be a problem unique to Mac clients and non-optimized SMB servers.
I was concentrating on directory listing but I suspect that the same problem also applies to bad performance in deleting and copying many files.

It would be extremely great if this issue was investigated and eventually solved.

 

[anodos]

This is latest iteration of changes in 4.15 (based on 13.0 master). Same instructions / caveats as before. Parameter to check out is ixnas:dirent_optimization = true on share.

Flamegraphs show something much more reasonable, but still not quite up to where Samba 4.18 is.

Edit: removed link due to issue found in testing.

Looks like we're making another namei round-trip due to requests to list xattrs. Will optmize that away and spin new build.

 

[awasb]

Just out of curiosity: How exactly did you call dtrace? Something like this?

Code:

$ dtrace -x ustackframes=100 -n 'profile-99 /execname == "smbd" && arg1/ {@[ustack()] = count(); } tick-60s { exit(0); }' -o out.stacks
$ ./stackcollapse.pl out.stacks > out.folded
$ ./flamegraph.pl out.folded > out.svg

 

[anodos]

Just out of curiosity: How exactly did you call dtrace? Something like this?

Code:

$ dtrace -x ustackframes=100 -n 'profile-99 /execname == "smbd" && arg1/ {@[ustack()] = count(); } tick-60s { exit(0); }' -o out.stacks
$ ./stackcollapse.pl out.stacks > out.folded
$ ./flamegraph.pl out.folded > out.svg

I used pmcstat to get data for flamegraph. I'm still looking at MacOS client. It's just pathologically bad at listing directories regardless of server. Against a Windows server with 200K files in a dir it's around 50x slower than other clients because it spins on repeatedly asking server to rescan dir from beginning. I'm having to adjust my strategy for dealing with clients to assume that the client may repeatedly rescan same directory (perhaps 100 times) while generating a listing.

 

[awasb]

Maybe it's a problem with dir notify and/or dir caching. Is there any "actual" (open) source doc available? Is

Source Browser

still valid?

 

[anodos]

Maybe it's a problem with dir notify and/or dir caching. Is there any "actual" (open) source doc available? Is

Source Browser

still valid?

That's the latest MacOS source available, but it's out of date. I generally keep an old laptop on roughly same version of their SMB client with SIP disabled so that I can dtrace SMB client, but I'm not there yet for this. I just need to do what I can to make opening files listing dirs over SMB faster since I can't fix the client.

 

[awasb]

Sure.

And thanks for the hint. I'm just trying to understand (better) what's going on and what option does _exactly_ what.

 

[anodos]

Sure.

And thanks for the hint. I'm just trying to understand (better) what's going on and what option does _exactly_ what.

I think most nsmb.conf stuff people write about is mostly voodoo.

Here is some sample dtrace output on a MacOS Big Sur client while listing dir with 200K files on Windows.

Code:

dtrace: script '/Users/awalker/smb.dt' matched 193 probes
^C


              smbfs`smb2fs_smb_cmpd_query_dir_one+0x247
              smbfs`smbfs_lookup+0x1d7
              smbfs`smbfs_vnop_lookup+0x5a3
              kernel`lookup+0x36e
              kernel`namei+0x1124
              kernel`0xffffff800d528b80+0x79
              kernel`0xffffff800d52ec40+0xfe
              kernel`fstatat64+0x6c
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_smb_get_parent                                              1

              smbfs`smbfs_getattr+0x88
              kernel`vnode_getattr+0xae
              kernel`vn_stat_noauth+0xdf
              kernel`vn_stat+0xaf
              kernel`0xffffff800d52ec40+0x265
              kernel`fstatat64+0x6c
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_update_cache                                                1

              smbfs`smb_fphelp+0x1f5
              smbfs`smb2_smb_create+0x322
              smbfs`smb2fs_smb_ntcreatex+0x236
              smbfs`smbfs_smb_findnext+0x3e0
              smbfs`smbfs_findnext+0x5f
              smbfs`smbfs_fetch_new_entries+0x409
              smbfs`smbfs_vnop_getattrlistbulk+0x1396
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_smb_get_parent                                              1

              smbfs`smbfs_update_cache+0xa0
              smbfs`smbfs_vnop_getattr+0x1d3
              kernel`vnode_getattr+0xae
              kernel`vn_stat_noauth+0xdf
              kernel`vn_stat+0xaf
              kernel`0xffffff800d52ec40+0x265
              kernel`fstatat64+0x6c
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_attr_cachelookup                                            1

              smbfs`smb2fs_smb_cmpd_query_dir_one+0x98f
              smbfs`smbfs_lookup+0x1d7
              smbfs`smbfs_vnop_lookup+0x5a3
              kernel`lookup+0x36e
              kernel`namei+0x1124
              kernel`0xffffff800d528b80+0x79
              kernel`0xffffff800d52ec40+0xfe
              kernel`fstatat64+0x6c
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_ntwrkname_tolocal                                           1

              smbfs`smbfs_vnop_lookup+0x5a3
              kernel`namei+0x1124
              kernel`0xffffff800d528b80+0x79
              kernel`0xffffff800d52ec40+0xfe
              kernel`fstatat64+0x6c
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_lookup                                                      1

              kernel`0xffffff800d51c4f0+0x667
              kernel`lookup+0x2a0
              kernel`namei+0x1124
              kernel`0xffffff800d528b80+0x79
              kernel`0xffffff800d52ec40+0xfe
              kernel`fstatat64+0x6c
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_vnop_access                                                 1

              kernel`0xffffff800d51c4f0+0x667
              kernel`vn_stat+0x71
              kernel`0xffffff800d52ec40+0x265
              kernel`fstatat64+0x6c
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_vnop_access                                                 1

              kernel`0xffffff800d51c4f0+0x667
              kernel`0xffffff800d528d20+0x1ba
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_vnop_access                                                 1

              kernel`lookup+0x36e
              kernel`0xffffff800d528b80+0x79
              kernel`0xffffff800d52ec40+0xfe
              kernel`fstatat64+0x6c
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_vnop_lookup                                                 1

              smbfs`smbfs_lookup+0x1d7
              kernel`lookup+0x36e
              kernel`namei+0x1124
              kernel`0xffffff800d528b80+0x79
              kernel`0xffffff800d52ec40+0xfe
              kernel`fstatat64+0x6c
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_lookup                                                      1

              smbfs`smbfs_vnop_close+0xee
              kernel`0xffffff800d542b80+0x102
              kernel`0xffffff800d7e9590+0x13e
              kernel`0xffffff800d7eae70+0x53e
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_closedirlookup                                              1

              smbfs`smb2_smb_parse_create+0xacc
              smbfs`smbfs_findnext+0x5f
              smbfs`smbfs_fetch_new_entries+0x409
              smbfs`smbfs_vnop_getattrlistbulk+0x1396
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_get_epoch_delta                                             1

              smbfs`smbfs_vnop_getattr+0x1d3
              kernel`vn_stat_noauth+0xdf
              kernel`vn_stat+0xaf
              kernel`0xffffff800d52ec40+0x265
              kernel`fstatat64+0x6c
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_getattr                                                     1

              kernel`vnode_getattr+0xae
              kernel`vn_stat+0xaf
              kernel`0xffffff800d52ec40+0x265
              kernel`fstatat64+0x6c
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_vnop_getattr                                                1

              smbfs`smb_fphelp+0x1a2
              smbfs`smb2_smb_create+0x322
              smbfs`smb2fs_smb_ntcreatex+0x236
              smbfs`smbfs_smb_findnext+0x3e0
              smbfs`smbfs_findnext+0x5f
              smbfs`smbfs_fetch_new_entries+0x409
              smbfs`smbfs_vnop_getattrlistbulk+0x1396
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_smb_get_parent                                              1

              smbfs`smb_fphelp+0x1a2
              smbfs`smb2_smb_create+0x322
              smbfs`smb2fs_smb_ntcreatex+0x236
              smbfs`smb2fs_smb_cmpd_query_dir_one+0x2f6
              smbfs`smbfs_lookup+0x407
              smbfs`smbfs_lookup+0x1d7
              smbfs`smbfs_vnop_lookup+0x5a3
              kernel`lookup+0x36e
              kernel`namei+0x1124
              kernel`0xffffff800d528b80+0x79
              kernel`0xffffff800d52ec40+0xfe
              kernel`fstatat64+0x6c
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_smb_get_parent                                              1

              kernel`VNOP_COMPOUND_OPEN+0x139
              kernel`0xffffff800d529b30+0x128
              kernel`open_nocancel+0x139
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_vnop_compound_open                                          3

              smbfs`smbfs_getattr+0x88
              kernel`vnode_getattr+0xae
              Sandbox`derive_vnode_type+0x64
              Sandbox`eval+0x8f1
              Sandbox`sb_evaluate_internal+0xc9
              Sandbox`hook_vnode_check_open+0xb8
              kernel`mac_vnode_check_open+0x9a
              kernel`vn_authorize_open_existing+0xbf
              smbfs`smbfs_vnop_compound_open+0x491
              kernel`VNOP_COMPOUND_OPEN+0x139
              kernel`vn_open_auth+0x1a1
              kernel`0xffffff800d529b30+0x128
              kernel`open_nocancel+0x139
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_update_cache                                                3

              smbfs`smbfs_getattr+0x88
              kernel`vnode_getattr+0xae
              Sandbox`derive_vnode_has_multiple_names+0x80
              Sandbox`derive_vnode_storage_class+0x528
              Sandbox`eval+0x1310
              Sandbox`sb_evaluate_internal+0xc9
              Sandbox`hook_vnode_check_open+0xb8
              kernel`mac_vnode_check_open+0x9a
              kernel`vn_authorize_open_existing+0xbf
              smbfs`smbfs_vnop_compound_open+0x491
              kernel`VNOP_COMPOUND_OPEN+0x139
              kernel`vn_open_auth+0x1a1
              kernel`0xffffff800d529b30+0x128
              kernel`open_nocancel+0x139
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_update_cache                                                3

              smbfs`smbfs_update_cache+0xa0
              smbfs`smbfs_vnop_getattr+0x1d3
              kernel`vnode_getattr+0xae
              Sandbox`derive_vnode_type+0x64
              Sandbox`eval+0x8f1
              Sandbox`sb_evaluate_internal+0xc9
              Sandbox`hook_vnode_check_open+0xb8
              kernel`mac_vnode_check_open+0x9a
              kernel`vn_authorize_open_existing+0xbf
              smbfs`smbfs_vnop_compound_open+0x491
              kernel`VNOP_COMPOUND_OPEN+0x139
              kernel`vn_open_auth+0x1a1
              kernel`0xffffff800d529b30+0x128
              kernel`open_nocancel+0x139
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_attr_cachelookup                                            3

              smbfs`smbfs_update_cache+0xa0
              smbfs`smbfs_vnop_getattr+0x1d3
              kernel`vnode_getattr+0xae
              Sandbox`derive_vnode_has_multiple_names+0x80
              Sandbox`derive_vnode_storage_class+0x528
              Sandbox`eval+0x1310
              Sandbox`sb_evaluate_internal+0xc9
              Sandbox`hook_vnode_check_open+0xb8
              kernel`mac_vnode_check_open+0x9a
              kernel`vn_authorize_open_existing+0xbf
              smbfs`smbfs_vnop_compound_open+0x491
              kernel`VNOP_COMPOUND_OPEN+0x139
              kernel`vn_open_auth+0x1a1
              kernel`0xffffff800d529b30+0x128
              kernel`open_nocancel+0x139
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_attr_cachelookup                                            3

              kernel`0xffffff800d51c4f0+0x667
              kernel`vn_authorize_open_existing+0x13c
              smbfs`smbfs_vnop_compound_open+0x491
              kernel`VNOP_COMPOUND_OPEN+0x139
              kernel`vn_open_auth+0x1a1
              kernel`0xffffff800d529b30+0x128
              kernel`open_nocancel+0x139
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_vnop_access                                                 3

              smbfs`smbfs_vnop_compound_open+0x95d
              kernel`vn_open_auth+0x1a1
              kernel`0xffffff800d529b30+0x128
              kernel`open_nocancel+0x139
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_vnop_open_common                                            3

              smbfs`smbfs_vnop_getattr+0x1d3
              Sandbox`derive_vnode_type+0x64
              Sandbox`eval+0x8f1
              Sandbox`sb_evaluate_internal+0xc9
              Sandbox`hook_vnode_check_open+0xb8
              kernel`mac_vnode_check_open+0x9a
              kernel`vn_authorize_open_existing+0xbf
              smbfs`smbfs_vnop_compound_open+0x491
              kernel`VNOP_COMPOUND_OPEN+0x139
              kernel`vn_open_auth+0x1a1
              kernel`0xffffff800d529b30+0x128
              kernel`open_nocancel+0x139
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_getattr                                                     3

              smbfs`smbfs_vnop_getattr+0x1d3
              Sandbox`derive_vnode_has_multiple_names+0x80
              Sandbox`derive_vnode_storage_class+0x528
              Sandbox`eval+0x1310
              Sandbox`sb_evaluate_internal+0xc9
              Sandbox`hook_vnode_check_open+0xb8
              kernel`mac_vnode_check_open+0x9a
              kernel`vn_authorize_open_existing+0xbf
              smbfs`smbfs_vnop_compound_open+0x491
              kernel`VNOP_COMPOUND_OPEN+0x139
              kernel`vn_open_auth+0x1a1
              kernel`0xffffff800d529b30+0x128
              kernel`open_nocancel+0x139
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_getattr                                                     3

              kernel`vnode_getattr+0xae
              Sandbox`eval+0x8f1
              Sandbox`sb_evaluate_internal+0xc9
              Sandbox`hook_vnode_check_open+0xb8
              kernel`mac_vnode_check_open+0x9a
              kernel`vn_authorize_open_existing+0xbf
              smbfs`smbfs_vnop_compound_open+0x491
              kernel`VNOP_COMPOUND_OPEN+0x139
              kernel`vn_open_auth+0x1a1
              kernel`0xffffff800d529b30+0x128
              kernel`open_nocancel+0x139
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_vnop_getattr                                                3

              kernel`vnode_getattr+0xae
              Sandbox`derive_vnode_storage_class+0x528
              Sandbox`eval+0x1310
              Sandbox`sb_evaluate_internal+0xc9
              Sandbox`hook_vnode_check_open+0xb8
              kernel`mac_vnode_check_open+0x9a
              kernel`vn_authorize_open_existing+0xbf
              smbfs`smbfs_vnop_compound_open+0x491
              kernel`VNOP_COMPOUND_OPEN+0x139
              kernel`vn_open_auth+0x1a1
              kernel`0xffffff800d529b30+0x128
              kernel`open_nocancel+0x139
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_vnop_getattr                                                3

              kernel`vn_close+0x12b
              kernel`0xffffff800d7e9590+0x13e
              kernel`0xffffff800d7eae70+0x53e
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_vnop_close                                                  3

              kernel`vn_pathconf+0xf5
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_vnop_pathconf                                               4

              kernel`0xffffff800d50d160+0x342
              kernel`namei+0x1124
              kernel`pathconf+0xc9
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_root                                                        4

              kernel`0xffffff800d50d160+0x342
              kernel`namei+0x1124
              kernel`getattrlist+0x132
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_root                                                        6

              kernel`vfs_getattr+0x2e
              kernel`0xffffff800d4f0920+0x216
              kernel`getattrlist+0x170
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_vfs_getattr                                                 6

              kernel`0xffffff800d526990+0x6b
              kernel`sync+0x17
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_sync                                                        7

              smbfs`smbfs_vnop_getattrlistbulk+0x1396
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_fetch_new_entries                                           8

              kernel`vnode_iterate+0x294
              kernel`0xffffff800d526990+0x6b
              kernel`vfs_iterate+0x21c
              kernel`sync+0x17
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_sync_callback                                              14

              smbfs`smbfs_vnop_getattrlistbulk+0xa75
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_verify_resume                                              16

              smbfs`smb_fphelp+0x1f5
              smbfs`smb2_smb_create+0x322
              smbfs`smb2fs_smb_ntcreatex+0x236
              smbfs`smb2fs_smb_cmpd_query_async_fill+0x103
              smbfs`smb2fs_smb_cmpd_query_async+0x341
              smbfs`smb_dir_cache_get_attrs+0x8f
              smbfs`smbfs_fetch_new_entries+0x609
              smbfs`smbfs_vnop_getattrlistbulk+0x1396
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_smb_get_parent                                             80

              smbfs`smb_fphelp+0x1a2
              smbfs`smb2_smb_create+0x322
              smbfs`smb2fs_smb_ntcreatex+0x236
              smbfs`smb2fs_smb_cmpd_query_async_fill+0x103
              smbfs`smb2fs_smb_cmpd_query_async+0x341
              smbfs`smb_dir_cache_get_attrs+0x8f
              smbfs`smbfs_fetch_new_entries+0x609
              smbfs`smbfs_vnop_getattrlistbulk+0x1396
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_smb_get_parent                                             80

              smbfs`smbfs_vnop_getattrlistbulk+0x130e
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_fetch_new_entries                                         193

              smbfs`smb_fphelp+0x1f5
              smbfs`smb2_smb_create+0x322
              smbfs`smb2fs_smb_ntcreatex+0x236
              smbfs`smbfs_smb_findnext+0x3e0
              smbfs`smbfs_findnext+0x5f
              smbfs`smbfs_fetch_new_entries+0x38e
              smbfs`smbfs_vnop_getattrlistbulk+0x130e
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_smb_get_parent                                            193

              smbfs`smb2_smb_parse_create+0xacc
              smbfs`smbfs_findnext+0x5f
              smbfs`smbfs_fetch_new_entries+0x38e
              smbfs`smbfs_vnop_getattrlistbulk+0x130e
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_get_epoch_delta                                           193

              smbfs`smbfs_closedirlookup+0x8b
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_smb_findclose                                             193

              smbfs`smb_fphelp+0x1a2
              smbfs`smb2_smb_create+0x322
              smbfs`smb2fs_smb_ntcreatex+0x236
              smbfs`smbfs_smb_findnext+0x3e0
              smbfs`smbfs_findnext+0x5f
              smbfs`smbfs_fetch_new_entries+0x38e
              smbfs`smbfs_vnop_getattrlistbulk+0x130e
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_smb_get_parent                                            193

              smbfs`smbfs_vnop_getattrlistbulk+0x12f0
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_closedirlookup                                            193

              smbfs`smbfs_fetch_new_entries+0x2ef
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_closedirlookup                                            194

              smbfs`smbfs_fetch_new_entries+0x32f
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_smb_findopen                                              194

              smbfs`smbfs_vnop_getattrlistbulk+0xf6e
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_verify_resume                                             387

              smbfs`smbfs_verify_resume+0x5d
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_entries_match                                             403

              kernel`getattrlistbulk+0x608
              kernel`hndl_unix_scall64+0x16
  smbfs_vnop_getattrlistbulk                                      404

              kernel`0xffffff800d51c4f0+0x667
              kernel`getattrlistbulk+0x362
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_vnop_access                                               405

              smbfs`smb_fphelp+0x1f5
              smbfs`smb2_smb_create+0x322
              smbfs`smb2fs_smb_ntcreatex+0x236
              smbfs`smb2fs_smb_cmpd_query_async_fill+0x103
              smbfs`smb2fs_smb_cmpd_query_async+0x341
              smbfs`smb_dir_cache_get_attrs+0x8f
              smbfs`smbfs_fetch_new_entries+0x609
              smbfs`smbfs_vnop_getattrlistbulk+0x130e
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_smb_get_parent                                           1920

              smbfs`smb_fphelp+0x1a2
              smbfs`smb2_smb_create+0x322
              smbfs`smb2fs_smb_ntcreatex+0x236
              smbfs`smb2fs_smb_cmpd_query_async_fill+0x103
              smbfs`smb2fs_smb_cmpd_query_async+0x341
              smbfs`smb_dir_cache_get_attrs+0x8f
              smbfs`smbfs_fetch_new_entries+0x609
              smbfs`smbfs_vnop_getattrlistbulk+0x130e
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_smb_get_parent                                           1920

              smbfs`smb_fphelp+0x1f5
              smbfs`smb2_smb_create+0x322
              smbfs`smb2fs_smb_ntcreatex+0x236
              smbfs`smb2fs_smb_cmpd_query_async_fill+0x103
              smbfs`smb2fs_smb_cmpd_query_async+0x576
              smbfs`smb_dir_cache_get_attrs+0x8f
              smbfs`smbfs_fetch_new_entries+0x609
              smbfs`smbfs_vnop_getattrlistbulk+0x1396
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_smb_get_parent                                           7920

              smbfs`smb_fphelp+0x1a2
              smbfs`smb2_smb_create+0x322
              smbfs`smb2fs_smb_ntcreatex+0x236
              smbfs`smb2fs_smb_cmpd_query_async_fill+0x103
              smbfs`smb2fs_smb_cmpd_query_async+0x576
              smbfs`smb_dir_cache_get_attrs+0x8f
              smbfs`smbfs_fetch_new_entries+0x609
              smbfs`smbfs_vnop_getattrlistbulk+0x1396
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_smb_get_parent                                           7920

              smbfs`smb2_smb_parse_query_info+0x2d9
              smbfs`smb2fs_smb_cmpd_query_async+0x549
              smbfs`smb_dir_cache_get_attrs+0x8f
              smbfs`smbfs_fetch_new_entries+0x609
              smbfs`smbfs_vnop_getattrlistbulk+0x1396
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_ntwrkname_tolocal                                        8000

              smbfs`smb2_smb_parse_query_info+0x308
              smbfs`smb2fs_smb_cmpd_query_async+0x549
              smbfs`smb_dir_cache_get_attrs+0x8f
              smbfs`smbfs_fetch_new_entries+0x609
              smbfs`smbfs_vnop_getattrlistbulk+0x1396
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_smb_undollardata                                         8000

              smbfs`smbfs_nget+0x211
              smbfs`smbfs_vnop_getattrlistbulk+0xe15
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_hash                                                     8000

              smbfs`smbfs_findnext+0x18e
              smbfs`smbfs_vnop_getattrlistbulk+0x1396
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_ntwrkname_tolocal                                        8000

              smbfs`smbfs_findnext+0x5f
              smbfs`smbfs_vnop_getattrlistbulk+0x1396
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_smb_findnext                                             8002

              smbfs`smbfs_vnop_getattrlistbulk+0xe15
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_add_dir_entry_attr                                       8016

              smbfs`smbfs_nget+0x211
              smbfs`smbfs_vnop_getattrlistbulk+0x143a
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_hash                                                    47856

              smbfs`smbfs_vnop_getattrlistbulk+0x143a
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_add_dir_entry_attr                                      48048

              smbfs`smbfs_nget+0x211
              smbfs`smbfs_vnop_getattrlistbulk+0xff3
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_hash                                                   144144

              smbfs`smbfs_vnop_getattrlistbulk+0xff3
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_add_dir_entry_attr                                     144339

              smbfs`smb_fphelp+0x1f5
              smbfs`smb2_smb_create+0x322
              smbfs`smb2fs_smb_ntcreatex+0x236
              smbfs`smb2fs_smb_cmpd_query_async_fill+0x103
              smbfs`smb2fs_smb_cmpd_query_async+0x576
              smbfs`smb_dir_cache_get_attrs+0x8f
              smbfs`smbfs_fetch_new_entries+0x609
              smbfs`smbfs_vnop_getattrlistbulk+0x130e
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_smb_get_parent                                         190080

              smbfs`smb_fphelp+0x1a2
              smbfs`smb2_smb_create+0x322
              smbfs`smb2fs_smb_ntcreatex+0x236
              smbfs`smb2fs_smb_cmpd_query_async_fill+0x103
              smbfs`smb2fs_smb_cmpd_query_async+0x576
              smbfs`smb_dir_cache_get_attrs+0x8f
              smbfs`smbfs_fetch_new_entries+0x609
              smbfs`smbfs_vnop_getattrlistbulk+0x130e
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_smb_get_parent                                         190080

              smbfs`smb2_smb_parse_query_info+0x2d9
              smbfs`smb2fs_smb_cmpd_query_async+0x549
              smbfs`smb_dir_cache_get_attrs+0x8f
              smbfs`smbfs_fetch_new_entries+0x609
              smbfs`smbfs_vnop_getattrlistbulk+0x130e
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_ntwrkname_tolocal                                      192000

              smbfs`smb2_smb_parse_query_info+0x308
              smbfs`smb2fs_smb_cmpd_query_async+0x549
              smbfs`smb_dir_cache_get_attrs+0x8f
              smbfs`smbfs_fetch_new_entries+0x609
              smbfs`smbfs_vnop_getattrlistbulk+0x130e
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_smb_undollardata                                       192000

              smbfs`smbfs_add_dir_entry_attr+0x25c
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_nget                                                   200000

              smbfs`smbfs_fetch_new_entries+0x409
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_findnext                                               200001

              smbfs`smb_iod_sendall+0x154
              kernel`call_continuation+0x2e
  smbfs_is_going_away                                         1849705

              smbfs`smbfs_fetch_new_entries+0x38e
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_findnext                                             20072000

              smbfs`smbfs_findnext+0x18e
              smbfs`smbfs_vnop_getattrlistbulk+0x130e
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_ntwrkname_tolocal                                    20264000

              smbfs`smbfs_findnext+0x5f
              smbfs`smbfs_vnop_getattrlistbulk+0x130e
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_smb_findnext                                         20264387

Of key relevance here is

Code:

             smbfs`smbfs_fetch_new_entries+0x2ef
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_closedirlookup                                            194

              smbfs`smbfs_fetch_new_entries+0x32f
              kernel`getattrlistbulk+0x608
              kernel`unix_syscall64+0x2ce
              kernel`hndl_unix_scall64+0x16
  smbfs_smb_findopen                                              194

That's a minimum of 194 open, read, close of the directory contents. (Not counting bulkattr calls, and calls to refresh the directory).

The MacOS dtrace is fairly simple in this case:

Code:

#!/usr/sbin/dtrace -s
fbt::smbfs*:entry
{
@[stack(),probefunc] = count()
}

Which is what roughly typical for what I do when beginning to investigate some issue or other on MacOS. You get basically an overview of which functions are being called and then can start drilling into particular code-paths.

Anyway, the moral of this story is: don't put _huge_ numbers of files in directories. My guess is that my client is absolutely blowing through its directory cache size and generating tons of extra network IO.

 

[anodos]

TrueNAS-13.0-CUSTOM_RESOLVE_NO_SYMLINK_15-manual-update-unsigned.tar

drive.google.com

This is latest experimental build with 4.15 if you feel like taking it for a spin ixnas:dirent_optimization = true is the parameter to enable the fast-path for dir listing (though it only helps marginally for extremely pathological cases - like 200K files in a dir on MacOS). I still have some cleanups to do before merge for U5 (where it will be off-by-default till there is enough testing).

 

[simonj]

TrueNAS-13.0-CUSTOM_RESOLVE_NO_SYMLINK_15-manual-update-unsigned.tar

drive.google.com

This is latest experimental build with 4.15 if you feel like taking it for a spin ixnas:dirent_optimization = true is the parameter to enable the fast-path for dir listing (though it only helps marginally for extremely pathological cases - like 200K files in a dir on MacOS). I still have some cleanups to do before merge for U5 (where it will be off-by-default till there is enough testing).

Great. Will try that out ASAP.

Anyway, the moral of this story is: don't put _huge_ numbers of files in directories. My guess is that my client is absolutely blowing through its directory cache size and generating tons of extra network IO.

The problem is that in a professional film post workflow you don't get around having folders with 150K files. We do post for feature films and often need to work with whole movies as DPX or TIFF sequences (90 minutes * 60 seconds * 24 frames). That's also a deliverable that we need to provide for 99% of the clients and many professional digital film cameras (like the Arri Alexa) record footage as image sequences. We could try to split the movies into 3-4 acts but I guess that wouldn't change speed substantially and create a lot of hassle. Same probably for the professional audio workflow where you cannot work around that Avid Pro Tools will need to save sessions containing ten-thousands of files.

I used pmcstat to get data for flamegraph. I'm still looking at MacOS client. It's just pathologically bad at listing directories regardless of server. Against a Windows server with 200K files in a dir it's around 50x slower than other clients because it spins on repeatedly asking server to rescan dir from beginning. I'm having to adjust my strategy for dealing with clients to assume that the client may repeatedly rescan same directory (perhaps 100 times) while generating a listing.

Ok. So clearly Mac clients do directory listings in a super inefficient way compared to Win/Linux.
But why are the listings relatively fast when Mac clients access folders on Mac servers?
Wouldn't it be worth to investigate that to possibly replicate the behavior of a Mac server?

@anodos Do you have some feedback on my tests always performing better with Apple Extensions off?

In Scale it seems possible to install an elastic search / spotlight backend. Might this help?

 

[simonj]

Interesting info on Apple extensions here:

https://www.delltechnologies.com/asset/en-us/products/storage/industry-market/h17613_wp_isilon_mac_os_performance_optimization.pdf.external

+ a newer version of the pdf: https://www.delltechnologies.com/as...owerscale-mac-os-performance-optimization.pdf

 

[tiberiusQ]

Hi,

Same issues here and glad to read that I am not alone ;-) !
Looking forward for U5 !

 

[Nicolas_Studiokgb]

TrueNAS-13.0-CUSTOM_RESOLVE_NO_SYMLINK_15-manual-update-unsigned.tar

drive.google.com

This is latest experimental build with 4.15 if you feel like taking it for a spin ixnas:dirent_optimization = true is the parameter to enable the fast-path for dir listing (though it only helps marginally for extremely pathological cases - like 200K files in a dir on MacOS). I still have some cleanups to do before merge for U5 (where it will be off-by-default till there is enough testing).

Hi everybody. I was away from office for two weeks.
Trying this new version today and let you know.

@anodos : is there a way to analyse what a mac does when it act as a smb share server ? It works fine between macs so it means there is something that make it work when the smb share is done on a mac.

Thanks