SMBD process goes 100% and copy speed collapse, ARC request demand metadata issue ?

[Nicolas_Studiokgb]

Hello
Here the result of the command while deleting

I've used `smbstatus` to get pid of the SMB session, then run smbcontrol all profile flush, then smbcontrol <pid> profile on, then repeat the operation and run smbstatus -P to dump profiling metrics.

Approx 40k files with the Mac Os finder from Truenas.
ARC Req Demand_metadata goes up to 45M
Smbd processes still go 100%

Following mac settings :
defaults write com.apple.desktopservices DSDontWriteNetworkStores -bool TRUE
(This one might maybe limit to 45M instead of 65M)

and in /etc/nsmb.conf
[default] dir_cache_max_cnt=0 signing_required=no

Bellow the screen shot of ZFS reporting in the GUI
Let me know

root@truenas[~]# smbstatus -P **** "SMBD loop" ************************************************************** connect_count: 0 disconnect_count: 0 idle_count: 193634 idle_time: 348954598 cpu_user_time: 846156025831 cpu_system_time: 1863381399126 request_count: 572837 push_sec_ctx_count: 2659 push_sec_ctx_time: 5427 set_sec_ctx_count: 149 set_sec_ctx_time: 1966 set_root_sec_ctx_count: 2608 set_root_sec_ctx_time: 27965 pop_sec_ctx_count: 2659 pop_sec_ctx_time: 12944 **** "System Calls" *********************************************************** syscall_opendir_count: 0 syscall_opendir_time: 0 syscall_fdopendir_count: 150798 syscall_fdopendir_time: 7891913 syscall_readdir_count: 1590322 syscall_readdir_time: 1005890 syscall_seekdir_count: 46 syscall_seekdir_time: 16 syscall_telldir_count: 1587898 syscall_telldir_time: 217212 syscall_rewinddir_count: 0 syscall_rewinddir_time: 0 syscall_mkdirat_count: 0 syscall_mkdirat_time: 0 syscall_closedir_count: 150798 syscall_closedir_time: 281739 syscall_open_count: 0 syscall_open_time: 0 syscall_openat_count: 12455398 syscall_openat_time: 44795372 syscall_createfile_count: 0 syscall_createfile_time: 0 syscall_close_count: 10612575 syscall_close_time: 363817569 syscall_pread_count: 0 syscall_pread_time: 0 syscall_pread_idle: 0 syscall_pread_bytes: 0 syscall_asys_pread_count: 0 syscall_asys_pread_time: 0 syscall_asys_pread_idle: 0 syscall_asys_pread_bytes: 0 syscall_pwrite_count: 0 syscall_pwrite_time: 0 syscall_pwrite_idle: 0 syscall_pwrite_bytes: 0 syscall_asys_pwrite_count: 0 syscall_asys_pwrite_time: 0 syscall_asys_pwrite_idle: 0 syscall_asys_pwrite_bytes: 0 syscall_lseek_count: 0 syscall_lseek_time: 0 syscall_sendfile_count: 0 syscall_sendfile_time: 0 syscall_sendfile_idle: 0 syscall_sendfile_bytes: 0 syscall_recvfile_count: 0 syscall_recvfile_time: 0 syscall_recvfile_idle: 0 syscall_recvfile_bytes: 0 syscall_renameat_count: 0 syscall_renameat_time: 0 syscall_asys_fsync_count: 0 syscall_asys_fsync_time: 0 syscall_asys_fsync_idle: 0 syscall_asys_fsync_bytes: 0 syscall_stat_count: 91360564 syscall_stat_time: 655678870 syscall_fstat_count: 13438224 syscall_fstat_time: 13166601 syscall_lstat_count: 4236748 syscall_lstat_time: 58065639 syscall_get_alloc_size_count: 1931060 syscall_get_alloc_size_time: 438872 syscall_unlinkat_count: 39370 syscall_unlinkat_time: 678113 syscall_chmod_count: 0 syscall_chmod_time: 0 syscall_fchmod_count: 0 syscall_fchmod_time: 0 syscall_fchown_count: 0 syscall_fchown_time: 0 syscall_lchown_count: 0 syscall_lchown_time: 0 syscall_chdir_count: 26317084 syscall_chdir_time: 296844384 syscall_getwd_count: 123 syscall_getwd_time: 2706 syscall_fntimes_count: 0 syscall_fntimes_time: 0 syscall_ftruncate_count: 0 syscall_ftruncate_time: 0 syscall_fallocate_count: 0 syscall_fallocate_time: 0 syscall_fcntl_lock_count: 0 syscall_fcntl_lock_time: 0 syscall_kernel_flock_count: 0 syscall_kernel_flock_time: 0 syscall_fcntl_count: 2453766 syscall_fcntl_time: 1795903 syscall_linux_setlease_count: 0 syscall_linux_setlease_time: 0 syscall_fcntl_getlock_count: 0 syscall_fcntl_getlock_time: 0 syscall_readlinkat_count: 0 syscall_readlinkat_time: 0 syscall_symlinkat_count: 0 syscall_symlinkat_time: 0 syscall_linkat_count: 0 syscall_linkat_time: 0 syscall_mknodat_count: 0 syscall_mknodat_time: 0 syscall_realpath_count: 15042398 syscall_realpath_time: 459450632 syscall_get_quota_count: 0 syscall_get_quota_time: 0 syscall_set_quota_count: 0 syscall_set_quota_time: 0 syscall_get_sd_count: 0 syscall_get_sd_time: 0 syscall_set_sd_count: 0 syscall_set_sd_time: 0 syscall_brl_lock_count: 0 syscall_brl_lock_time: 0 syscall_brl_unlock_count: 0 syscall_brl_unlock_time: 0 syscall_brl_cancel_count: 0 syscall_brl_cancel_time: 0 syscall_asys_getxattrat_count: 0 syscall_asys_getxattrat_time: 0 syscall_asys_getxattrat_idle: 0 syscall_asys_getxattrat_bytes: 0 **** "ACL Calls" ************************************************************** get_nt_acl_count: 0 get_nt_acl_time: 0 get_nt_acl_at_count: 0 get_nt_acl_at_time: 0 fget_nt_acl_count: 0 fget_nt_acl_time: 0 fset_nt_acl_count: 0 fset_nt_acl_time: 0 **** "Stat Cache" ************************************************************* statcache_lookups_count: 0 statcache_misses_count: 0 statcache_hits_count: 0 **** "SMB Calls" ************************************************************** SMBmkdir_count: 0 SMBmkdir_time: 0 SMBrmdir_count: 0 SMBrmdir_time: 0 SMBopen_count: 0 SMBopen_time: 0 SMBcreate_count: 0 SMBcreate_time: 0 SMBclose_count: 0 SMBclose_time: 0 SMBflush_count: 0 SMBflush_time: 0 SMBunlink_count: 0 SMBunlink_time: 0 SMBmv_count: 0 SMBmv_time: 0 SMBgetatr_count: 0 SMBgetatr_time: 0 SMBsetatr_count: 0 SMBsetatr_time: 0 SMBread_count: 0 SMBread_time: 0 SMBwrite_count: 0 SMBwrite_time: 0 SMBlock_count: 0 SMBlock_time: 0 SMBunlock_count: 0 SMBunlock_time: 0 SMBctemp_count: 0 SMBctemp_time: 0 SMBmknew_count: 0 SMBmknew_time: 0 SMBcheckpath_count: 0 SMBcheckpath_time: 0 SMBexit_count: 0 SMBexit_time: 0 SMBlseek_count: 0 SMBlseek_time: 0 SMBlockread_count: 0 SMBlockread_time: 0 SMBwriteunlock_count: 0 SMBwriteunlock_time: 0 SMBreadbraw_count: 0 SMBreadbraw_time: 0 SMBreadBmpx_count: 0 SMBreadBmpx_time: 0 SMBreadBs_count: 0 SMBreadBs_time: 0 SMBwritebraw_count: 0 SMBwritebraw_time: 0 SMBwriteBmpx_count: 0 SMBwriteBmpx_time: 0 SMBwriteBs_count: 0 SMBwriteBs_time: 0 SMBwritec_count: 0 SMBwritec_time: 0 SMBsetattrE_count: 0 SMBsetattrE_time: 0 SMBgetattrE_count: 0 SMBgetattrE_time: 0 SMBlockingX_count: 0 SMBlockingX_time: 0 SMBtrans_count: 0 SMBtrans_time: 0 SMBtranss_count: 0 SMBtranss_time: 0 SMBioctl_count: 0 SMBioctl_time: 0 SMBioctls_count: 0 SMBioctls_time: 0 SMBcopy_count: 0 SMBcopy_time: 0 SMBmove_count: 0 SMBmove_time: 0 SMBecho_count: 0 SMBecho_time: 0 SMBwriteclose_count: 0 SMBwriteclose_time: 0 SMBopenX_count: 0 SMBopenX_time: 0 SMBreadX_count: 0 SMBreadX_time: 0 SMBwriteX_count: 0 SMBwriteX_time: 0 SMBtrans2_count: 0 SMBtrans2_time: 0 SMBtranss2_count: 0 SMBtranss2_time: 0 SMBfindclose_count: 0 SMBfindclose_time: 0 SMBfindnclose_count: 0 SMBfindnclose_time: 0 SMBtcon_count: 0 SMBtcon_time: 0 SMBtdis_count: 0 SMBtdis_time: 0 SMBnegprot_count: 0 SMBnegprot_time: 0 SMBsesssetupX_count: 0 SMBsesssetupX_time: 0 SMBulogoffX_count: 0 SMBulogoffX_time: 0 SMBtconX_count: 0 SMBtconX_time: 0 SMBdskattr_count: 0 SMBdskattr_time: 0 SMBsearch_count: 0 SMBsearch_time: 0 SMBffirst_count: 0 SMBffirst_time: 0 SMBfunique_count: 0 SMBfunique_time: 0 SMBfclose_count: 0 SMBfclose_time: 0 SMBnttrans_count: 0 SMBnttrans_time: 0 SMBnttranss_count: 0 SMBnttranss_time: 0 SMBntcreateX_count: 0 SMBntcreateX_time: 0 SMBntcancel_count: 0 SMBntcancel_time: 0 SMBntrename_count: 0 SMBntrename_time: 0 SMBsplopen_count: 0 SMBsplopen_time: 0 SMBsplwr_count: 0 SMBsplwr_time: 0 SMBsplclose_count: 0 SMBsplclose_time: 0 SMBsplretq_count: 0 SMBsplretq_time: 0 SMBsends_count: 0 SMBsends_time: 0 SMBsendb_count: 0 SMBsendb_time: 0 SMBfwdname_count: 0 SMBfwdname_time: 0 SMBcancelf_count: 0 SMBcancelf_time: 0 SMBgetmac_count: 0 SMBgetmac_time: 0 SMBsendstrt_count: 0 SMBsendstrt_time: 0 SMBsendend_count: 0 SMBsendend_time: 0 SMBsendtxt_count: 0 SMBsendtxt_time: 0 SMBinvalid_count: 0 SMBinvalid_time: 0 **** "Trans2 Calls" *********************************************************** Trans2_open_count: 0 Trans2_open_time: 0 Trans2_findfirst_count: 0 Trans2_findfirst_time: 0 Trans2_findnext_count: 0 Trans2_findnext_time: 0 Trans2_qfsinfo_count: 0 Trans2_qfsinfo_time: 0 Trans2_setfsinfo_count: 0 Trans2_setfsinfo_time: 0 Trans2_qpathinfo_count: 0 Trans2_qpathinfo_time: 0 Trans2_setpathinfo_count: 0 Trans2_setpathinfo_time: 0 Trans2_qfileinfo_count: 0 Trans2_qfileinfo_time: 0 Trans2_setfileinfo_count: 0 Trans2_setfileinfo_time: 0 Trans2_fsctl_count: 0 Trans2_fsctl_time: 0 Trans2_ioctl_count: 0 Trans2_ioctl_time: 0 Trans2_findnotifyfirst_count: 0 Trans2_findnotifyfirst_time: 0 Trans2_findnotifynext_count: 0 Trans2_findnotifynext_time: 0 Trans2_mkdir_count: 0 Trans2_mkdir_time: 0 Trans2_session_setup_count: 0 Trans2_session_setup_time: 0 Trans2_get_dfs_referral_count: 0 Trans2_get_dfs_referral_time: 0 Trans2_report_dfs_inconsistancy_count: 0 Trans2_report_dfs_inconsistancy_time: 0 **** "NT Transact Calls" ****************************************************** NT_transact_create_count: 0 NT_transact_create_time: 0 NT_transact_ioctl_count: 0 NT_transact_ioctl_time: 0 NT_transact_set_security_desc_count: 0 NT_transact_set_security_desc_time: 0 NT_transact_notify_change_count: 0 NT_transact_notify_change_time: 0 NT_transact_rename_count: 0 NT_transact_rename_time: 0 NT_transact_query_security_desc_count: 0 NT_transact_query_security_desc_time: 0 NT_transact_get_user_quota_count: 0 NT_transact_get_user_quota_time: 0 NT_transact_set_user_quota_count: 0 NT_transact_set_user_quota_time: 0 **** "SMB2 Calls" ************************************************************* smb2_negprot_count: 0 smb2_negprot_time: 0 smb2_negprot_idle: 0 smb2_negprot_inbytes: 0 smb2_negprot_outbytes: 0 smb2_sesssetup_count: 0 smb2_sesssetup_time: 0 smb2_sesssetup_idle: 0 smb2_sesssetup_inbytes: 0 smb2_sesssetup_outbytes: 0 smb2_logoff_count: 0 smb2_logoff_time: 0 smb2_logoff_idle: 0 smb2_logoff_inbytes: 0 smb2_logoff_outbytes: 0 smb2_tcon_count: 0 smb2_tcon_time: 0 smb2_tcon_idle: 0 smb2_tcon_inbytes: 0 smb2_tcon_outbytes: 0 smb2_tdis_count: 0 smb2_tdis_time: 0 smb2_tdis_idle: 0 smb2_tdis_inbytes: 0 smb2_tdis_outbytes: 0 smb2_create_count: 190694 smb2_create_time: 145301569 smb2_create_idle: 0 smb2_create_inbytes: 55413160 smb2_create_outbytes: 30264872 smb2_close_count: 190690 smb2_close_time: 384450612 smb2_close_idle: 0 smb2_close_inbytes: 16780720 smb2_close_outbytes: 24405484 smb2_flush_count: 0 smb2_flush_time: 0 smb2_flush_idle: 0 smb2_flush_inbytes: 0 smb2_flush_outbytes: 0 smb2_read_count: 0 smb2_read_time: 0 smb2_read_idle: 0 smb2_read_inbytes: 0 smb2_read_outbytes: 0 smb2_write_count: 0 smb2_write_time: 0 smb2_write_idle: 0 smb2_write_inbytes: 0 smb2_write_outbytes: 0 smb2_lock_count: 0 smb2_lock_time: 0 smb2_lock_idle: 0 smb2_lock_inbytes: 0 smb2_lock_outbytes: 0 smb2_ioctl_count: 0 smb2_ioctl_time: 0 smb2_ioctl_idle: 0 smb2_ioctl_inbytes: 0 smb2_ioctl_outbytes: 0 smb2_cancel_count: 0 smb2_cancel_time: 0 smb2_cancel_idle: 0 smb2_cancel_inbytes: 0 smb2_cancel_outbytes: 0 smb2_keepalive_count: 1 smb2_keepalive_time: 1 smb2_keepalive_idle: 0 smb2_keepalive_inbytes: 68 smb2_keepalive_outbytes: 68 smb2_find_count: 151450 smb2_find_time: 2083487101 smb2_find_idle: 0 smb2_find_inbytes: 22128546 smb2_find_outbytes: 280014099 smb2_notify_count: 1 smb2_notify_time: 1082401 smb2_notify_idle: 1082164 smb2_notify_inbytes: 96 smb2_notify_outbytes: 73 smb2_getinfo_count: 621 smb2_getinfo_time: 298476 smb2_getinfo_idle: 0 smb2_getinfo_inbytes: 64584 smb2_getinfo_outbytes: 66896 smb2_setinfo_count: 39370 smb2_setinfo_time: 1910650 smb2_setinfo_idle: 0 smb2_setinfo_inbytes: 4094480 smb2_setinfo_outbytes: 2834640 smb2_break_count: 0 smb2_break_time: 0 smb2_break_idle: 0 smb2_break_inbytes: 0 smb2_break_outbytes: 0 root@truenas[~]#

View attachment 64976

 

[Nicolas_Studiokgb]

NFS Test
Created a nfs share, connect with the mac

Copy 107GB 16k folder = 5 minutes (same approx than SMB)
Delete = 35 seconds instead of 16 minutes in SMB

To take in consideration : How NFS handle Mac Os metadata, alternate streams, resource forks.... ? I don't know for now. This test was just to see what happen when using NFS setup as default settings (everyone has access) No other settings set appart than default ones.

Also we see that ARC react quite differently
Copy : no more than 1,5M Req Demand_metadata (instead of about 15/20M in smb)
Delete : no more than 2,2M instead of about 45M/65M

 

[Nicolas_Studiokgb]

Hi guys :)

My little today's tests :

So I think I touch to problem today.
It is something between
Enabling or not Alternate Data streams (Shares options) and
Enabling or not Apple SMB2/3 Extentions (SMB service)

For a set of 107GB 16k items (audiofiles)

Copy Time : There is no much difference between any enabled or disabled option : It's about 5/6 minutes copy.
Delete Time :
Both enabled : 15 minutes
Disable Apple SMB2/3 extensions / Enable Alternate data streams : about 6 minutes
Enable Apple SMB2/3 extensions / Disable Alternate data streams : about 15 minutes
Disable Both : 2 minutes 15secondes !!!!! (not the perf of NFS but clearly fine.)

And the less the delete takes time the less the ARC Request Demand_Metadata goes up :
for the fastest delete : less than 15M
for the longest one : about 40M

Is it really surprising as all this might be related to the way Mac Os / Truenas handle File metadata...

If any idea don't be shy, let me know :D

 

[Nicolas_Studiokgb]

Testparm -s for the 4 configurations

Both enabled : 15 minutes

root@truenas[~]# testparm -s
Load smb config files from /usr/local/etc/smb4.conf
Loaded services file OK.
Weak crypto is allowed

Server role: ROLE_STANDALONE

# Global parameters
[global]
aio max threads = 2
bind interfaces only = Yes
disable spoolss = Yes
dns proxy = No
enable web service discovery = Yes
kernel change notify = No
load printers = No
logging = file
max log size = 5120
nsupdate command = /usr/local/bin/samba-nsupdate -g
registry shares = Yes
restrict anonymous = 2
server multi channel support = No
server role = standalone server
server string = TrueNAS Server
unix extensions = No
idmap config *: range = 90000001-100000000
fruit:nfs_aces = No
rpc_server:mdssvc = disabled
rpc_daemon:mdssd = disabled
idmap config * : backend = tdb
directory name cache size = 0
dos filemode = Yes


[noads]
ea support = No
mangled names = no
path = /mnt/RAID0/noads
read only = No
smbd max xattr size = 2097152
vfs objects = catia fruit streams_xattr ixnas zfs_core aio_fbsd
fruit:resource = stream
fruit:metadata = stream
fruit:encoding = native
nfs4:chown = true
ixnas:dosattrib_xattr = false

Disable Apple SMB2/3 extensions / Enable Alternate data streams : about 6 minutes

root@truenas[~]# testparm -s
Load smb config files from /usr/local/etc/smb4.conf
Loaded services file OK.
Weak crypto is allowed

Server role: ROLE_STANDALONE

# Global parameters
[global]
aio max threads = 2
bind interfaces only = Yes
disable spoolss = Yes
dns proxy = No
enable web service discovery = Yes
kernel change notify = No
load printers = No
logging = file
max log size = 5120
nsupdate command = /usr/local/bin/samba-nsupdate -g
registry shares = Yes
restrict anonymous = 2
server multi channel support = No
server role = standalone server
server string = TrueNAS Server
unix extensions = No
idmap config *: range = 90000001-100000000
rpc_server:mdssvc = disabled
rpc_daemon:mdssd = disabled
idmap config * : backend = tdb
directory name cache size = 0
dos filemode = Yes




[noads]
ea support = No
mangled names = no
path = /mnt/RAID0/noads
read only = No
smbd max xattr size = 2097152
vfs objects = catia streams_xattr ixnas zfs_core aio_fbsd
catia:mappings = 0x01:0xf001,0x02:0xf002,0x03:0xf003,0x04:0xf004,0x05:0xf005,0x06:0xf006,0x07:0xf007,0x08:0xf008,0x09:0xf009,0x0a:0xf00a,0x0b:0xf00b,0x0c:0xf00c,0x0d:0xf00d,0x0e:0xf00e,0x0f:0xf00f,0x10:0xf010,0x11:0xf011,0x12:0xf012,0x13:0xf013,0x14:0xf014,0x15:0xf015,0x16:0xf016,0x17:0xf017,0x18:0xf018,0x19:0xf019,0x1a:0xf01a,0x1b:0xf01b,0x1c:0xf01c,0x1d:0xf01d,0x1e:0xf01e,0x1f:0xf01f,0x22:0xf020,0x2a:0xf021,0x3a:0xf022,0x3c:0xf023,0x3e:0xf024,0x3f:0xf025,0x5c:0xf026,0x7c:0xf027
nfs4:chown = true
ixnas:dosattrib_xattr = false

Enable Apple SMB2/3 extensions / Disable Alternate data streams : about 15 minutes

root@truenas[~]# testparm -s
Load smb config files from /usr/local/etc/smb4.conf
Loaded services file OK.
Weak crypto is allowed

Server role: ROLE_STANDALONE

# Global parameters
[global]
aio max threads = 2
bind interfaces only = Yes
disable spoolss = Yes
dns proxy = No
enable web service discovery = Yes
kernel change notify = No
load printers = No
logging = file
max log size = 5120
nsupdate command = /usr/local/bin/samba-nsupdate -g
registry shares = Yes
restrict anonymous = 2
server multi channel support = No
server role = standalone server
server string = TrueNAS Server
unix extensions = No
idmap config *: range = 90000001-100000000
fruit:nfs_aces = No
rpc_server:mdssvc = disabled
rpc_daemon:mdssd = disabled
idmap config * : backend = tdb
directory name cache size = 0
dos filemode = Yes


[noads]
ea support = No
mangled names = no
path = /mnt/RAID0/noads
read only = No
vfs objects = catia fruit streams_xattr ixnas zfs_core aio_fbsd
fruit:resource = stream
fruit:metadata = stream
fruit:encoding = native
nfs4:chown = true
ixnas:dosattrib_xattr = false

Disable Both : 2 minutes 15secondes !!!!! (not the perf of NFS but clearly fine.)

root@truenas[~]# testparm -s
Load smb config files from /usr/local/etc/smb4.conf
Loaded services file OK.
Weak crypto is allowed

Server role: ROLE_STANDALONE

# Global parameters
[global]
aio max threads = 2
bind interfaces only = Yes
disable spoolss = Yes
dns proxy = No
enable web service discovery = Yes
kernel change notify = No
load printers = No
logging = file
max log size = 5120
nsupdate command = /usr/local/bin/samba-nsupdate -g
registry shares = Yes
restrict anonymous = 2
server multi channel support = No
server role = standalone server
server string = TrueNAS Server
unix extensions = No
idmap config *: range = 90000001-100000000
rpc_server:mdssvc = disabled
rpc_daemon:mdssd = disabled
idmap config * : backend = tdb
directory name cache size = 0
dos filemode = Yes




[noads]
ea support = No
mangled names = no
path = /mnt/RAID0/noads
read only = No
vfs objects = catia ixnas zfs_core aio_fbsd
catia:mappings = 0x01:0xf001,0x02:0xf002,0x03:0xf003,0x04:0xf004,0x05:0xf005,0x06:0xf006,0x07:0xf007,0x08:0xf008,0x09:0xf009,0x0a:0xf00a,0x0b:0xf00b,0x0c:0xf00c,0x0d:0xf00d,0x0e:0xf00e,0x0f:0xf00f,0x10:0xf010,0x11:0xf011,0x12:0xf012,0x13:0xf013,0x14:0xf014,0x15:0xf015,0x16:0xf016,0x17:0xf017,0x18:0xf018,0x19:0xf019,0x1a:0xf01a,0x1b:0xf01b,0x1c:0xf01c,0x1d:0xf01d,0x1e:0xf01e,0x1f:0xf01f,0x22:0xf020,0x2a:0xf021,0x3a:0xf022,0x3c:0xf023,0x3e:0xf024,0x3f:0xf025,0x5c:0xf026,0x7c:0xf027
nfs4:chown = true
ixnas:dosattrib_xattr = false

 

[Nicolas_Studiokgb]

Today some other tries and now even copies are not working...
It goes 30M ARC req demand metadata // smbd process 100% // network speed goes to nothing... and copy time goes up and up.. until I cancel because I won't sleep in front of it...

As always if anybody met this problem before let me know...

 

[Nicolas_Studiokgb]

Hi everybody
I've tried another one today:
I copy (still the same set of data) 107GB 16k items (audiofiles .wav)
Copy is about 5 minutes to copy as usual
And I overwrite this folder again by itself (do you want to replace files : yes)
I takes 21 minutes to overwrite 107GB 16k items.
There is 3 folders with files inside and you can see the metadata again goes 45M before actually start the copy of each folder (the part where metadata go down to 10M is the files actually been transfered)
smbd processes @ 100%
root@truenas[~]# sysctl -a | grep kstat.zfs.misc.arcstats.metadata_size
kstat.zfs.misc.arcstats.metadata_size: 208209920
Size of the ARC aprox 104GB (RAM is 256GB)

 

[simonj]

Hi. I just wanted to chime in here telling that we see very similar problems with Mac clients and folders containing many files:

Finder-browsing gets SMB stuck at 100% cpu - in otherwise almost perfect film post-production setup with Mac clients

Happy new year everyone We are a small post-production house and switched to TrueNAS around 6 months ago. Everything runs well but we could not get rid of an annoying issue when browsing a folder containing a large number of files stalls both the client and the server. While the stall happens...

www.truenas.com

- Even accessing the folder in Finder stalls the server and Mac client with SMBD on 100% on one core. Sometimes takes around 20minutes to resolve
- Deleting directories with many files is impossible. We workaround and delete them on the NAS with midnight commander.
- Also seeing random disconnects sometimes when video editing with image sequences. We have one server running Core and switching to AFP helps.

I have come to the conclusion that TrueNAS' SMB implementation is partially broken for Mac. IDK if they fixed this in their enterprise version.
We had a QNAP NAS before and while it had other issues it did not show such problems. So this seems not to be an insoluble problem.

 

[Nicolas_Studiokgb]

H Simon
"Glad" to see that other users also have the issue. I was wondering what was wrong in my setup.
It seems there is an issue between mac (specially the finder) and truenas if using SMB. It's a metadata issue as if you disable apple extensions and alternate data streams you get a clearly faster responses at the cost of some compatibility or metadata issue (?) certainly I don't know.
I don't know if it's a difference between enterprise and community version. I hope not as it not advertised to be different on this point.

 

[simonj]

H Simon
"Glad" to see that other users also have the issue. I was wondering what was wrong in my setup.
It seems there is an issue between mac (specially the finder) and truenas if using SMB. It's a metadata issue as if you disable apple extensions and alternate data streams you get a clearly faster responses at the cost of some compatibility or metadata issue (?) certainly I don't know.
I don't know if it's a difference between enterprise and community version. I hope not as it not advertised to be different on this point.

Hi Nicolas

I totally agree with you. It's a metadata issue, mostly limited to Apple Finder - maybe not only as we often get disconnects when accessing folders with image sequences with Davinci Resolve
Actually your thread was super helpful. I was too narrow minded to consider turning off Apple extensions. Just did a quick test with extensions and alternate data streams off: wrote around 5K of image files into a directory and the directory listed almost immediately in Finder without the CPU spiking. Earlier this could have easily stalled the server and Mac for several minutes. Will do some more in-depth testing.

I didn't find any problems, lower performance or other disadvantages with extensions and alternate streams disabled. Did you find any?

Still seems very odd that the Apple Extensions, which are supposed to help with Mac clients are causing those problems.
I also assume that the Enterprise version is the same but maybe they do some custom tuning to make this work. I find it hard to imagine that professional clients would accept those troubles with Mac clients.

 

[Nicolas_Studiokgb]

Hi Simon
The main difference to be checked is that without Alternate data stream if my understanding is good it would write some hidden files along the main file (begining with . ) where to put some info.
I think you can check by showing hidden files on the finder (or check locally in truenas shell)

Yes I also think it's not intended that enabling those setting worsen performance. I also hope they are working on finding a solution. I had no news for a while about this problem.

In the first place I was about to order some server and enterprise version from Ix, but for financial reason we have been forced to delay investment on this side so I decided to run some more test with some server hardware we already had.
This issue, if unsolved, will prevent me to "buy to try" enterprise version as if nobody can confirm it works as intended with mac client.

I think that for Ix, have the community version work best as possible is the best advertising campaign for their commercial product. The other way would not make any sense. In tech world it always verify that if the free/trial/community version does not work, the paid one won't do better (except clearly stated that some fonctionnality would be disabled or limited in such version of course).

Hope this would find a solution as Truenas is on other points clearly way ahead any competition.

 

[anodos]

Alternate data streams are used for storing MacOS metadata and generally shouldn't be disabled. I'm currently working on other issues related to SMB, and will eventually get to investigating this.

 

[awasb]

[...] I find it hard to imagine that professional clients would accept those troubles with Mac clients.

Even though I can't imagine that anyone at iX is intentionally de-optimizing products (individual optimization costs, of course), your above notion must literally apply to Apple.

The SMB standard exists. It is/was just not sufficient for Apple. It has/had to be "extended" proprietary with dubious features. And then the standard is/was "extended", too, so that the specifics of Apple also work. And then Apple changes with (felt) every minor release a small thing: At the implementation, the configuration ... without any documentation/instructions, of course.

So ... I think it's wrong to blame the OSS community for any performance problems. Look at other projects: OpenSolaris/Illumos or an average Linux distribution with Apple clients. Does it run perfectly out of the box or are there also "adjustments" necessary? I wouldn't say anything if Apple itself still had a product in its portfolio that covered these functions. But where is the file server with Xsan configurator that I can click into my shopping cart?

Honestly, just because it's Apple, we put up with it.

 

[anodos]

If you're interested you can test a non-default parameter I put into U4 for testing purposes (an optimization that hasn't been characterized yet). fruit:streamname_optimization = true. IIRC this removed some amplification of file opens when MacOS clients were trying to read streams. But that's just something I noticed when looking at vfs_fruit a while back.

 

[Nicolas_Studiokgb]

Hello @anodos
Thanks for your feedback :)
Yes I think we should not disable ADS for Mac Os shares. (Testing shows better perf but it was just to investigate our issue)

I'll test the parameter and let you know if it give us some better perf.

I won't speak here about what I think about Apple but believe me, it's nothing good... Us users are captive of their commercial wills... They should learn more what "pro" means... lol. (And Devs clearly must live a nightmare for a few years...)

 

[Nicolas_Studiokgb]

Hi @anodos
Should I add this fruit:streamname_optimization = trueat SMB service level or at share level as auxiliary parameter ?
Thanks

 

[Nicolas_Studiokgb]

or in tunables ;)

 

[Davvo]

vfs_fruit should be a tunable iirc.

 

[Nicolas_Studiokgb]

Hello, good news ahead :)
So on a newly created pool with this news setting fruit:streamname_optimization = true added as aux parameter at share level
We can clearly see it's better than without (I will double check to be sure)
Demand metadata is clearly lower, smbd process goes mainly around 60% (instead of 100%)

Copy is around 5 minutes
Delete is around 7 minutes (so half the time we had before)
We are not as fast as between 2 macs but we are getting closer.. GOOD GOOD GOOD :)

root@truenas[~]# testparm -s
Load smb config files from /usr/local/etc/smb4.conf
Loaded services file OK.
Weak crypto is allowed

Server role: ROLE_STANDALONE

# Global parameters
[global]
aio max threads = 2
bind interfaces only = Yes
disable spoolss = Yes
dns proxy = No
enable web service discovery = Yes
kernel change notify = No
load printers = No
logging = file
max log size = 5120
nsupdate command = /usr/local/bin/samba-nsupdate -g
registry shares = Yes
restrict anonymous = 2
server multi channel support = No
server role = standalone server
server string = TrueNAS Server
unix extensions = No
idmap config *: range = 90000001-100000000
fruit:nfs_aces = No
rpc_server:mdssvc = disabled
rpc_daemon:mdssd = disabled
idmap config * : backend = tdb
directory name cache size = 0
dos filemode = Yes


[test]
access based share enum = Yes
ea support = No
mangled names = no
path = /mnt/RAID0/test
read only = No
smbd max xattr size = 2097152
vfs objects = catia fruit streams_xattr ixnas zfs_core aio_fbsd
fruit:streamname_optimization = true
fruit:resource = stream
fruit:metadata = stream
fruit:encoding = native
nfs4:chown = true
ixnas:dosattrib_xattr = false
root@truenas[~]#

 

[Nicolas_Studiokgb]

Another test with 670 GB 55k items

Copy to server 21 minutes
Delete 26 minutes.

Same on copy and less than half the time than before on delete.

Better better :)

Delete from 19:02

 

[anodos]

TrueNAS-13.0-CUSTOM_SAMBA_PORT_UPDATE_01-manual-update-unsigned.tar

drive.google.com

I spotted an inefficiency in file / dir deletion in case of streams. Assuming that our config hasn't been hacked in such a way that streams are being written to files rather than xattrs the above should significantly improve perf on deletion.

NOTES:
0. update file is based on 13-stable MASTER (not through QE). It is provided for validation purposes _only_.
1. this is a tarball and should be copied as-is to NAS
2. update may be installed via freenas-update <path to tar>
3. new boot environment should be visible in beadm list output with an "R" next to it.
4. if (3), then reboot NAS.

This will only improve deletion perf in cases where streams are present on files. For general metadata perf improvements (reduction of opens, shared mode lock contention, etc) we'd have to upgrade from Samba 4.15 to 4.18, which is a bridge too far for 13.0 U release.