I have been trying to troubleshoot this for the past weekend and would like to hear your experience/advice
I got a relatively powerful FN box (spec in signature) and a ESXi box (dual 2667 v2, 128G RAM, a Dell 10GB NIC made by boardcom) connected by the a 10G switch. The FN box servers ESXi host as an iSCSI target, which is backed by the mirror pool. There is another pool of a 8 disk raid Z2 vdev, that backs SMB shares. The iSCSI part works great (~800MB/s seq, ~40MB/s 4K @QD1, ~400MB/s 4K @QD32), however the SMB share does not work very well with my Ubuntu VMs:
1. Server side copy never seems to work. According to Samba wiki this is suppose to work on client Linux OS as well windows ones, but not sure how relevant it is to FN. Just wondering if any of you have been able to get it to work on a Linux client...
2. Even without the server copy offload, SMB shares are still unreasonably slow on Ubuntu VMs. Usually about ~70-200MB/s read/write (measured by using coping ~10GB files using rsync), while on windows(10 or 2016) VMs it can range from 300MB/s to 1000MB/s read/write/server side copy (copying same files using windows explorer).
Here are something I have tried:
1. Iperf test shows >9 Gbit/s both ways between FN box and Ubuntu VM
2. Ran iozone test on Ubuntu VM against the SMB mount point. The result shows almost 1000MB/s read/write, this clearly results from highly compressible test data being used. But also suggests that the bottleneck is somewhere is FN the box? unless Ubuntu is also doing compression on the client side.
3. During testing the other VMs are relatively idle, only a few windows server VM providing active directory service. Both pools are almost empty (less than 5% used)
Here is my fstab
My smb4.conf
I got a relatively powerful FN box (spec in signature) and a ESXi box (dual 2667 v2, 128G RAM, a Dell 10GB NIC made by boardcom) connected by the a 10G switch. The FN box servers ESXi host as an iSCSI target, which is backed by the mirror pool. There is another pool of a 8 disk raid Z2 vdev, that backs SMB shares. The iSCSI part works great (~800MB/s seq, ~40MB/s 4K @QD1, ~400MB/s 4K @QD32), however the SMB share does not work very well with my Ubuntu VMs:
1. Server side copy never seems to work. According to Samba wiki this is suppose to work on client Linux OS as well windows ones, but not sure how relevant it is to FN. Just wondering if any of you have been able to get it to work on a Linux client...
2. Even without the server copy offload, SMB shares are still unreasonably slow on Ubuntu VMs. Usually about ~70-200MB/s read/write (measured by using coping ~10GB files using rsync), while on windows(10 or 2016) VMs it can range from 300MB/s to 1000MB/s read/write/server side copy (copying same files using windows explorer).
Here are something I have tried:
1. Iperf test shows >9 Gbit/s both ways between FN box and Ubuntu VM
2. Ran iozone test on Ubuntu VM against the SMB mount point. The result shows almost 1000MB/s read/write, this clearly results from highly compressible test data being used. But also suggests that the bottleneck is somewhere is FN the box? unless Ubuntu is also doing compression on the client side.
3. During testing the other VMs are relatively idle, only a few windows server VM providing active directory service. Both pools are almost empty (less than 5% used)
Here is my fstab
Code:
//freenasFQDM/Download-Temp /mnt/Download-Temp cifs credentials=/path/.smbcredentials,iocharset=utf8,uid=user,gid=group 0 0
My smb4.conf
Code:
[global] server min protocol = SMB2_02 server max protocol = SMB3 interfaces = 127.0.0.1 192.168.21.25 bind interfaces only = yes encrypt passwords = yes dns proxy = no strict locking = no aio max threads = 2 oplocks = yes deadtime = 15 max log size = 51200 private dir = /var/db/samba4/private max open files = 3772379 logging = file load printers = no printing = bsd printcap name = /dev/null disable spoolss = yes getwd cache = yes guest account = nobody map to guest = Bad User obey pam restrictions = yes ntlm auth = no directory name cache size = 0 kernel change notify = no nsupdate command = /usr/local/bin/samba-nsupdate -g server string = FreeNAS Server ea support = yes store dos attributes = yes lm announce = yes hostname lookups = yes acl allow execute always = false dos filemode = yes multicast dns register = yes domain logons = no idmap config *: backend = tdb idmap config *: range = 90000001-100000000 server role = member server workgroup = MYITLAB realm = AD.MYITLAB.local security = ADS client use spnego = yes local master = no domain master = no preferred master = no ads dns update = no winbind cache time = 7200 winbind offline logon = yes winbind enum users = yes winbind enum groups = yes winbind nested groups = yes winbind use default domain = yes winbind refresh tickets = yes idmap config MYITLAB: backend = rid idmap config MYITLAB: range = 20000-90000000 allow trusted domains = yes client ldap sasl wrapping = seal template shell = /bin/sh template homedir = /mnt/Z2/HomefolderShare/%D/%U netbios name = FREENAS create mask = 0666 directory mask = 0777 client ntlmv2 auth = yes dos charset = CP437 unix charset = UTF-8 log level = 1 [Download-Complete] path = "/mnt/Z2/Download-Complete" printable = no aio write size = 0 veto files = /.snapshot/.windows/.mac/.zfs/ writeable = yes browseable = yes access based share enum = yes vfs objects = shadow_copy_zfs zfs_space zfsacl streams_xattr hide dot files = yes hosts allow = 192.168.21.0/24 192.168.30.0/24 192.168.50.30 192.168.50.0/24 guest ok = yes nfs4:mode = special nfs4:acedup = merge nfs4:chown = true zfsacl:acesort = dontcare [Download-Temp] path = "/mnt/Mirrors/Download-Temp" printable = no aio write size = 0 veto files = /.snapshot/.windows/.mac/.zfs/ writeable = yes browseable = yes access based share enum = yes vfs objects = shadow_copy_zfs zfs_space zfsacl streams_xattr hide dot files = yes hosts allow = 192.168.21.0/24 192.168.30.0/24 192.168.50.30 192.168.50.0/24 guest ok = yes nfs4:mode = special nfs4:acedup = merge nfs4:chown = true zfsacl:acesort = dontcare [homes] valid users = %U path = "/mnt/Z2/HomefolderShare/%D/%U" comment = Home Directories printable = no aio write size = 0 veto files = /.snapshot/.windows/.mac/.zfs/ writeable = yes browseable = no access based share enum = yes recycle:repository = .recycle/%U recycle:keeptree = yes recycle:versions = yes recycle:touch = yes recycle:directory_mode = 0777 recycle:subdir_mode = 0700 vfs objects = shadow_copy_zfs zfs_space zfsacl streams_xattr recycle crossrename hide dot files = yes guest ok = no nfs4:mode = special nfs4:acedup = merge nfs4:chown = true zfsacl:acesort = dontcare [Z2 shares] path = "/mnt/Z2/Shares" printable = no aio write size = 0 veto files = /.snapshot/.windows/.mac/.zfs/ writeable = yes browseable = yes access based share enum = yes vfs objects = shadow_copy_zfs zfs_space zfsacl streams_xattr hide dot files = yes guest ok = no nfs4:mode = special nfs4:acedup = merge nfs4:chown = true zfsacl:acesort = dontcare