Hi Everyone
Processor:
Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz (12 cores)
Memory:
64 GiB
HDD:
OS: 2x Samsung 883 DCT Series - 240GB SATA 2.5" MZ-7LH240NE
Data: 4x Samsung 883 DCT Series - 3.84TB SATA 2.5" MZ-7LH3T8NE
Raid:
Z1
NIC:
Solareflare Flareon SFN7002F Dual-Port 10GbE
I am having issues with performance on FreeNAS-11.2-U5 particularly with cifs and small files. I am looking to migrate a windows file share server based that hosts small files and folders. Primarily csv files approx. 29.5 KB sample folder I was testing with has approx. 38,500 files and is 1.8GB. The issue I am having is related to a much larger folder with millions of files but I was using this sample set to test the performance of smb on Freenas vs a windows share. I get good performance for large files but not as good as a windows share. Below are the times I am seeing using robocopy. I know samba slows down when it uses small files but I did not expect for performance to be this bad. I have also included a copy of the smb.conf. Ive tried multichannel but it doesn't seem to be working with RSS Receive Side Scaling. If you see the output from the client server using Get-SmbMultichannelConnection, I see its making a multichannel connection, but Client RSS Capable returns False even though the Solarflare card supports RSS. The Windows file share returns True. Furthermore when transferring I also run vmstat -i to see the interrupts and I don't see the interrupts being spread accross the tx/rx queues. Running htop shown smb pegged at 100%. I have scoured the forums for answers and tested tons of parameters to no avail. Am I missing something or is this the type of performance that is expected? Please let me know if I missed any details. Thanks in advance!
Processor:
Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz (12 cores)
Memory:
64 GiB
HDD:
OS: 2x Samsung 883 DCT Series - 240GB SATA 2.5" MZ-7LH240NE
Data: 4x Samsung 883 DCT Series - 3.84TB SATA 2.5" MZ-7LH3T8NE
Raid:
Z1
NIC:
Solareflare Flareon SFN7002F Dual-Port 10GbE
I am having issues with performance on FreeNAS-11.2-U5 particularly with cifs and small files. I am looking to migrate a windows file share server based that hosts small files and folders. Primarily csv files approx. 29.5 KB sample folder I was testing with has approx. 38,500 files and is 1.8GB. The issue I am having is related to a much larger folder with millions of files but I was using this sample set to test the performance of smb on Freenas vs a windows share. I get good performance for large files but not as good as a windows share. Below are the times I am seeing using robocopy. I know samba slows down when it uses small files but I did not expect for performance to be this bad. I have also included a copy of the smb.conf. Ive tried multichannel but it doesn't seem to be working with RSS Receive Side Scaling. If you see the output from the client server using Get-SmbMultichannelConnection, I see its making a multichannel connection, but Client RSS Capable returns False even though the Solarflare card supports RSS. The Windows file share returns True. Furthermore when transferring I also run vmstat -i to see the interrupts and I don't see the interrupts being spread accross the tx/rx queues. Running htop shown smb pegged at 100%. I have scoured the forums for answers and tested tons of parameters to no avail. Am I missing something or is this the type of performance that is expected? Please let me know if I missed any details. Thanks in advance!
Code:
COPY FROM WINDOWS 2012 Host to Freenas SMB SHARE: 1 file (5.6GB) robocopy source destination /mir /mt:128 /sec /r:0 /w:0 /ns /nc /nfl /ndl /log:c:\freenas.txt Total Copied Skipped Mismatch FAILED Extras Dirs : 1 1 0 0 0 0 Files : 2 1 1 0 0 0 Bytes : 5.662 g 5.662 g 26.0 k 0 0 0 Times : 0:00:41 0:00:10 0:00:00 0:00:10 Ended : Tuesday, August 27, 2019 9:18:02 AM COPY FROM WINDOWS 2012 HOST > Windows 2012 RAID 5 SSD SHARE: 1 file (5.6GB) robocopy source destination /mir /mt:128 /sec /r:0 /w:0 /ns /nc /nfl /ndl /log:c:\freenas.txt Total Copied Skipped Mismatch FAILED Extras Dirs : 1 1 0 0 0 0 Files : 2 2 0 0 0 0 Bytes : 5.662 g 5.662 g 0 0 0 0 Times : 0:00:10 0:00:02 0:00:00 0:00:02 Ended : Tuesday, August 27, 2019 9:19:37 AM COPY FROM WINDOWS 2012 Host to Freenas SMB SHARE: (38477 files 1.8GB) robocopy source destination /mir /mt:128 /sec /r:0 /w:0 /ns /nc /nfl /ndl /log:c:\freenas.txt Total Copied Skipped Mismatch FAILED Extras Dirs : 333 333 332 0 0 0 Files : 38477 38476 1 0 0 0 Bytes : 1.814 g 1.814 g 26.0 k 0 0 0 Times : 4:30:01 0:02:06 0:00:00 0:00:01 Ended : Tuesday, August 27, 2019 9:33:53 AM COPY FROM WINDOWS 2012 Host to Freenas SMB SHARE: (38477 files 1.8GB) robocopy source destination /mir /mt:128 /sec /r:0 /w:0 /ns /nc /nfl /ndl /log:c:\freenas.txt Total Copied Skipped Mismatch FAILED Extras Dirs : 333 333 332 0 0 0 Files : 38477 38476 1 0 0 1 Bytes : 1.814 g 1.814 g 26.0 k 0 0 5.662 g Times : 0:17:31 0:00:08 0:00:00 0:00:00 Ended : Tuesday, August 27, 2019 9:25:37 AM OUTPUT FROM CLIENT: Get-SmbMultichannelConnection Server Name Selected Client IP Server IP Client Interface Index Server Interface Index Client RSS Capable Client RDMA Capable ----------- -------- --------- --------- ---------------------- ---------------------- ------------------ ------------------- FREENAS True 10.24.14.70 10.24.14.211 17 8 False False WINDOWS FILE SHARE True 10.24.14.70 10.24.14.164 17 19 True False smb.conf [global] aio max threads = 2 allow trusted domains = No client ldap sasl wrapping = plain deadtime = 15 disable spoolss = Yes dns proxy = No domain master = No dos charset = CP437 hostname lookups = Yes kernel change notify = No lm announce = Yes load printers = No local master = No logging = file max log size = 51200 max open files = 1883558 nsupdate command = /usr/local/bin/samba-nsupdate -g obey pam restrictions = Yes preferred master = No printcap name = /dev/null realm = DOMAIN.COM security = ADS server min protocol = SMB2_02 server multi channel support = Yes server role = member server server string = FreeNAS Server template shell = /bin/sh winbind cache time = 7200 winbind enum groups = Yes winbind enum users = Yes winbind offline logon = Yes winbind refresh tickets = Yes workgroup = DOMAIN idmap config domain: range = 20000-90000000 idmap config domain: backend = rid idmap config *: range = 90000001-100000000 idmap config * : backend = tdb acl allow execute always = Yes create mask = 0666 directory mask = 0777 directory name cache size = 0 dos filemode = Yes strict locking = No [root_shares] aio write size = 0 path = "/mnt/pool1/root_shares" read only = No veto files = /.snapshot/.windows/.mac/.zfs/ vfs objects = zfs_space zfsacl streams_xattr zfsacl:acesort = dontcare nfs4:chown = true nfs4:acedup = merge nfs4:mode = special