- Joined
- Mar 6, 2014
- Messages
- 9,554
This is a brief write-up of some testing I did. Feel free to make suggestions for further experimentation.
Experiment setup:
Server 1
Server 2
Folder to be copied
Network
Raw results
Observations:
Conclusions
Errata
Disabling case sensitivity on a samba share involves two steps:
For those interested, here are some stats I pulled during the middle of the transfer:
Experiment setup:
Server 1
- Supermicro X8DTH, Xeon E5645, 48GB ECC Memory, 12 * 7200 RPM Seagate Constellation drives configured in 6 mirrored vdevs.
Server 2
- Supermicro X9SCI, Xeon E3-1220, 32GB ECC Memory, 8 * 7200 RPM Seagate Constellation drives configured in single RAIDZ2 vdev.
- Windows 10. 8GB RAM. Crucial BX100 SSD. Pentium G3250 processor. No specialized security software. Realtek gigabit NIC.
Folder to be copied
- Size - 1.17GiB, Contains - 91,145 Files, 7,963 Folders. Average file size = 13.78KB. Unzipped copies of the samba source from various samba versions.
- Robocopy command - Code:
robocopy C:\Users\anodos\Downloads\SAMBAS\ \\FREENAS\CaseInsensitive\SAMBAS /MIR /COPY:DT /FFT /LOG:C:\Users\anodos\Desktop\Robocopy_FN_CI.txt"
Network
- Gigabit. All devices on same switch.
- AD functional level Server 2008R2.
- AD member servers
- Server max protocol - SMB3_00.
- Log level 1
- AIO was enable by setting "aio write size = 1"
- vfs_aio_pthread was tested using the default 100 threads and subsequently reduced to 10 threads.
Raw results
Code:
Server 1 - Unmodified setup
------------------------------------------------------------------------------
Total Copied Skipped Mismatch FAILED Extras
Dirs : 7964 7963 1 0 0 0
Files : 91145 91011 134 0 0 0
Bytes : 1.177 g 1.172 g 4.48 m 0 0 0
Times : 0:10:19 0:09:12 0:00:00 0:01:06
Speed : 2279616 Bytes/sec.
Speed : 130.440 MegaBytes/min.
Ended : Monday, May 2, 2016 6:33:40 PM
Server 1 - with AIO write size = 1
------------------------------------------------------------------------------
Total Copied Skipped Mismatch FAILED Extras
Dirs : 7964 7963 1 0 0 0
Files : 91145 91011 134 0 0 0
Bytes : 1.177 g 1.172 g 4.48 m 0 0 0
Times : 0:10:20 0:09:13 0:00:00 0:01:07
Speed : 2276990 Bytes/sec.
Speed : 130.290 MegaBytes/min.
Ended : Monday, May 2, 2016 6:53:06 PM
Server 1 - with all VFS objects turned off and AIO write size = 1
------------------------------------------------------------------------------
Total Copied Skipped Mismatch FAILED Extras
Dirs : 7964 7963 1 0 0 0
Files : 91145 91011 134 0 0 0
Bytes : 1.177 g 1.172 g 4.48 m 0 0 0
Times : 0:10:13 0:09:06 0:00:00 0:01:06
Speed : 2303966 Bytes/sec.
Speed : 131.834 MegaBytes/min.
Ended : Monday, May 2, 2016 7:28:43 PM
Server 1 - with Case Sensitivity modifications and AIO write size = 1
------------------------------------------------------------------------------
Total Copied Skipped Mismatch FAILED Extras
Dirs : 7964 7963 1 0 0 0
Files : 91145 91011 134 0 0 0
Bytes : 1.177 g 1.172 g 4.48 m 0 0 0
Times : 0:09:41 0:08:38 0:00:00 0:01:03
Speed : 2427587 Bytes/sec.
Speed : 138.907 MegaBytes/min.
Ended : Monday, May 2, 2016 7:48:17 PM
Server 2 - with Case Sensitivity modifcations and AIO write size = 1
------------------------------------------------------------------------------
Total Copied Skipped Mismatch FAILED Extras
Dirs : 7964 7963 1 0 0 0
Files : 91145 91011 134 0 0 0
Bytes : 1.177 g 1.172 g 4.48 m 0 0 0
Times : 0:09:24 0:08:27 0:00:00 0:00:57
Speed : 2481255 Bytes/sec.
Speed : 141.978 MegaBytes/min.
Ended : Monday, May 2, 2016 8:32:47 PM
Server 2- with Case Sensitivity modifications and aio_pthread vfs object enabled
------------------------------------------------------------------------------
Total Copied Skipped Mismatch FAILED Extras
Dirs : 7964 7963 1 0 0 0
Files : 91145 91145 0 0 0 0
Bytes : 1.177 g 1.177 g 0 0 0 0
Times : 0:13:00 0:12:03 0:00:00 0:00:57
Speed : 1747038 Bytes/sec.
Speed : 99.966 MegaBytes/min.
Ended : Tuesday, May 3, 2016 8:10:10 AM
Server 2 - with Case Senstivity modifications and receive buffer tuning bullsh*t
------------------------------------------------------------------------------
Total Copied Skipped Mismatch FAILED Extras
Dirs : 7964 7963 1 0 0 0
Files : 91145 91145 0 0 0 0
Bytes : 1.177 g 1.177 g 0 0 0 0
Times : 0:08:42 0:07:46 0:00:00 0:00:56
Speed : 2711848 Bytes/sec.
Speed : 155.173 MegaBytes/min.
Ended : Tuesday, May 3, 2016 9:38:54 AM
Server 2 - with Case Sensitivity and Windows Defender disabled - DOH!
------------------------------------------------------------------------------
Total Copied Skipped Mismatch FAILED Extras
Dirs : 7964 7963 1 0 0 0
Files : 91145 91145 0 0 0 0
Bytes : 1.177 g 1.177 g 0 0 0 0
Times : 0:07:53 0:06:59 0:00:00 0:00:54
Speed : 3015391 Bytes/sec.
Speed : 172.542 MegaBytes/min.
Ended : Tuesday, May 3, 2016 10:39:34 AM
Observations:
- Prior to disabling Windows Defender, it was using approx 25% CPU on the Windows 10 machine. Robocopy used approx 15%-20% CPU during copy.
- After disabling windows defender and enabling case sensitivity, htop showed CPU core consistently at 100%.
Conclusions
- I saw basically zero zil usage in all tests performed - you probably shouldn't waste money on one of these if you're using samba.
- No significant performance difference between a single 8-disk RAIDZ2 vdev and 6 mirrored vdevs.
- Enabling AIO did not appear to have a significant impact on write speed.
- Enabling vfs_aio_pthread had a deleterious effect on write speed for this workload.
- Disabling VFS modules did not appear to have a significant impact on write speed.
- Disabling oplocks did not appear to have a significant impact on write speed.
- Increasing the receive buffer in samba to 32KB may have had an impact (but it's arguably tuning for a benchmark). Note that samba devs consider modifying "socket options" to be "voodoo bullsh*t".
- Case sensitivity modifications appear to have increased write speed by approximately 10 Megabytes/minute.
- Disabling A/V heuristics increased write speed by approximately 20 Megabytes/minute.
- With tweaks I was only able to increase speed from 130 MegaBytes / minute to 172 MegaBytes /minute.
- Speed appears to be CPU-bound server-side (single core of E3-1220 pegged at 100%). Possible solution - need moar CPU.
Errata
Disabling case sensitivity on a samba share involves two steps:
(1) create a dataset with the property "case insenitive".
(2) add a share pointing to the dataset with the auxiliary parameter "case sensitive = true"
Note - I have not thoroughly tested these changes. Test on your own before putting into production.
(2) add a share pointing to the dataset with the auxiliary parameter "case sensitive = true"
Note - I have not thoroughly tested these changes. Test on your own before putting into production.
For those interested, here are some stats I pulled during the middle of the transfer:
Code:
Elapsed Times for PID 20321,
SYSCALL TIME (ns)
fstatfs 7906
madvise 8806
getdirentries 29591
fstatat 55799
close 184569
mkdir 206742
__getcwd 744294
pwrite 847921
getgroups 1006371
__sysctl 1425662
setgroups 1551697
ftruncate 1611680
setregid 1824582
getegid 2528751
geteuid 2531561
setreuid 2654942
utimensat 3582704
openat 4105631
writev 6103947
fstat 6241720
__acl_get_file 6363662
sendmsg 8157406
readv 8811186
fcntl 8972272
stat 9092675
lstat 19978389
poll 380015963
TOTAL: 478646429
CPU Times for PID 20321,
SYSCALL TIME (ns)
fstatfs 5990
madvise 7031
getdirentries 21938
fstatat 45195
close 145129
mkdir 200055
geteuid 304880
getgroups 339524
getegid 340312
__getcwd 513722
__sysctl 746940
pwrite 789925
setregid 867182
setgroups 914937
ftruncate 1519549
setreuid 1691843
utimensat 3377230
fstat 3622460
openat 3977775
poll 4262059
fcntl 5271314
writev 5340557
readv 5429879
__acl_get_file 5791786
sendmsg 7678999
stat 8030229
lstat 18113941
TOTAL: 79350381
Syscall Counts for PID 20321,
SYSCALL COUNT
madvise 11
fstatfs 12
mkdir 15
getdirentries 24
fstatat 103
pwrite 135
ftruncate 142
close 198
openat 232
utimensat 561
__getcwd 771
sendmsg 1101
__acl_get_file 1476
fstat 1640
writev 2054
__sysctl 2196
setgroups 2196
getgroups 2216
poll 2237
stat 2796
setregid 3302
setreuid 3302
readv 4332
lstat 5483
getegid 7718
geteuid 7726
fcntl 13099
TOTAL: 65078
Last edited:
