Finder doing weird things with SMB. 10Gbps Benchmarks inside [SMB vs NFS: FIGHT!]

[vicmarto]

Hello my friends!

"Help me Obi Wan Kenobi, you're my only hope". Well, I feel now like Leia, but of course, with much less sex appeal!

I was planning to left NFS and embrace SMB but... It's impossible: The macOS Finder do so many weird things that it's just impossible. The worst is when working with directories with big quantities of files. For example:

3 Tests were doing with a specialy complicated folder consisting of 23,699 small files, with an average file size of 792KB, and a total weigh of 18.77GB.

1) Copy the folder from the Mac SSD to the share:​

SMB: 54MB/s​

NFS: 271MB/s​

2) Duplicate the folder in the share:​

SMB: 16MB/s [!!!!!!!!!]​

NFS: 109MB/s​

3) Open the folder in the share:​

SMB: 01m 26s [!!!!!!!!!]​

NFS: 17s​

The real problem here is the Finder, because if the same tests are repeated but using the macOS's terminal instead, the differences are not so big:

1) Copy the folder from the Mac SSD to the share:​

SMB: 122MB/s​

NFS: 214MB/s​

2) Duplicate the folder in the share:​

SMB: 50MB/s​

NFS: 101MB/s​

Yes my friends, using the Finder 16MB/s, and using the Terminal 50MB/s........



The SAMBA configuration is all default, except:

Services > SMB​

Auxiliary Parameters:​

case sensitive = yes​

strict sync = no​

store dos attributes = no​

map archive = no​

map hidden = no​

map readonly = no​

map system = no​

The SMB test was repeated but this time using vfs_fruit (and the rest of configuration the same as before)... With even worse results:

1) Copy the folder from the Mac SSD to the share:​

SMB with vfs_fruit: 53MB/s​

SMB without vfs_fruit: 54MB/s​

2) Duplicate the folder in the share:​

SMB with vfs_fruit: 12MB/s​

SMB without vfs_fruit: 16MB/s​

3) Open the folder in the share:​

SMB with vfs_fruit: 03m 01s​

SMB without vfs_fruit: 01m 26s​

Are the SMB and the Finder really this awful combination? There is nothing we can do? Maybe I misconfigured something?



##########################################################################################
Things I have tried without success:
· NAS

aio write size = 0 + aio read size = 0​

smb2 leases = no​

· Mac

net.inet.tcp.delayed_ack=0 /etc/sysctl.conf -----> Performance is worse consistently, unlike other reports all over the internet​

signing_required=no /etc/nsmb.conf ---> Didn't make a difference in my case, at least in my macOS Mojave 10.14.3 it was "no" at default. Forcing it to "yes" makes a huge difference (to bad)​

##########################################################################################
About SPECS:
· NAS [FreeNAS-11.2-U2.1 (no autotune) (clean install, clean configuration)]

Supermicro X10SLM+-F​

Intel i3-4330​

2x Hynix HMT41GU7AFR8A (16GB RAM ECC)​

Intel X550-T1 (directly connected to a Mac mini Aquantia AQC107-AFW client, without using jumbo frames)​

Be Quiet! Straight Power 10 500W CM​

2x SSD Samsung 860 EVO 1TB (configured as a striped vdev, to maximize the output for the test)​

· Mac [macOS Mojave 10.14.3]

Mac Mini (2008) basic upgraded with "10Gbps Ethernet", "3.2GHz 6‑core i7" and "16GB RAM"​

##########################################################################################
About the test CONDITIONS:

· NAS was restarted between each test​

· LAN bandwidth is not the problem:​

Code:

$ iperf -c 192.168.10.250
------------------------------------------------------------
Client connecting to 192.168.10.250, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  4] local 192.168.10.100 port 53201 connected with 192.168.10.250 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  10.9 GBytes  9.37 Gbits/sec

· Dataset relevant properties:​

Code:

recordsize         128K
compression        off (To eliminate the compression variable from the test)
atime              off
sync               disabled (To eliminate the sync variable from the test)
casesensitivity    insensitive (To use "case sensitive = yes" in SAMBA)

· Dataset creation using GUI:​

zfast/test/nfs was created as "Share Type: Unix"​

zfast/test/smb was created as "Share Type: Windows"​

 

[Basil Hendroff]

Have a look at the section 'Speed things up in directories with lots of files' in SMB Tips and Tricks

 

[anodos]

Try disabling ds stores in finder:
defaults write com.apple.desktopservices DSDontWriteNetworkStores -bool TRUE
Then log out of your macOS account and log back in. During the copy, look at top output and gstat output to see if you're experiencing CPU or disk saturation.

 

[anodos]

Have a look at the section 'Speed things up in directories with lots of files' in SMB Tips and Tricks

I went ahead and removed that bit of advice. That item doesn't yield any performance improvements in real life (IIRC), and I was seeing a lot of getcwd() calls returning -1 so I need to investigate it further when I have time. I wrote quite a few things here before I worked for iX, and some of them are wrong. This was one of those things.

 

[seanm]

I went ahead and removed that bit of advice.

What part was that? (I've relied heavily on that page for my configuration... wondering if I should change something now...)

 

[anodos]

What part was that? (I've relied heavily on that page for my configuration... wondering if I should change something now...)

Just the part about case-insensitive lookups. I think it needs further research into why getcwd() is failing on case insensitive datasets. (perhaps has to do with path component being case sensitive (/mnt) ).

 

[seanm]

My pool and all datasets are case *in*sensitive. Per your (previous) suggestions, I've set:
- mangled names = illegal
- case sensitive = yes

If I understand right, you're now saying we should keep the former, but ditch the latter?

Sorry to divert your thread vicmarto... I've also been performance testing SMB, and have found that performance with many small files is pretty bad compared to single large files. I've been using Terminal, and not even tried Finder. I'm still unsure if I should be expecting better with my hardware or not. Guess that doesn't help you, but you are not alone. :)

I do encourage you to file a bug at https://bugreport.apple.com if Finder is so much worse than Terminal.

 

[vicmarto]

Try disabling ds stores in finder:
defaults write com.apple.desktopservices DSDontWriteNetworkStores -bool TRUE
Then log out of your macOS account and log back in. During the copy, look at top output and gstat output to see if you're experiencing CPU or disk saturation.

Thanks anodos for your reply. Unfortunately disabling .DS_Store is not enough to justify a performance difference so huge between SMB and NFS...

Are your observation/test similar to mine? Is the Finder + SMB combination so terrible? There is nothing we can do?

Which is the "correct/supported/best" FreeNAS way to create a dataset for share exclusively to a MacOS client by SMB? Share type: Windows? Maybe share type: Unix? Both of them are ok?

And what about the VFS objects? zfs_space zfsacl fruit streams_xattr? Or maybe only zfs_space fruit streams_xattr?

What more?

Sorry to divert your thread vicmarto... I've also been performance testing SMB, and have found that performance with many small files is pretty bad compared to single large files. I've been using Terminal, and not even tried Finder. I'm still unsure if I should be expecting better with my hardware or not. Guess that doesn't help you, but you are not alone. :)

No problem seanm, we are in the same boat, maybe we can work together to get more performance from SMB with many small files.

Finder is a totally different animal than Terminal, unfortunately.

Using this command:

Code:

mkdir 23699-small-files && cd 23699-small-files && dd if=/dev/urandom bs=1024 count=18958834 | split -a 4 -b 800k - file.

Will result in a test folder similar to mine: just one folder full of 23,699 files of 800KB each one, for a total weight of around 20GB. If you try to duplicate this folder using Finder, are your times similar to mine?

My pool and all datasets are case *in*sensitive. Per your (previous) suggestions, I've set:
- mangled names = illegal
- case sensitive = yes

If I understand right, you're now saying we should keep the former, but ditch the latter?

I'm really interested too in this question. Which are now the recommended parameters to maximize the SMB performance with many small files?

Now I'm using:

Code:

mangled names = illegal
case sensitive = yes
strict sync = no
store dos attributes = no
map archive = no
map hidden = no
map readonly = no
map system = no

mangled names = illegal​

case sensitive = yes (+ casesensitivity=insensitive) ---> As was recommended here: SMB Tips and Tricks​

​

strict sync = no ---> As recommended here: AFP vs SMB 10GbE network speed problem in FreeNAS​

​

store dos attributes = no​

map archive = no​

map hidden = no​

map readonly = no​

map system = no ---> As recommended here: CIFS directory browsing slow? Try this (but without the ea support = no part, because it's needed by streams_xattr, am I right?)​

 

[Basil Hendroff]

Is this a related post? Both seem to revolve around Mac clients and SMB performance. SMB performance drops over time on 11.2.U2

 

[Basil Hendroff]

@vicmarto What is SMB performance like when using a Windows client? Is it similar? Why I ask is because I'm a Windows shop and haven't noticed the issues described in this thread with FreeNAS 11.2-U2.1 at all.

 

[vicmarto]

@Seymour Butt, I have repeated the "duplicate test" with a Windows 10 client, with... disappointing results:

Duplicate the folder in the share:

Windows 10 SMB: 41MB/s​

macOS Finder SMB: 16MB/s​

macOS Shell SMB: 50MB/s​

​

macOS Finder NFS: 109MB/s​

Windows 10 is quicker than Finder, as expected. But, much slower than NFS when treating with many small files. Please note that in the case of big files there is no problem and performance is on par between SMB and NFS, the problem is with many small files.

#######################################################################################

By "duplicate test" I mean, copy que 23699-small-files folder from the share, and paste it in this same share. The 23699-small-files folder is done with this command in the FreeNAS Shell:

Code:

mkdir 23699-small-files && cd 23699-small-files && dd if=/dev/urandom bs=1024 count=18958834 | split -a 4 -b 800k - file.

Please, someone can repeat the "duplicate test" and share its results? Thanks.

 

[Basil Hendroff]

My test results: Mac and Windows client duplicate speeds of large and small files in an SMB share.

Notes:

1. I'm not familiar enough with the Mac OS to do the macOS Shell SMB or Finder NFS test, so these have been omitted from the table.
2. Clients and server coexist on a 1Gb/s network.
3. Parameters have not been tuned on datasets and shares.
4. Tests were done against approximately 18 GB each of large files and small files.
5. A low specced Mac was all I had to do the comparison with (see attachment). It's possible the OS X performance figures would have been better had I used a more current and better specced machine.

 

[Basil Hendroff]

Are the SMB and the Finder really this awful combination? There is nothing we can do? Maybe I misconfigured something?

I think it's pretty much par for the course and an internet search supports your observations (and my test results). eg.
AFP vs. SMB and NFS file sharing for network clients
NFS, AFP, SMB… Advantages and drawbacks on a Mac OS system
Slow transfer speeds with many small files

 

[vicmarto]

View attachment 29652

Thank you very much @Seymour Butt, just 2 questions:

1) Are this 5.14 number megabytes per second?
2) Your small files test is duplicating the 23699-small-files folder?