SOLVED 10Gb SMB Performance Mac OS 10.12 and 10.13

[dkusek]

I will start this off with I am very familiar with the nsmb.conf fix for signing issues with Mac OS. So the details:

On FreeNAS 11.0-U4, everything is normal. I get write speeds around 800Mb/s and read speeds of around 1000Mb/s. These are generated by using AJA system test running 64GB file transfers at 4k. Thats how we have been doing benchmarks for 10Gb ethernet performance since about 9.2.18.

Made the switch to FreeNAS-11.1-RELEASE. Speeds wont go over 70-75Mb/s but write speeds are still up around 1000Mb/s.

I have tried with compression on (default LZ4) and compression off. Same result. This system has 3 vdevs, 8 disks a piece with 0TB used. It is literally what we use for testing.
We also have a media server with 5 vdevs and get the same settings.

Does anyone know why there would be a cap with 11.1 on write speeds? Seems pretty weird. I thought it might be a 10.13.1 Mac issue, but again, ran the same tests on a 10.12 instance and got the same results.

Our next step will be to test on Windows 10 to confirm this is a server side issue. I have debugs saved from both the 11.0-U4 instance and the 11.1-RELEASE instance. The only real difference I found in the 11.1 debug was in the SMB dump.txt file. the field "server MAX protocol = SMB3" is no longer in the global configuration. Nor is there a place in Service > SMB to select the max protocol. On that note, I also tried adding that line in the global configuration in 11.1. That yields no chance. Please let me know if any specific logs would be valuable in diagnosing this further.

I also created an AFP share on the 11.1 instance. Speeds were write = 600Mb/s and read = 800Mb/s. With that, Im guessing then that this has something to do with SMB in 11.1.

 

[dkusek]

Just tested the same scenario in Windows 10. NO ISSUE. There is something that is specific to FreeNAS-11.1 and Mac clients (both 10.12 and 10.13) that is throttling write speeds from the client machine to around 70-Mb/s over a 10Gb connection. Windows write and read speeds are 900Mb/s and 850Mb/s respectively.

 

[dkusek]

To fix the signing issue in Mac OS clients since 10.12.5, one would have to put the parameter "signing_required=no" into the client machine. That would negate the signing requirement and thus drastically increase speed. This no longer works. That specific parameter does not seem to be recognized by Samba 4.7.0 which is what FreeNAS 11.1 is now running. I have tried other parameters per the Samba documentation; specifically "client signing = disabled" and "server signing = disabled". Neither of those seems to bring any change.

 

[dkusek]

FreeNAS 11.0-U4 with Samba version 4.6.8-GIT-7a91926 does not have any issue.

 

[dkusek]

It seems that this SMB signing fix : https://support.apple.com/en-us/HT205926 is no longer relevant

 

[dkusek]

DISREGARD ALL OF THIS...
It seems the problem is with AJA System test. We are not sure as to what the deficiency is as this has been a proven network testing tool for Mac OS. This can be closed out. I apologize if anyone has put any time or effort into this issue.

 

[dkusek]

For anyone that was following this and perhaps saw similar problems...
After following up with AJA, they instructed me to turn off "sync" on the dataset I was testing with. This was his response in the issue:

"What I suspect, rather than hardware issue, is that for some reason the new combination of software is causing the data AJA System Test is writing to all be O_Sync, which is in turn forcing it to flush immediately to disk rather than buffer in RAM and write sequentially, as it usually would. If you can, try to set sync=disabled on the dataset ("zfs set sync=disabled /tank/xyz"). You may not want that in production, but if it immediately brings performance with AJA System Test to expectations, it lets us know sync writes are the problem. Default is sync=standard (aka listen to what the OS/App tells it)."

Doing that resolved the issue immediately. As there is no problem on the FreeNAS side with moving normal data, I don't see this as a FreeNAS bug or anything. But good to know for anyone that uses AJA to test network speeds.

They, AJA, are putting a feature in the next application update to include the option to set/unset O_Sync. As they stated in a follow up, O_Sync should probably be defaulted to off as far as the test application goes.

 

[chandlerphilly]

OH MY GOSH. THANK THE LORD FOR PEOPLE WHO FOLLOW UP ON THEIR OWN PROBLEM ON FORUMS. Sorry for the caps but I'm on day 7 of dealing with this issue and you just SAVED. MY. ASS. It's so nice when flipping a switch solves a problem. I'm very relieved right now.

We don't use FreeNAS, but I did just finish building a custom ZFS file server for the video production company I work at. We also use the AJA tool for benchmarks from workstations because, well, that's the best option out there for our industry and comparing results. I'm always unsure about it's reliability but this is actually an interesting peek under the hood. Anyway, I've been having this exact issue, while all of our Windows machines had reasonable write speeds.

I was drifting further and further away from ZFS tuneables as helpful, given it was only affecting macOS systems with are known for crap SMB performance anyway. The confusing part is that this same server, when running the OS it shipped with, gets great write speeds on the EXT4 RAID it sets up. I was also getting reasonable speeds to an SSD-only test server I set up using the same exact setup as this new one.

sync=disabled immediately fixed this. I suppose this was a broad-stroke troubleshooting step I should have tried earlier, but I never thought the AJA tool itself would be the one to trigger sync writes (much less only on one operating system). But there you go. Now all I have to do is work on some much easier decisions about how to improve the write speed when sync is enabled. Thanks to you, I think all of the ghosts in the machine are gone and I can actually settle on this config being good for production. Thank you, thank you, thank you!

 

[lonelyzinc]

Was having issues with SMB not performing as fast as AFP for read speeds (nearly half as slow). Trying to figure out if there is a Samba option I need to enable. It's slow on both Mac and Windows

UPDATE: No voodoo magic required, simply enabling jumbo frames solved the SMB performance problems for me on macOS 10.15.1 and FreeNAS 11.2-U6