TrueNAS 13.0.U4 SMB and Mac OS best Settings

[Nicolas_Studiokgb]

Hello All
I'm back with some serious move on TrueNAS install.
I want to make a file server for my 20 Mac clients. We now go SMB as AFP is not supported anymore by Apple.

Server :
Super Micro X10DRH-CLN4 with LSI 3008 IT mode SAS12gb adapter
Dual E5-2620 V4
Ram 256GB
System on mirrored sata SSD (mirrored pool in true nas)
Storage : 12 x 18TB Ultrastar SAS12gb 7200rpm HDD
Testing Pool Stripe of 12 hdd (I wanted the fastest pool to test performance. I'll go Raid-Z in a second time) Record block 128k (default ?)
Network 10Gb ethernet
SMB Apple SMB2 3 Extension Enabled

Client :
Macmini i7 6cores
Ram 64gb
10Gb Ethernet
OS Monterey 12.6.2

Client Server direct 10Gb Ethernet connection MTU default

Testing material on internal nvme :
100 GB of 5x 20gb Quicktime files
180 GB of Sounds (Wav, AAF, Protools sessions....)

Black magic speed test (5GB) on mac local nvme : 2600 MB/sec
Black magic speed test (5GB) on truenas share : 950 MB Write 1050 MB Read / Sec

Copying From mac to Server 100GB of big files : approx 9 Gb/sec (nearly wire speed) Approx 2 minutes
Copying From mac to Server 180GB of "small files approx 20MB each" : approx 2Gb/sec. Approx 12 minutes
From server to mac 100GB large files :
From server to mac 180GB "small files" :

The mac is never really overloading (Kernel task 20% Max)
I've tried multiple times each test to test arc zfs-cache fill-up (225GB of ram) Result are similar in all cases.

So Is there a Mac/SMB updated best practice post for recent TrueNAS versions ?

Do you see any obvious setting that I would have missed ?

Is there any performance impact depending record block size ?

Thanks a lot

Nicolas

 

[Nicolas_Studiokgb]

Some update about all this :

Added strict sync = no to the auxiliary parameters in SMB service config
it's a little better (25%) on the 180GB folder of "small files approx 20MB each" (audio .wav) but still to be improved.
It's not the disks that seems to be the problem as the network speed are the same when reading from zfs cache (during copy from the server to the mac of the 180 GB folder there was no disk activity in reporting/ disk/ i/O)

in the srcreenshot :
1 : Copy Mac to Server 100GB 5x20GB quicktimes (2 minutes )
2 : Copy Max to Server 180GB of "small files approx 20MB each" (audio .wav) (8 minutes)
3 : Copy Server to Mac 180GB of "small files approx 20MB each" (audio .wav) (7 minutes)
4 : Copy Server to Mac 100GB 5x20GB quicktimes (3 minutes )

Any idea ?

Thanks

 

[anodos]

I haven't seen conclusive reason to change defaults (create share with SMB preset, and share with default share options) for MacOS. Turning off strict sync means that SMB2 flush requests become no-ops. This isn't exactly a great thing and may be inappropriate for some workloads.
Hence, it's not a default.

Turning on AAPL extensions globally can help performance in some circumstances with directory listing via MacOS SMB client (because it packs finder metadata into the SMB2 FIND response), but many userspace apps on MacOS will still send per-file compounded ops to check for different xattrs (including Finder in newer MacOS versions) - which somewhat negates the benefit. Finder in MacOS ventura currently has a bug that prevents opening files after moving them between directories when AAPL extensions have been negotiated (confirmed against MacOS Server) and so using them is definitely a mixed bag.

The next major version update of Samba that we put into Core / SCALE should improve performance for metadata ops, but that's a ways off.

 

[Nicolas_Studiokgb]

Hello Anodos
Thanks for your reply.
Is SMB 3 act the same as SMB 2 ? (I've checked that mac and server negociate SMB 3) (smbutil statshares -a on the mac).
I'll try some more things, I would really like to get better perf for file transfers)
Thanks

 

[anodos]

Hello Anodos
Thanks for your reply.
Is SMB 3 act the same as SMB 2 ? (I've checked that mac and server negociate SMB 3) (smbutil statshares -a on the mac).
I'll try some more things, I would really like to get better perf for file transfers)
Thanks

You should use SMB3 (default). In general using non-default SMB client settings exposes you to risk of bugs that are perhaps not covered in Apple regression tests. I would never consider this for a production environment.

 

[Nicolas_Studiokgb]

Yes of course.
Monterey by default negociate SMB3.
I've trash and reset a new dataset on the same stripe test pool with these settings and perf is better writing for the 180GB folder with audio files I go up to 5Gbit

I'll try with the same folder of files between 2 10Gb mac mini and let you know
Thanks

 

[Nicolas_Studiokgb]

Another try with same setup and same files (I had the same issue yesterday)
It start normally and speed fall completely and CPU cores randomly goes 100%

Network speed falls completely (on the right )
15 minutes later there is still 30GB to copy and it goes slower than 10mbits

SMBD goes 100% on random cores

Something is clearly not working in this config....

 

[ChrisRJ]

I must admit that I did not spent 10 minutes analyzing the initial mail. But it is not entirely clear to me what the problem is. Probably the actual performance is worse than expected. But at least for me some additional explanation would be helpful.

 

[Nicolas_Studiokgb]

Hello Chris
Thanks for jumping in
The problem was to get the best performance between mac mini client (10Gbe) and TrueNas Server (10Gbe). So we are in SMB3 (as AFP is deprecated). The pool is 12 x 18TB Ultrastar SAS12gb 7200rpm HDD so the problem is not here.
I was surprised of the difference in perf between 2 set of data :
100 GB of 5x 20gb Quicktime files
180 GB of Sounds (Wav, AAF, Protools sessions....)
All files on the nvme storage of the mini.

Of course I knew that a few big files would be faster than a bunch of smaller ones, but I thought the difference would be smaller.

Finally after another test I ran just now, I get the same result between 2 mac mini (I manage to get a second one for testing).

But those tests shown that there is some difference in perf depending on SMB sharing and service settings.

The last thing in this thread that has to be addressed is the smbd that goes 100% on Truenas during the copy for unknown reason. (the post just before yours).

Thanks

 

[anodos]

SMBD goes 100% on random cores

This typically just means the process is CPU-bound. Samba forks separate processes SMB sessions. That session is busy. If you look at smbstatus output you'll probably see that the pid is associated with your MacOS client.

Of course I knew that a few big files would be faster than a bunch of smaller ones, but I thought the difference would be smaller.

It depends on what the client is doing. If you're just doing finder stuff (not a `cp` via terminal), then it could be doing lots of metadata-related ops per-write. (Opening / creating file, checking for xattrs, checking dir for dot-underscore files, opening .DS_STORE file, etc, etc) this all generates additional ops and network latency, Compared with a "large" file where it does metadata stuff once, opens file, then primarily sends write requests to the open file. This is generally how network shares work. A better thing to do is think about what workloads are important to you (generally bulk copy is a one-off thing during data migration) and see what you can do to improve that (or not improve it since it might be good enough).

 

[Nicolas_Studiokgb]

hello
My problem with that is why in the exact same configuration (same mac, same folder of files...) one copy (simple copy with the finder) goes like the first burst on the left of the graph and half an hour later the same copy goes like the part on the right. it start slower in the first place around 3,7 gbs instead of 5,5 and after a while it goes slower and slower. so slow that I had to force cancel the copy from the mac side.
Copying 200GB of sound rushes is a complete normal workflow for us and should work normally and consistently

In this test configuration there is only the server and one mac mini in the setup, so there is no network traffic from any other thing. and the server has nothing else to deal with, there is only one share and no other task configured. it's a brand new install.

To be noted that I don't do anything on the mac or the server during the copy instead of watching the network graph

 

[Nicolas_Studiokgb]

I precise I copy in a different folder of the server when I copy the same set of files so there no file replacement or erasing during the copy.
Also dedup and compression is not activated.

 

[Nicolas_Studiokgb]

For a clearer view I've continued the SMBD 100% "issue" on a new thread

SMBD process goes 100% and copy speed collapse, ARC request demand metadata issue ?

Hello I'm following up with my test setup for TrueNAS 13 U4 Server : Super Micro X10DRH-CLN4 with LSI 3008 IT mode SAS12gb adapter Dual E5-2620 V4 Ram 256GB System on mirrored sata SSD (mirrored pool in true nas) Storage : 12 x 18TB Ultrastar SAS12gb 7200rpm HDD Testing Pool Stripe of 12 hdd...

www.truenas.com

Thanks