SMBD process goes 100% and copy speed collapse, ARC request demand metadata issue ?

[simonj]

Just to underline that this is an isolated SMB issue some real world tests rendering TIFF sequences to SMB and iSCSI from Davinci Resolve Mac on the same server and dataset with same network connection.

7:04 24fps 2K scope 2048x858 16bit TIFF

TIFF on SMB > TIFF on iSCSI:

~24fps (252MB/s)

TIFF on iSCSI > TIFF on iSCSI:

~33fps (346MB/s)

TIFF on iSCSI > TIFF on SMB:

~11fps (115MB/s)

TIFF on iSCSI > ProRes4444 on SMB:

336fps (3538MB/s)

TIFF on SMB > ProRes4444 on iSCSI:

42fps (441MB/s)

ProRes4444 on SMB > TIFF on iSCSI

60fps (630MB/s)

ProRes4444 on iSCSI > TIFF on SMB

12fps (126MB/s)

Bonus: This is how the network speed drops when writing a TIFF sequence to the SMB share:

smbd process at over 100%

Maybe @anodos could have a look at that?

 

[mJh78B]

For comparison with my previous post, here's (most of) the interaction between a macOS client and server. This is the same example I did before where I am moving a file from one directory containing 10k files to another via terminal, while Finder is watching the source directory.

Honestly I'm not really seeing a night and day difference between samba and a macOS server, though the macOS version on this server is quite old. It is a little faster, but this Mac has an SSD vs my TrueNAS machine's HDDs (though I'd think the file metadata should be cached in RAM), and this Mac's CPU is slower in single thread workloads than my TrueNAS machine's CPU (minus KVM overhead).

While doing move/delete/etc requests, the asymmetry in network bandwidth is quite severe for an operation that really should just be instructing the server to do something. From the perspective of the client:

This behavior seems to be more or less identical between doing an operation via terminal versus directly in Finder.

Once you open a folder on a share with Finder, it gets the find * kiss of death, even for a while after you close that Finder window. Again from the perspective of the client, here I opened the folder containing lots of files in Finder then closed it soon after. This time I was creating a bunch of files back on TrueNAS/samba, but I don't think the specific operation really matters. Finder kept trying to find * the directory, presumably to keep its internal cache up to date if I were to open the folder again soon. Once Finder stopped doing that, however, I saw that all the find * requests disappeared in Wireshark and I was able to create about 30-40 files/second instead of maybe 1 file/second, and the network bandwidth became far more symmetric.

Since samba (and probably macOS's smbd) gives a single thread per client, these find * requests slow the whole operation down since the actually useful requests don't get handled until the erroneous find * requests are dealt with. I wonder if something could be done to direct those find * requests to a separate smbd thread that can be busy dealing with Finder's find * obsession, while another smbd thread could respond more readily to the useful requests.

I would also be curious to see the Wireshark captures of people who are experiencing vastly improved performance between a macOS client and server versus samba, and if Finder or some other directory viewer is involved.

 

[Nicolas_Studiokgb]

For comparison with my previous post, here's (most of) the interaction between a macOS client and server. This is the same example I did before where I am moving a file from one directory containing 10k files to another via terminal, while Finder is watching the source directory.

View attachment 74661

Honestly I'm not really seeing a night and day difference between samba and a macOS server, though the macOS version on this server is quite old. It is a little faster, but this Mac has an SSD vs my TrueNAS machine's HDDs (though I'd think the file metadata should be cached in RAM), and this Mac's CPU is slower in single thread workloads than my TrueNAS machine's CPU (minus KVM overhead).

While doing move/delete/etc requests, the asymmetry in network bandwidth is quite severe for an operation that really should just be instructing the server to do something. From the perspective of the client:

View attachment 74663

This behavior seems to be more or less identical between doing an operation via terminal versus directly in Finder.

Once you open a folder on a share with Finder, it gets the find * kiss of death, even for a while after you close that Finder window. Again from the perspective of the client, here I opened the folder containing lots of files in Finder then closed it soon after. This time I was creating a bunch of files back on TrueNAS/samba, but I don't think the specific operation really matters. Finder kept trying to find * the directory, presumably to keep its internal cache up to date if I were to open the folder again soon. Once Finder stopped doing that, however, I saw that all the find * requests disappeared in Wireshark and I was able to create about 30-40 files/second instead of maybe 1 file/second, and the network bandwidth became far more symmetric.

View attachment 74665

Since samba (and probably macOS's smbd) gives a single thread per client, these find * requests slow the whole operation down since the actually useful requests don't get handled until the erroneous find * requests are dealt with. I wonder if something could be done to direct those find * requests to a separate smbd thread that can be busy dealing with Finder's find * obsession, while another smbd thread could respond more readily to the useful requests.

I would also be curious to see the Wireshark captures of people who are experiencing vastly improved performance between a macOS client and server versus samba, and if Finder or some other directory viewer is involved.

Hello @anodos , Hello Truenas Team and forum :)
Happy new year.

I was reconnecting a bit to the last comment on the thread I've initiated I'm curious about the wireshark logs that shows a large amount of command which might be our smbd overload issue ?

Let us know

 

[tiberiusQ]

Hi,

and or will Core 13.1 with smb 4.18.x will fix the reported issues ?
Btw. Is there a Roadmap for 13.1 ?

Thx & Best

 

[mJh78B]

Hello @anodos , Hello Truenas Team and forum :)
Happy new year.

I was reconnecting a bit to the last comment on the thread I've initiated I'm curious about the wireshark logs that shows a large amount of command which might be our smbd overload issue ?

Let us know

The main thing I've learned from using Wireshark is that if Finder has opened a directory, it constantly tries to list all the files in the directory, which slows all SMB operations down if there are a lot of files in there. If Finder (or any other program that does this) stops, the SMB operations become much faster. However, I was still only able to create 30-40 files/second which is still relatively slow. You might want to try using Wireshark and look for any SMB2 packets that contain "find *" commands to see if the Finder behavior might be your problem.

 

[Nicolas_Studiokgb]

What would be definitely interesting would comparing the same copy operation in 2 scenarios :
Between 2 macs in smb
Between 1 mac and truenas in smb

We might see how the mac smb react in comparison with samba/truenas

The main thing I've learned from using Wireshark is that if Finder has opened a directory, it constantly tries to list all the files in the directory, which slows all SMB operations down if there are a lot of files in there. If Finder (or any other program that does this) stops, the SMB operations become much faster. However, I was still only able to create 30-40 files/second which is still relatively slow. You might want to try using Wireshark and look for any SMB2 packets that contain "find *" commands to see if the Finder behavior might be your problem.

 

[Falcon]

What would be definitely interesting would comparing the same copy operation in 2 scenarios :
Between 2 macs in smb
Between 1 mac and truenas in smb

We might see how the mac smb react in comparison with samba/truenas

I know this may not be the case for you, but I had the same problem with high cpu load.

Someone in the thread was talking about it was a Mac problem I remember that I am running BlueHarvest
and it is cleaning directories after each write. When I quit the program I got full speed (4-6 GB) transfer speed all the time.

I have now excluded the cleanup on the network shares.

Maybe some of you are running BlueHarvest too on the network drives ?

Description on BlueHarvest
(http://www.zeroonetwenty.com/blueharvest/)
(BlueHarvest automatically removes .DS_Store and ._ AppleDouble files from your USB keys, SD cards, music players, file servers or any non Mac disk. BlueHarvest removes these items as they’re created or modified so you’ll always be metadata free without you needing to lift a finger.)

 

[mJh78B]

I know this may not be the case for you, but I had the same problem with high cpu load.

Someone in the thread was talking about it was a Mac problem I remember that I am running BlueHarvest
and it is cleaning directories after each write. When I quit the program I got full speed (4-6 GB) transfer speed all the time.

I have now excluded the cleanup on the network shares.

Maybe some of you are running BlueHarvest too on the network drives ?

Description on BlueHarvest
(http://www.zeroonetwenty.com/blueharvest/)
(BlueHarvest automatically removes .DS_Store and ._ AppleDouble files from your USB keys, SD cards, music players, file servers or any non Mac disk. BlueHarvest removes these items as they’re created or modified so you’ll always be metadata free without you needing to lift a finger.)

Sounds like it would be doing lots of directory listings, which is the same basic operation that I saw slowing down my NAS.

Also FYI there are some samba options that can be enabled to reject .DS_Store, etc files server-side if you want to retain that functionality.