Extremely slow copy - emails archive (65 000 tiny files) - Mac

[overshoot]

Hi,

I have a Dell R320 running FreeNAS 11.2U6 on a 10TB pool made of 8 2TB SAS drives.
The Server has 16GB of RAM.

The client is a Mac connected through Ethernet Gigabit.
Large files copy run just fine.
However I've been experiencing extremely slow copy for tiny files.

A perfect example is, today, I have an archive of my email account to copy to the Server. It's approximately 65 000 files to copy for a total of around 9GB.
Whichever software I use, Finder or Carbon Copy Cloner (based on rsync), the copy is showing me hours to complete.

Using CCC, in an hour, the copy only did 1GB which is 16MB/minute or about 300KB/s.

The main folder containing the archive is on a USB 3 stick. Copying its content to the desktop takes 3 mins.
Compressing that folder did create an archive of 5GB which took a minute to copy over to the FreeNAS server.

Using Netdata to check how loaded is the Server, it's doing nothing. Ram is used at 89%. CPU never goes beyond 10%.

I've read a few posts on the subject of slow CIFS connections between Macs and Samba on FreeNAS and I understand it's a common problem.
Though it seems to be extremely slow in my case.

Anyway, here are my Server settings:

My dataset is set to case sensitivity = insensitive.
I have no antivirus software running.
I've tried aio read and write to 0 or to 1 with no difference.
Strict sync is set to no.
VFS objects used are: catia zfs_space zfsacl fruit streams_xattr

Client side:
Mac client has packet signing disabled.
Running on Mojave 10.14.6
MacPro 2012 with 48GB of RAM and Ethernet Gigabit.

Anything else that I could improve?
Thanks!

 

[dlavigne]

Did you find a way to improve this?

 

[HoneyBadger]

I've read a few posts on the subject of slow CIFS connections between Macs and Samba on FreeNAS and I understand it's a common problem.
Though it seems to be extremely slow in my case.

Possibly one of my posts? I recall writing this blurb more than a couple times recently.

It's a Mac. You need to disable SMB strict sync.

In the SMB auxiliary settings (Services -> SMB -> Little Wrench Icon) set strict sync = no; you may need to restart the service to see the effects.

 

[overshoot]

Hi,

Possibly one of my posts? I recall writing this blurb more than a couple times recently.

It's a Mac. You need to disable SMB strict sync.

In the SMB auxiliary settings (Services -> SMB -> Little Wrench Icon) set strict sync = no; you may need to restart the service to see the effects.

I already did as can be read in my first post. Before using "strict sync = no", it was clearly worth but the results I am giving here are with "strict sync = no" set and are still not acceptable in my opinion, as if something else was wrong?!

Thanks for your help guys.

 

[anodos]

My dataset is set to case sensitivity = insensitive.

Did you set the auxiliary parameter case sensitive = true on the SMB share and then restart the SMB service? Both are required for that particular optimization in 11.2. In pcaps I've looked at in the past, MacOS rescans a directory after _every_ file creation or delete. This kills performance when manipulating large numbers of files. Without the case-sensitivity optimization on samba, the scan is O(n^2). After the optimization, the scan is O(n).

This is a significant improvement, but an even larger improvement would be to not rescan the directories at all. You can achieve this by using a Windows or Linux client to transfer files. That said, large sequential transfers and small file IO will always have different performance characteristics.

 

[HoneyBadger]

Hi,

I already did as can be read in my first post. Before using "strict sync = no", it was clearly worth but the results I am giving here are with "strict sync = no" set and are still not acceptable in my opinion, as if something else was wrong?!

Thanks for your help guys.

My mistake, I clearly overlooked that line.

From what I've searched, it seems that signing might still be on if it is being requested by the server side. With the connection up, run smbutil statshares -a and see if the SIGNING_ON value is true.

There is also an Apple knowledgebase article which talks about changing the SMB browsing behavior and disabling the .DS_Store metadata file:

Adjust SMB browsing behavior in macOS - Apple Support (CA)

The default settings for browsing network folders such as Server Message Block (SMB) shares are ideal for most organizations and users. But you can make adjustments to optimize SMB browsing in enterprise environments.

support.apple.com

 

[seanm]

@anodos, I've read the samba smb.conf docs but am confused by all the various interactions. If one sets both strict sync = no and case sensitive = yes, is there some trade off?

I'm trying to decide if I should use these settings too...

In my case, all ZFS datasets use case sensitivity = insensitive and I only want the default Windows and macOS behaviour of being case insensitive but case preserving.

 

[anodos]

@anodos, I've read the samba smb.conf docs but am confused by all the various interactions. If one sets both strict sync = no and case sensitive = yes, is there some trade off?

I'm trying to decide if I should use these settings too...

In my case, all ZFS datasets use case sensitivity = insensitive and I only want the default Windows and macOS behaviour of being case insensitive but case preserving.

"strict sync = no" causes samba to basically no longer honor fsync requests. "case sensitive = true" will be automatically applied on case insensitive datasets in 11.3 when ixnas is loaded (so this aspect of configuration will not be a concern). As far as whether it's an acceptable configuration, you will need to determine whether the particular application requires a case-sensitive filesystem.

 

[seanm]

Thanks @anodos. Nothing I use requires a case-sensitive file system (but I do need case preserving). So sounds like "case sensitive = true" is fine. (Seems backwards that it will be auto-applied on case insensitive datasets. :))

But ignoring fsync... sounds like something databases and time machine might not appreciate... any guidance there?

 

[anodos]

Code:

	if (config->time_machine) {
		DBG_NOTICE("Enabling durable handles for Time Machine "
			   "support on [%s]\n", service);
		lp_do_parameter(SNUM(handle->conn), "durable handles", "yes");
		lp_do_parameter(SNUM(handle->conn), "kernel oplocks", "no");
		lp_do_parameter(SNUM(handle->conn), "kernel share modes", "no");
		if (!lp_strict_sync(SNUM(handle->conn))) {
			DBG_WARNING("Time Machine without strict sync is not "
				    "recommended!\n");
		}
		lp_do_parameter(SNUM(handle->conn), "posix locking", "no");
	}

See warning in vfs_fruit source.

 

[seanm]

Well, this past weekend I tried strict sync = no on a share, and indeed it's about 1.8x faster now. Nice.