11.1U6 Update > Transfer aborts on SMB Share (Mac Client)

[moda]

Updated from U5 to U6 and now I am unable to transfer larger Files from a Mac Client (OSX 10.12.6) to my SMB Share.
I am able to connect to the share and start a transfer, but the transfer repeatedly stops after about 5-10 sec.
Already tried reverting back to activating min Protocol SMB1, but that didnt help it. A Win10 Client is acting normal.
Finder throws “The Finder can’t complete the operation because some data in “FileName” can’t be read or written. (Error code -36)”

Any suggestions on how to diagnose this?

 

[anodos]

Updated from U5 to U6 and now I am unable to transfer larger Files from a Mac Client (OSX 10.12.6) to my SMB Share.
I am able to connect to the share and start a transfer, but the transfer repeatedly stops after about 5-10 sec.
Already tried reverting back to activating min Protocol SMB1, but that didnt help it. A Win10 Client is acting normal.
Finder throws “The Finder can’t complete the operation because some data in “FileName” can’t be read or written. (Error code -36)”

Any suggestions on how to diagnose this?

Sounds like possibly a bug.

Post output of testparm -s
Also, open terminal on OS-X and run the command xattr /path/to/filename and post output here.

type ifconfig to determine the interface that you're currently bound to, then run the following command to generate a pcap tcpdump -i em0 -w /var/log/smb.pcap -s 0 -p host <ip address of your MAC> and port 445
Replace em0 with your interface.

If the resulting file is less that about 6MB, PM me the pcap. If it's larger, generate a redmine ticket and upload it and a debug to it. redmine.ixsystems.com

 

[moda]

Thanks for the guidance

Code:

[global]																														  
		dos charset = CP437																										
		multicast dns register = No																								
		netbios name = X																										
		server string = FreeNAS Server																							
		lm announce = Yes																										  
		nsupdate command = /usr/local/bin/samba-nsupdate -g																		
		logging = file																											
		max log size = 51200																									  
		kernel change notify = No																								  
		panic action = /usr/local/libexec/samba/samba-backtrace																	
		disable spoolss = Yes																									  
		load printers = No																										
		printcap name = /dev/null																								  
		server min protocol = NT1																								  
		time server = Yes																										  
		map to guest = Bad User																									
		obey pam restrictions = Yes																								
		security = USER																											
		server role = standalone server																							
		deadtime = 15																											  
		hostname lookups = Yes																									
		max open files = 468101																									
		dns proxy = No																											
		idmap config *: range = 90000001-100000000																				
		idmap config * : backend = tdb																							
		store dos attributes = Yes																								
		strict locking = No																										
		directory name cache size = 0																							  
		dos filemode = Yes																										
		acl allow execute always = Yes																							
		ea support = Yes																										  
		create mask = 0666																										
		directory mask = 0777

[X]																														  
		path = "/mnt/X"																						  
		veto files = /.snapshot/.windows/.mac/.zfs/																				
		read only = No																											
		vfs objects = zfs_space zfsacl streams_xattr																			  
		zfsacl:acesort = dontcare																								  
		nfs4:chown = true																										  
		nfs4:acedup = merge																										
		nfs4:mode = special
  

xattr gives no output, ls instead shows the following permissions, its definitly the file because I tried different files and mounts

Code:

-rw-r--r--  1 X  staff  946113241 28 Aug 22:45 /Users/X/X

Console shows the following messages

Code:

Received claim <private>
Claim F67229F7-CB8B-447D-989E-2ADD8E4D6344 granted in server
Claim F67229F7-CB8B-447D-989E-2ADD8E4D6344 invoked in server
Claim F67229F7-CB8B-447D-989E-2ADD8E4D6344 granted in client
Claim F67229F7-CB8B-447D-989E-2ADD8E4D6344 invoked in client
Received claim <private>
Claim AA5A340B-B1BA-457C-B76E-C9D06F252420 granted in server
Claim AA5A340B-B1BA-457C-B76E-C9D06F252420 invoked in server
Claim AA5A340B-B1BA-457C-B76E-C9D06F252420 granted in client
Claim AA5A340B-B1BA-457C-B76E-C9D06F252420 invoked in client
smb2_smb_parse_change_notify: smb_rq_reply failed 60
smb2_smb_read_write_async: smb_rq_reply failed 60
smb2_smb_parse_change_notify: smb_rq_reply failed 60
smb_iod_reconnect: Reconnected share X with server X
Claim AA5A340B-B1BA-457C-B76E-C9D06F252420 was revoked
Claim F67229F7-CB8B-447D-989E-2ADD8E4D6344 was revoked

Regarding the tcpdump, generate it on the client or freenas?
I will further verify the behaviour on a Win7, 10.13.6 and a second 10.12.6 Client

 

[anodos]

Perform tcpdump on the freenas server.

 

[moda]

10.13.6 Client: Shows similar behaviour, but it doesnt abort the file transfer > 200Mb - 300MB are transfered fine > Transfer rate drops to 0MB/s > waits for 20s > repeat

No Issues when transferring files from the NAS to the clients, only clients to NAS!

Edit:

After further investigation: no problems on a second 10.12.6 system, which is connected wireless. When first 10.12.6 client is also connected wireless > no error appears. Error appears when connection is estabilished over a wired network, in this case wired to a wireless bridge. Two different ports on this maschine, the bridge and different cables show the same result. I am rather clueless here. I will test with rolling back to U5 if this issue is independent from update or not.

 

[rungekutta]

Same issue as mine. https://forums.freenas.org/index.php?threads/network-issues-after-upgrading-to-11-1-u6.69506/

Glad it's not just me. You seem to have come further in debugging it. NB rolling back to U5 removed all problems for me.

 

[anodos]

Same issue as mine. https://forums.freenas.org/index.php?threads/network-issues-after-upgrading-to-11-1-u6.69506/

Glad it's not just me. You seem to have come further in debugging it. NB rolling back to U5 removed all problems for me.

Please create a bug ticket on bugs.freenas.org and link here.

 

[rungekutta]

Ok so for everyone's benefit... I did just that, and the problem has been identified.

The problem is this patch in U6 which introduces a limit on the size of TCP reorder queue to limit potential of DoS attacks. The default value is 100, and in these scenarios described above (and likely others!) this is clearly too restrictive, which leads to FreeBSD starting to drop packets. The workaround for now is to set net.inet.tcp.reass.maxqueuelen sysctl to something much bigger (than 100).

Solved the problem for me at least. Remains to be seen what iXsystems does about this and what defaults are put into U7.

 

[nojohnny101]

@rungekutta I believe I am also facing this issue with SMB shares and slow performance.

Could you post a little more more detail about setting net.net.tcp.reass to something bigger than 100?

 

[rungekutta]

@rungekutta I believe I am also facing this issue with SMB shares and slow performance.

Could you post a little more more detail about setting net.net.tcp.reass to something bigger than 100?

Go to System -> Tunables and select Add Tunable. Set Variable=net.inet.tcp.reass.maxqueuelen, Value=4096, Type=Sysctl. Reboot. Done.

There has been some more discussion with the iX devs on this on the ticket I created. It’s marked private as it contains my log files so no point in posting a link. But in summary, confirmed that it relates to the FreeBSD change I mentioned further up and the iX dev is in discussion with the original author of that patch.

And here’s my *own* opinion - that change was clearly not thought through nor tested properly in real world scenarios. It’s a work around for an inefficiency in FreeBSD (making it vulnerable to DoS-attacks) but the default setting as of that patch clearly too conservative for most. I believe a rewrite on the cards for FreeBSD 12, until then need to balance vulnerability to DoS with usability in real world scenarios. The (default) value of 100 of that parameter does not strike the right balance.

Edit: copy-paste error

 

[nojohnny101]

@rungekutta thanks for that, I got it now.

Rebooted and speeds are back to normal. Thanks for chasing this and if can remember, I would appreciate it if you could update this thread if your ticket get resolved.

 

[moda]

I can confirm the solution posted by @rungekutta works, file transfers are back to normal and no more transaction aborts! Thanks for tracking it down!

 

[Jon Moog]

For anyone familiar with CLI this can be done without a reboot by "sysctl net.inet.tcp.reass.maxqueuelen=4096" for a temporary fix. I mention it just to save anyone the reboot trouble if they can navigate a terminal. In my case I have a million things running on the box and the persistent fix is nice but being able to change that value on the fly saves a long reboot cycle and the cleanup afterwards.

 

[White95supra]

Having the same problem on 11.2 beta 3, this workaround seems to have fixed the problem, thanks.

 

[rungekutta]

Has gone very quiet since iX commented (on the bug report) they’re talking to FreeBSD people about it. The bug is still open and not marked to any release yet. Bit odd but I’ll guess we just have to leave the workaround in place until resolved in the product itself. I’d imagine quite a few people may have problems with this, was very easy to trigger at least in my environment, which isn’t particularly exotic. Just a few diverse clients (Mac, Win10, Linux), an opnsense router and a few switches.

Edit: typo

 

[ajschot]

same problem in 11.2 RC2

 

[rungekutta]

There’s a bit of movement (discussion) on the bug report now but no firm conclusion and as far as I can see not assigned to a release version yet. Looks like iX will change the default value of this sysctl back to something less restrictive to get closer to the previous behaviour but unclear to what exactly, and as mentioned also when it will go into a release.

For what it’s worth my view is throwing away packets with the effect of creating instability / potentially killing clients should be avoided at almost any cost even if at the expense of CPU spikes, unless those are severe enough to DoS the whole server (no examples of that in normal use as far as I know - however there clearly *are* examples of packets being thrown away in perfectly normal and legitimate use cases i.e. this thread).

 

[rungekutta]

By way of update on this. Default value in FreeNAS will be set back close to what it was pre the CVE-2018-6922 patch. In testing, bug is #43558. Target versions 11.1-U7 and 11.2-U1, which I guess means it's released, or not? Bit confusing, this page says "due in 24 days" but 11.2-U1 is available for download at iX download page.

Personally I'm still on 11.1-U6 but plan to upgrade to 11.2 at U2 or U3 and will keep my net.inet.tcp.reass.maxqueuelen sysctl until confimed fix.

 

[anodos]

There's a bit of confusion because we had to perform an out-of-band security release for the netatalk CVE. In that release, we only applied the CVE fix. You're safe to keep the sysctl in place. It won't impact future upgrades.