SOLVED Error code -36

[wagnerone]

I've been troubleshooting this and searching around for a while and can't figure it out. I have found a few threads searching here, but often can't get the thread to render in any browser on my system for some reason despite being logged into the forum (e.g., https://forums.freenas.org/posts/53114)

My FreeNAS has 1 volume with a handful of datasets.

I have 3 datasets with SMB shares for archiving of misc files. I have 3 datasets with AFP shares that serve as network Time Machine targets.

I've copied a large amount of various file types and sizes to the 3 SMB shares. The AFP shares get regular Time Machine backups all day long from 3 Macs.

All data transfers are from Macs to the FreeNAS system. Most of the time I transfer files I have no problem. However, for the last few macOS version (currently on the latest, High Sierra 10.13.4), I can't get a series of files to copy to the SMB shares on the FreeNAS.

For example, I have zipped versions of the past several macOS installer apps. They're several GB each. When I try to copy them to the NAS, I get the error "The Finder can't complete the operation because some data in "xxx" can't be read or written. (Error code -36)"

I can copy these files off the source Mac to an external disk or via Remote Desktop to another Mac on my network, so I don't see that it's a read issue.

I've tried copying the files via WIFI and via a wired connection. They fail both ways.

I have a suspicion it has to do with ACLs or some other metadata associated with these particular files. I've copied tons of other files to the various shares and it continues to take Time Machine backups to the AFP shares daily with no apparent issue.

When I get the -36 error, the system sometimes (maybe always - this I just discovered) reboots.

FreeNAS version: FreeNAS-11.1-U4
motherboard make and model: ASUS P8P67 LE (REV 3.0) LGA 1155 Intel P67 SATA 6Gb/s USB 3.0 ATX
CPU make and model: Intel(R) Core(TM) i5-2500K CPU @ 3.30GHz
RAM quantity: 8 GB
hard drives, quantity, model numbers, and RAID configuration: (2) WD Red 6TB NAS Hard Disk Drive - WD60EFRX, raid1
hard disk controllers: built-in SATA controller
network cards: 3com 1Gb

 

[joeschmuck]

(e.g., https://forums.freenas.org/posts/53114)

Instead of this link try this one, it's the same thread number but not sure this is what you wanted:
https://forums.freenas.org/index.php?threads/53114/

Have you had this problem on earlier versions of FreeNAS? If not then I'd recommend you roll back and submit a bug report.

 

[wagnerone]

Have you had this problem on earlier versions of FreeNAS? If not then I'd recommend you roll back and submit a bug report.

Thanks. I've had it for quite a few versions of FreeNAS. Just every time I try to get time to troubleshoot it I get sidetracked. I recently changed trains to the latest thinking that might help, but no dice.

 

[joeschmuck]

That sucks that you have had this problem for a while now.

Have you looked at the internet to see if you could fix this issue?

Here are a link that looked good and I saw a lot more and all seem to indicate it's the HFS+ file system (I seem to recall hearing something about this file syste and things not going all fantastic):

https://errorcodespro.com/how-to-fix-error-code-36-mac/

Good Luck, I'm sure you will need it.

 

[wagnerone]

The fact the FreeNAS box reboots as a result of copying the file makes me think it isn't related to the local Mac filesystem.

How best does one go about diagnosing why FreeNAS would fail to the point of rebooting during a file transfer?

 

[joeschmuck]

The fact the FreeNAS box reboots as a result of copying the file makes me think it isn't related to the local Mac filesystem.

Yup, you got me there, I'm not sure why the FreeNAS would reboot. You could look at the log files "/var/log/" to see if anything shows up but it may not.

 

[wagnerone]

Doing some more testing, samba transfers of these files always fail (using SMB1 or SMB2 negotiation).

However, if I enable AFP for the same share and copy the files that way, the transfers succeed.

I did tail various logs while attempting the transfers with SMB. Nothing out of line is apparent in daemon, messages, or smb logs. System load is quite low when attempting the transfer. CPU use is nominal as is RAM as far as I can tell.

Does anyone have any additional guidance for me to troubleshoot this? I prefer to have SMB functionality to support both my Windows and Mac machines.

I can provide any additional info or am eager to hear any other troubleshooting tips.

I see this in /var/log/samba4/log.smbd right when the Finder SMB transfer stops

Code:

[2018/04/11 07:56:20.568819,  2] ../source3/smbd/close.c:789(close_normal_file)
  wagner closed file tmp/Install OS X El Capitan.app.zip (numopen=3) NT_STATUS_OK
[2018/04/11 07:56:20.569297,  2] ../source3/smbd/close.c:789(close_normal_file)
  wagner closed file tmp/Install OS X El Capitan.app.zip (numopen=0) NT_STATUS_OK
[2018/04/11 07:56:20.569380,  2] ../source3/smbd/service.c:1120(close_cnum)
  kashyyyk (ipv4:10.189.90.102:60932) closed connection to service archive
[2018/04/11 07:56:20.811165,  2] ../source3/smbd/service.c:841(make_connection_snum)
  kashyyyk (ipv4:10.189.90.102:60942) connect to service archive initially as user wagner (uid=1001, gid=1001) (pid 70187)
[2018/04/11 07:56:20.815544,  2] ../source3/smbd/open.c:1404(open_file)
  wagner opened file tmp/Install OS X El Capitan.app.zip read=Yes write=Yes (numopen=1)

And when the Finder finally times/errors out this single line pops into /var/log/samba4/log.smbd

Code:

2018/04/11 07:57:20.865199,  2] ../source3/smbd/close.c:789(close_normal_file)

Not sure if this is odd or not, but the files that did transfer have a timestamp of Jan 4. The ones that didn't transfer have a current timestamp

Code:

[root@tatooine /mnt/storage01/archive/tmp]# ls -la
total 19863208
drwxrwxr-x+  2 wagner  nas-archive		   8 Apr 11 07:53 .
drwxrwxr-x+ 13 root	nas-archive		  15 Apr 11 07:43 ..
-rwxrwxr-x+  1 wagner  nas-archive  5198018793 Jan  5 17:52 Install macOS High Sierra.app.zip
-rwxrwxr-x+  1 wagner  nas-archive  4965420477 Jan  2 21:05 Install macOS Sierra.app.zip
-rwxrwxr-x+  1 wagner  nas-archive		   0 Apr 11 07:55 Install OS X El Capitan.app.zip
-rwxrwxr-x+  1 wagner  nas-archive		   0 Apr 11 07:53 Install OS X Mavericks.app.zip
-rwxrwxr-x+  1 wagner  nas-archive		   0 Apr 11 07:53 Install OS X Yosemite.app.zip

 

[wagnerone]

More data about files refusing to copy. Any help or ideas??

* Tried copying these files from a different mac to the SMB share. They always fail at some point during the transfer.
* Tried copying the files first to an external disk (successful) and then from that disk to the smb share - same thing.
* I created a zip of a few of the "bad" files and tried to transfer the zip - same thing.
* Tried copying a 5 GB file generated with dd from the new mac to the same SMB share on the Freenas system for which the aforementioned files always fail to copy - this will copy consistently fine over and over from the mac to the nas
* Tried copying a 10 GB file generated with dd from the new mac to the same SMB share on the Freenas system for which the aforementioned files always fail to copy - this will copy consistently fine over and over
* Turned up the verbosity of the SMB log. When the "stall" occurs in the transfer the smb logging simply stops also
* The files that always fail to transfer start out at full speed. Sometimes the transfers will drop to zero, wait a while, and start back up at full or a slower speed and transfer a bit more before falling back to 0 and finally timing out
* Tried adding vfs_fruit to all the SMB shares, remounted share - no change
* Changed out 3com NIC with an Intel I210T1 - the transfers are surprisingly faster (while the run), but still fail similar to before
* Changed to "Unix" permissions at volume and share levels and pushed recursively perms through - no change
* On rare occasions one of the files will transfer fully. When it is done transfering, it sometimes stutters and at the end pauses for 20 seconds or so before it "finishes" and the transfer is considered complete. When I copy the 2 dd generated files mentioned above, they transfer at a constant rate and at the end of the transfer there is no pause whatever.
* I had had autotune enabled. I tried disabling that and removing all tunables. - no change
* Tried wired and wireless - no change
* Tried a different wired switch - no change

 

[joeschmuck]

* Tried copying a 5 GB file generated with dd from the new mac to the same SMB share on the Freenas system for which the aforementioned files always fail to copy - this will copy consistently fine over and over from the mac to the nas
* Tried copying a 10 GB file generated with dd from the new mac to the same SMB share on the Freenas system for which the aforementioned files always fail to copy - this will copy consistently fine over and over

Did you create the file using random or was it a null, or zero? Random is what you should have used.

You might need to submit a bug report.

 

[wagnerone]

Did you create the file using random or was it a null, or zero? Random is what you should have used.

You might need to submit a bug report.

Thanks, @joeschmuck. I appreciate your continued feedback.

I created the files like this - using zero

Code:

generate a 5GB file
time dd if=/dev/zero of=test5GB.bin bs=1000000000 count=5

generate a 10GB file
time dd if=/dev/zero of=test10GB.bin bs=1000000000 count=10

I'll try with random. How would this impact the test?

 

[joeschmuck]

I created the files like this - using zero

The problem is a file using all zeros is highly compressable and transfer differently than a file created as rondom. Also if your dataset is compressable then that also makes a large difference with the transfer. I know you are not doing benchmark testing but rather are troubleshooting a transfer issue.

I personnaly had a similar issue with transferring files due to my FreeNAS machine having a RealTek NIC. Switching to an add-on Intel NIC solved the problems. Eventually that machine became my second FreeNAS machine and still works great. Unfortunately your NIC is a Realtek® 8111E. I'm not saying this is your problem however it could be. Maybe you could borrow an Intel NIC add-on card for a test drive from someone?

Why would a RealTek NIC be the cause? Because all the NIC processing is done by the CPU, not the NIC and as such the device drivers for the specific OS must work well and when it comes to FreeBSD, well that isn't always the case. An Intel NIC does all the processing onboard, the CPU does not deal with it so the device drives are minimal, and Intel gets great support by FreeBSD.

 

[wagnerone]

I heard bad things about the NIC when I was building this FreeNAS and have avoided the RealTek on-mb NIC in this build since the beginning.

When I built this FreeNAS, I included an older 3com gig NIC I had rather than use the RealTek. However, thinking a proper NIC rated well with FreeNAS might be the cure, I last week purchased and installed an Intel I210T1.

This thing is super fast compared to the 3com to be sure. I'm impressed with it. However, it does the same thing on file transfers. The files transfer to a certain point and fail.

Thanks for the info though and for the insight on the /dev/zero generated test files. I'll retry that test with /dev/random generated files when I'm back home in a few days.

 

[joeschmuck]

I last week purchased and installed an Intel I210T1.

However, it does the same thing on file transfers.

Sorry to hear that.

I'll retry that test with /dev/random generated files when I'm back home in a few days.

Even if this passes or fails it will not fix your problem, but it will give you a data point.

Something else you might be able to try which will give you another data point is to boot up FreeBSD Live and then perform some file transfers.

So something easy you can try is to manually set the SMB protocol level...
Read this webpage, specifically the "client max protocol" setting and then place that value in the auxiliary parameters box under the SMB setup.

1) Turn off SMB protocol in the FreeNAS GUI.
2) In the FreeNAS GUI Services->SMB (wrench)->Auxiliary parameters:
max protocol = SMB2_10
min protocol = SMB2_02

3)Turn on SMB protocol and try the transfer again.

Read the link from top to bottom so you can see what is available, I'm not going to tellyou that you will understand it all, heck I don't either, but I do understand the values I listed.

 

[wagnerone]

Thanks for the continued feedback. I'll give these things a go for more data points. Looks like some nice, light reading. :)

One other analysis point - my storage disk setup = (2) WD Red 6TB NAS Hard Disk Drive - WD60EFRX, raid1

I have been able to, over time since I built this FreeNAS, fill these disks to 80%. So a heck of a lot of data did copy successfully to this setup since its inception.

 

[joeschmuck]

If you hit the 90% full mark then you will have a very slow write speed. It's an optimization for ZFS when the drive gets too full. That isn't your problem right now but it is something to keep in mind.

 

[SimonMdot]

Hey wagnerone,

did you solve the matter?
For me the same issues popped up just recently.
My dataset is quite loaded up (~80%) with personal media and now when copying/moving large files (>2gb) the error code 36 on my mac shows up.

I´m using WebDav as sharing method in order to have access to my files from all possible devices.

My assumption is that the load of the storage is causing that error. But want to know if the rest of my storage capacity is useless and I have to upgrade with additional space.

Any hint or idea in that regard would help me a lot.

 

[SimonMdot]

Nothin on your Side?

 

[wagnerone]

After all my tests and the feedback from people here (thank you!), it turned out my issue was related to my ZFS volume setup.

Partly because FreeNAS a complicated setup with lots of options for which I only had so many hours in the day to research and partly due to my taking advice from a ZFS nerd I know, I enabled both compression and deduplication on my big, single ZFS volume.

Now, he recommended it less from a home based FreeNAS context and from more a "big iron", enterprise context where there is a lot of $$ and a lot of RAM. Me being a newb at both FreeNAS and ZFS, I enabled it.

This didn't affect me for a long time, but ultimately it appears to have and for good reason considering the details I found after all other avenues of testing and trials (above) were to no avail and I finally focused on deduplication as a possible culprit.

I don't have nearly enough RAM (only 8 GB) to safely enable deduplication on a 6TB volume. Eventually it caught up with me.

As soon as I disabled dedupe, any and all files of all sizes and shapes happily copied to the volume at full, expected speeds. Much faster than before since my previous troubleshooting yielded a nice Intel NIC. :D

At any rate, I wanted to reply to let others know this could be their issue too. The thing that puzzles me is why there was no resource consumption warning of any kind either from FreeNAS or ZFS tools, etc. to lead me to this conclusion. It felt like stabbing in the dark repeatedly until something took.

Now that it's worked out though, I'm really back to loving FreeNAS.

Ultimately I'd like to offload all the data, rebuild the volume with deduplication off, and reload the data, but that's a hell of a lot of work for another day. I was able to delete a lot of data after disabling dedupe, but I don't know if that would release any RAM by reducing the dedupe table data.

 

[joeschmuck]

I don't have nearly enough RAM (only 8 GB) to safely enable deduplication on a 6TB volume. Eventually it caught up with me.

Ouch!

If you have a valid reason to use dedup then you will need to increase your RAM significantly.

Glad you figured it out.

 

[Stux]

-36 is a generic macos IO error. Ie because the ZFS server died.

Code:

ioErr = -36, /*I/O error (bummers)*/