Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

windows server robocopy to freenas extremely slow

Joined
Aug 22, 2019
Messages
9
Hi All

I'm trying to set up a backup server with freeNAS. I basically have 1 bigass zpool of about 300TB and datasets/file systems on them for each server. ( XEON skylake W-2102 CPU, 64GB of ram, no L2ARC) The pool consists of 6 vdevs of 8 disks in raidz2 config.

The initial copy from windows to freenas went fast. About 850 MB/s and I was very pleased :)

Now that the data is there (about 100TB) it has gotten extremely slow. It maxed out at about 40 MB/s for a single server and 60 MB/s if I run it from all servers at the same time.

I'm using "robocopy /mir" over SMB which looks at the datestamps and filesize of the src and dest and only copies new files. I ran into the "slow folder listing" threads and already disabled the DOS bits and increased the vfs.zfs.arc_meta_limit but that was't any help. The thing that strikes me the most is that even when all four nodes are copying to freeNAS (at 60mbps), I can still copy an additional 200GB file at 350 MB/s which maxes out the read spead of the windows drive array.

Any advice on how to debug this?

Cheers,
Marum
 
Joined
Aug 22, 2019
Messages
9
Somebody around that can give me some directions to search in? I really need this server in production... :s
 

sretalla

FreeNAS Expert
Joined
Jan 1, 2016
Messages
1,642
Have you looked at the type of data you're copying? if the first batch was lots of large files, your speed could have made sense, but if you're now into a section of files that are small and high in number, you will be spending much more time writing file metadata than actual file data, so copy speeds will drop like you have seen... there may be other causes too, but that's the first obvious one you can look at.
 
Joined
Aug 22, 2019
Messages
9
Thanks for the reply sretella. That would indeed make a lot of sense. Unfortunately ... it's the same folder/share I'm running the backup on.

The initial copy of share1 to the freenas box ran at speeds of 800 MBps. Now I rerun the same command for share1 and it only runs at 60MBps. The only difference is that there was no data to compare to in the initial copy, but there is for the 2nd backup. The file comparison is done by file size and timestamp.

I also notice that this goes fast for the first couple of seconds, but slows down tremendously after that. After a minute or so, it seems to proceed in a kind of bursted fashion. Usually this is a symptom of a bottleneck somewhere but I can't find what or where. io seems normal and smbd is only at ~20%. The arc hit ratio is about 90%. Ram is at a steady 4GB and 'wired' fills it up to 60GB as it should. Swap is free. No idea what to delve into. It might be samba but I don't know how to debug this.
 
Joined
Aug 22, 2019
Messages
9
I did the only test I could think of: create a massive folder with about 50k files on the freenas zfs and tried to view this folder from windows. If the culprit indeed is slow folder listing, well, this should be noticeable with a directory like that. Unfortunately/luckily the folders loads within a second..
 

garm

FreeNAS Expert
Joined
Aug 19, 2017
Messages
1,282
Turn of atime, use rsync instead of robocopy and divide your data into manageable folders. What I think happens is that robocopy reads the data already written, ZFS updates the access times and this cost you bandwidth. The application I manage professionally vaults multi TB CAD data but we never put more then 100 GB in a single folder (or vault).
 
Joined
Aug 22, 2019
Messages
9
Wow, atime.. that would make so much sense. I just checked and it's turned 'on' on all file systems. Switched it to Off. I'll rerun the backup tonight and report back. Thanks Garm!
 
Joined
Dec 14, 2019
Messages
4
marum I was has a problem like that and I wonder , Was your problem solved? as I have seen solutions and recommended programs
 
Joined
Dec 14, 2019
Messages
1
using rsync or gsrichcopy360 instead of robocopy will totally solve your problem , no need to consume more time as I think that you didn't find any solution till now with robocopy
 
Top