Improving performance for large directories

[John Doe]

I have a box running 9.2.0 on a small windows network using CIFS. I put it together to use old hardware and drives as back storage on the network so it actually spends most of the time turned off. The drives are organised as a couple of raidz volumes.

When I do turn it on I want to sync directory structures containing thousands of files with other machines on the network and it is terribly slow. A recursive directory listing processes a few tens of files per second. If I list a directory from cold I see freenas memory usage rising which I presume is it caching zfs metadata. If I list the same directory a second time files are processed tens of times faster. I think caching on the freenas box not local is making the difference.

If I sync a directory tree containing 60k files it takes 25 minutes to decide what needs to be synced. File transfer of larger files runs at 70-80MB/s in either direction so the network isn't inherently slow.

My question is are there any settings or tweaks which might improve performance for this usage, perhaps something to encourage read ahead of zfs metadata.

 

[cyberjock]

Can you post system specs or anything? There's nothing to go on based on your post...

And when you say "sync a directory tree" that could mean anything. And the number of files are only a small part of the performance numbers. Total number of GB that need to be synced can also make a HUGE difference in some applications.

Again, you need to provide MUCH more data than you have. Other than that, 70-80MB/sec doesn't seem to indicate any performance problem at all.

 

[John Doe]

The NAS had been running a while so its cache was not empty.

From a windows command prompt I time the execution of DIR /S on a directory on the NAS containing 67k files in 18k directories.

It takes 20 minutes 23 seconds.

I time it again and it takes 3 minutes 26 seconds.

A third and all subsequent times it takes 19.3 seconds.

Network traffic on the windows machine was inversely proportional to the run time of the command - there was no local caching.

The NAS has about 9TB of drives and 8GB of ram, possibly low on RAM but obviously enough to cache all this directory.

I am asking if there are any settings or tweaks to get the runtime of the first DIR/S closer to that of the 3rd.

Edit: I think I confused time of day and elapsed time on the 3rd and subsequent runs so I did it again after rebooting the NAS.

1st run 20 minutes 14 seconds.
2nd and subsequent runs between 41 and 42 seconds.
Freenas memory usage increased by about 1GB during the test.

Edit: I think the DIR/S counts . and .. as directories or something. Another tool thinks the files are in 6k directories - I thought it looked high.

 

[Starpulkka]

got files 10Tb usable mem 16gt same kindof unresponsivess in 2 mins then the arc builds up at steady 300k even when you dont do nothin (you can check it zpool io stat 1) stops around 4666M so arc takes almost 5Gt from the start. Got another machine with prefetch disabled and it responds like a snap. So you have like 3 options to what you can do. You have rebootet the nas but have you rebootet the windows?

 

[AlainD]

Hi take a look at :

http://forums.freenas.org/threads/nfs-slow-directory-listing.13653/#post-92029

Alain

 

[John Doe]

Hi take a look at :

http://forums.freenas.org/threads/nfs-slow-directory-listing.13653/#post-92029

Thanks for that link. I tried recursing directories on the server through smbclient. I didn't time it but it wasn't much faster than doing it across the network so time from turning on the server to having directories synced wouldn't improve much and starting syncs before the recurse had finished would probably thrash and make it slower.

I tried something else. I logged the directories in a file manager, sorted by directory and name then flipped the archive attribute on all files hoping that zfs's copy on write would improve locality of the directory information on the dives. I rebooted the server and did the DIR/S test again and it came down from 20 minutes 14 seconds to 14 minutes 50 seconds. Seems a worthwhile if not great improvement.

 

[Yatti420]

Hopefully more ram will help.. Remember CIFS is single threaded..

 

[John Doe]

Hopefully more ram will help.. Remember CIFS is single threaded..

After a server reboot and 15 minutes of DIR/S test the server is showing 6.3GB of free physical memory.

 

[Yatti420]

You are using ZFS I presume.. Still need specs/network setup etc The only other thing I can suggest is minimize/cleanup those directories..