Slow SAMBA When Moving large amount of files locally...

[Dreded]

Ok so our office nas which is on FreeNAS 9.1.1 is unusable slow listing files/dirs when someone else(myself) is moving files around from one set of folders to another(directly logged in via shell)(several 100GB's worth) obviously that scenario rarely happens(first time in 5yrs) but it brought another issue to light which we have been wondering about for some time.

Occasionally the file server is slower than it should be and I figured out that it is due to someone using the windows Search box to find a old job file(daily occurrence) or when doing backups to another NAS(once an hour) making everyone elses access to the server slow while windows searches through between 1,000-10,000 files/folders this has been an issue for years but barely causes us concern however I am in the process of building a new file server as this one is now 4yrs old and can be relegated to the backup off site server. So I would like to know if there is a solution to this problem and if it is a hardware solution I need to include it into the new file server.

Currently we have 12 people using the file server daily but only 2 of those are constant use(work from files on server) and very soon we will be adding 6 more people and I would like to implement a policy of working off the file server as I plan to go to 10Gbe network.

To clarify I know this is not a network issue as moving files between Datasets or doing find ./*/*/*/*/*/*/*MyFile* causes the same issue with unusable slow anything, renaming a file(over network from windows) took 1m22s while doing a deep directory find operation(in SSH directly on server)

During the same find operation listing a folder with 300 Folder entries took 34seconds

Saving an excel file to server from windows took 38s

should also add that doing a file copy over the network if its the only file being copied and nobody else is using the file server happens at 112MB/s

The Specs on this(4yr old) server

Intel(R) Pentium(R) CPU G2120 @ 3.10GHz
SuperMicro X9SCL-II Motherboard
16GB ECC Ram
3x 2TB WD Red HD(RaidZ1)
Supermicro Case/PSU

 

[BigDave]

So I would like to know if there is a solution to this problem and if it is a hardware solution I need to include it into the new file server.

Recommended minimum quad core with hyperthreading and 64GB of RAM (look @ new Skylake platform or better perhaps).
Combine those with the 10Gbit network upgrade AND configure the new pool with 7200rpm SAS drives OR SSDs.

What kind of SM case?

 

[Dreded]

Recommended minimum quad core with hyperthreading and 64GB of RAM (look @ new Skylake platform or better perhaps).
Combine those with the 10Gbit network upgrade AND configure the new pool with 7200rpm SAS drives OR SSDs.

What kind of SM case?

While we can definitely afford that without issue it would seem overkill for our use as we do not do a lot of concurrent file access, even with the planned 18 people working off it each one of those people will open 3-4 files a day, nothing crazy

Also it's not a media server or anything just serves excel and word files on a network only reason to go to 10Gb is for the cad guys

 

[Chris Moore]

Unless you upgrade the server, then why bother with 10GB?
You need to avail yourself of the resources section of the Forum.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[SweetAndLow]

You need more iops and some samba tuning. Mirrors are easy to setup and will get you the most improvement. Tuning will be hard and get you some improvement.

 

[Dreded]

So I am gathering by the responses that the performance I am seeing of it taking 30+ seconds to rename a single file while a copy operation of a decent sized file(same issue copying 1 4gb file or many small files) is taking place is completely normal?

 

[BigDave]

So I am gathering by the responses that the performance I am seeing of it taking 30+ seconds to rename a single file while a copy operation of a decent sized file(same issue copying 1 4gb file or many small files) is taking place is completely normal?

Normal, not at all!

You are using a machine that was not configured for speed, the core hardware is low end. It's usually not about how often you perform tasks, it's how long are the tasks taking.
And so far, no one has mentioned the client machines in your environment, they could also be part of the slowness...
If you have Pentiums/Celerons/Realtec NIC, you have to find your bottlenecks and plan for growth!

 

[Chris Moore]

So basically what you're doing is you're buying a 1990 Honda Civic and then trying to take it out to the racetrack and do competition against a Top Fuel dragster. You're not buying the right kind of equipment for the job.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Dreded]

Normal, not at all!

You are using a machine that was not configured for speed, the core hardware is low end. It's usually not about how often you perform tasks, it's how long are the tasks taking.
And so far, no one has mentioned the client machines in your environment, they could also be part of the slowness...
If you have Pentiums/Celerons/Realtec NIC, you have to find your bottlenecks and plan for growth!

I appreciate what you are saying and I am not trying to argue I could be easily missing something obvious here but...

My CPU is typically under 5% use, when doing an rsync copy the CPU use (according to top) goes to 10-12% for each end so doing the operation from one folder to another on the same machine is 25% use, each SMBD operation takes 2-3% of the CPU, and given that at most this file system will only ever see 2-3 concurrent file requests the CPU is not an issue.

I am mostly talking about local file moving so the network cards are not an issue but all of the machines have Intel 1Gb ethernet adapters and all the switches are quite adequate (the system I am performing these tests from actually has a direct connection to the server on the secondary NIC).

So to be 100% clear I am SSHed into the file server and performing a cp command copying from Dataset A to Dataset B then trying to perform a rename from Windows explorer over the network on a file in Dataset A... all of these files are on the same RAIDZ1 of 3x 2tb WD Red HDs during this cp process the rename takes 30+ seconds to perform

When I built this system 4 years ago it was the recommended hardware for a small-medium use case (maybe this isn't me and I just don't realize it), I have much much lower end hardware on my home FreeNAS (x2 255 8GB Ram) and it seems to get better performance (anecdotal at best but it runs 4-5 jails downloading and streaming etc all the time compared to this one that is only expected to allow 2-3 file operations at once at most).

So is it expected performance of ZFS on a single non mirrored RAIDZ1 to only allow 1 concurrent file operation? So if I am copying a large file from folder A to folder B LOCALLY it is completely normal (with my hardware) that a network user cannot rename a file in a timely manner?

 

[SweetAndLow]

What do you mean by rename? Like click on the little box and add a letter to the filename? It seems like this should be easy and not take 30sec. What modifications have you made to freenas? Usually things should just stick to the defaults.

 

[Dreded]

What do you mean by rename? Like click on the little box and add a letter to the filename? It seems like this should be easy and not take 30sec. What modifications have you made to freenas? Usually things should just stick to the defaults.

thank you thats my point this should not take 30seconds.. I literally mean add a letter to a filename... I have made no modifications.. there is no Jails running or anything its a base install

I am continuing to do further testing and it doesn't seem to be an I/Ops issue as even if I limit the mv command(now using rsync instead of mv) by doing

rsync -avhP --remove-source-files --bwlimit=10000 BackupOLD/RyanC/ RyanC/

which limits the copy speed to 10MB/s without the limit rsync reports upwards of 80MB/s

I still get the same slowness issue but instead of 30s it takes 15s to rename a file so there is an improvment it just doesnt make sense

it seems to mainly be slowing down the speed at which windows can list a directory or find a file in a list of files as even browsing the directory trees which each have 100-500 folders its slow as hell

 

[Chris Moore]

My CPU is typically under 5% use, when doing an rsync copy the CPU use (according to top) goes to 10-12% for each end so doing the operation from one folder to another on the same machine is 25% use, each SMBD operation takes 2-3% of the CPU, and given that at most this file system will only ever see 2-3 concurrent file requests the CPU is not an issue.

Agreed, but you will want a more powerful CPU in the new server to handle the larger amount of storage and the larger number of users.

all of these files are on the same RAIDZ1 of 3x 2tb WD Red HDs

This is actually where I see your problem originating. Your storage pool is poorly implemented. If you want faster storage, it needs to be done differently.

When I built this system 4 years ago it was the recommended hardware for a small-medium use case

The hardware may have been, but that drive layout, I don't think so. I remember building a system about that time and I recall that even then they were saying not to use RAID-z1 with drives larger than 1TB, but that is only part of the problem.

my home FreeNAS (x2 255 8GB Ram)

I don't even know what "x2 255" is supposed to mean. But it sounds like you want to go with a bare minimum system and still expect to get 10GB network speed from it. That is just not going to happen. You will be disappointed and blame FreeNAS when it is really down to your hardware choices. I want to make sure you know up-front, high speed data takes more drives and there are smart people on this Forum, that do this kind of thing for their "day job", that can point you in the right direction.

So is it expected performance of ZFS on a single non mirrored RAIDZ1 to only allow 1 concurrent file operation? So if I am copying a large file from folder A to folder B LOCALLY it is completely normal (with my hardware) that a network user cannot rename a file in a timely manner?

I would say that you should read the primer on zfs, but let me try to sum it up. The storage pool that you have is made up of a single vdev (virtual device) that consists of 3 physical devices. This has the effect of limiting your performance to the speed of a single hard drive.
If you want to go faster, you need more vdevs, and the easiest way to get more vdevs without having a really large number of drives is by using mirrored drives where each two drives is a vdev and to get to 10GB network speed, you will need a minimum of 10 drives (5 vdevs) depending on the performance of an individual drive. Anecdotally, the more users you have, the more drives you need. The organization I work for has many storage systems and one of those (called a SAN) has about 300 drives in order to have the speed needed to satisfy our users.
Would you like to know more, or receive recommendations on hardware?

 

[Chris Moore]

thank you thats my point this should not take 30seconds.. I literally mean add a letter to a filename... I have made no modifications.. there is no Jails running or anything its a base install

How full is your storage array?

 

[Chris Moore]

our office nas which is on FreeNAS 9.1.1

There is an updated version of FreeNAS with many fixes and improvements. Why is it that you have not updated?

 

[bigphil]

Do you happen to have dedup enabled on any of your datasets? You can also check this thread for some smb tuning advice.

 

[Dreded]

Agreed, but you will want a more powerful CPU in the new server to handle the larger amount of storage and the larger number of users.

This is actually where I see your problem orignating. Your storage pool is poorly implemented. If you want faster storage, it needs to be done differently.

The hardware may have been, but that drive layout, I don't think so. I remember building a system about that time and I recall that even then they were saying not to use RAID-z1 with drives larger than 1TB, but that is only part of the problem.

I don't even know what "x2 255" is supposed to mean. But it sounds like you want to go with a bare minimum system and still expect to get 10GB network speed from it. That is just not going to happen. You will be disappointed and blame FreeNAS when it is really down to your hardware choices. I want to make sure you know up-front, high speed data takes more drives and there are smart people on this Forum, that do this kind of thing for their "day job", that can point you in the right direction.

I would say that you should read the primer on zfs, but let me try to sum it up. The storage pool that you have is made up of a single vdev (virtual device) that consists of 3 physical devices. This has the effect of limiting your performance to the speed of a single hard drive.
If you want to go faster, you need more vdevs, and the easiest way to get more vdevs without having a really large number of drives is by using mirrored drives where each two drives is a vdev and to get to 10GB network speed, you will need a minimum of 10 drives (5 vdevs) depending on the performance of an individual drive. Anecdotally, the more users you have, the more drives you need. The organization I work for has many storage systems and one of those (called a SAN) has about 300 drives in order to have the speed needed to satisfy our users.
Would you like to know more, or receive recommendations on hardware?

appreciate the full response..

an x2 255 is a sh#$ty cpu with a passmark score of 1800, point is that system gets better performance than the work system which has higher end hardware which is why I used FreeNAS in the office in the first place because I was getting decent performance with it at home... and the office performance has always sucked

I have watched the ZFS Ninja videos as well as the Samba Videos and read the entire ZFS primer.. I definitely understand that the performance of Raidz1 will be at the performance of the slowest disk. I can accept all that and right now on my current system all I would like is the performance of a single disk but I am getting the performance of a Floppy Disk not a 100MB/s drive.

Also nothing discusses how performance scales.. so if I have 2 users copying two separate files how does it disperse that load(specifically on a RaidZ1) on a single drive on my windows or linux system it would attempt(key word attempt) to give both users 50% of available I/O but that is not what I am currently seeing with FreeNAS

I would love a recommendation on hardware for the new system.. however as not to muddle this discussion of my current problem further(my own fault) I will start a new thread for that.

How full is your storage array?

50%

There is an updated version of FreeNAS with many fixes and improvements. Why is it that you have not updated?

previously... because I didn't fully understand the extent of my problem until I started moving files around to organize our data before the move to the new office next month,
previously I thought it was an occasional Hiccup

now.. because i am building a new system next month so I don't feel its worth the risk of the added downtime even if the risk is super minimal(upgrade going bad) although if this is the only solution to my current issue then I will take the risk as I have a lot of files to move around to get our data better organized and I dont feel like doing this when everyone else isn't at work(over night)

however I find it hard to believe that the performance of samba was so bad on Freenas 9.1.1 as to not allow 2 concurrent file operations.

 

[Dreded]

Do you happen to have dedup enabled on any of your datasets? You can also check this thread for some smb tuning advice.

no compression and no dedup

 

[Dreded]

ok found my issue..

it was the CIFS "Support DOS File Attributes" which I had known previously to be a small issue however it was decided that the small occasional performance hit was preferable over seeing all of the Hidden MS Word/Excel files(the ~filename files) that excel and word create each time you open a file and sometimes leave behind even after the file is close(these ones are the real issue)

 

[Chris Moore]

Also nothing discusses how performance scales.. so if I have 2 users copying two separate files how does it disperse that load(specifically on a RaidZ1) on a single drive on my windows or linux system it would attempt(key word attempt) to give both users 50% of available I/O but that is not what I am currently seeing with FreeNAS

In my experience, how well a system handles multiple users accessing the storage simultaneously, is slightly dependent on processing speed and memory, but mostly depends on the number of disks involved because each disk has a mechanical limit on how fast it can go, so the more disks available, the faster the data can be moved.
I am also not sure to what degree the root user gets priority on the system. I studied this years ago and it may have changed but it was once the way that Unix and Linux operating systems worked that the users were assigned a priority and if the root user wanted something done, the system would do that first and make all the other users wait. I will look into that because I don't like not knowing for sure.
In your situation, I would say that many of the problems you are facing could be corrected by updating to a newer version of the operating system. If I recall correctly, they fixed a number of speed related bugs in the SMB file sharing software stack since the version of FreeNAS that you are using.

however I find it hard to believe that the performance of samba was so bad on Freenas 9.1.1 as to not allow 2 concurrent file operations.

SMB was made faster, but handling two or more concurrent operations is largely a factor of the number of vdevs. Put it this way, on my system at home, I can be resilvering a replaced drive, watching a video stream from the Plex (file on the NAS) with transcoding and be copying files to or from the NAS using my workstation and everything works smooth and fast. It is also common for my wife and I to both be accessing directories on the NAS at the same time and it doesn't make us wait at all. Very snappy. Part of that is due to the newer version of SMB because I remember a time when it wasn't so quick. The other thing I did to get more robust performance was to simply add a second vdev. I can't fully saturate a 10GB network connection to a single client with only 2 vdevs, but it is more than adequate that I am not waiting on the NAS for the vast majority of the copying, moving and renaming I do at home. The delays you are describing are bad, but there are two parts to the problem and, without making hardware changes, I think the big fix for now is the software update.
I am not sure what you are using for a boot drive, but the 'standard' configuration that a lot of people use is the USB memory stick. The whole operating system is on that and even if you pulled it out and put in a new one and did a fresh install, all your data would still be on the storage pool so you could import it in the fresh install, create your share and be back in biz. I can't see a way to lose your data if you just do an upgrade. Do backup your system config file first, as a precaution.

 

[Dreded]

I can't see a way to loose your data if you just do an upgrade. Do backup your system config file first, as a precaution.

i seem to recall a previous issue I had with encryption or something when updating from FreeNAS 8.x to 9.3 at home then having to delete all data on the HD's even though I had never enabled any encryption or anything.

Since then I've been scared to update the office FreeNAS as while we do have multiple backups they are on slow NAS boxes and would take days to restore from and I don't want to risk the downtime. One of the things I am going to fix while moving offices is not using the SUPER SLOW ReadyNAS we are currently using for backup and use the current FreeNAS box as the backup instead.