CIFS Slow to delete

[Jason Emery]

I have a new FreeNAS 9.3 setup on a 45disk array. I setup a CIFS share which seemed to work great once I figured out how to control permissions. The issue I am seeing is delete performance. I copied a folder to the share from a Windows 7 box. It was about 3GB in size with about 3900 files in it. The copy went fairly quickly. When I tried to delete the folder it was estimating over an hour to delete it. I tried it from command line and from Windows Explorer. I eventually just went into the FreeNAS shell to delete the folder. Any ideas? I have the DOS support turned off which did not seem to help at all.

 

[SweetAndLow]

what version of windows are you using and can you provide better hardware info and exact freenas version. If you know the version of smb you are using that would be great also. Are you doing the delete via explorer?

 

[joeschmuck]

This is likely a remote computer (your Windows computer) thing because you are trying to delete thousands of files and for each file deletion it is specifying each file name vice just the subdirectory. Writing is much faster because those get cached in your FreeNAS RAM and written faster. It's not you, its the way things work.

Also, sometimes those estimates are a bit over inflated and if you let it run the course, it may have been significantly faster that the estimate.

 

[BigDave]

I have a new FreeNAS 9.3 setup on a 45disk array.

We'd love to hear more about your 45 drive configuration

 

[Darren Myers]

We'd love to hear more about your 45 drive configuration

And hardware specs in general :P

 

[Jason Emery]

I have a Storinator from 45drives.com built around the BackBlaze design.

Here are my specs.

Build FreeNAS-9.3-STABLE-201506292130
Platform Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
Memory 32707MB

It seems especially slow. It was deleting 1K files at a rate of 1-2 items per second. It did it from multiple machines. It does not seem right to me. I have FreeNAS set to a minimum of SMB2 and a max of SMB3. I did all this work from Windows 7, Windows Server 2012R2 with the same results. Also did not matter if I used Windows Explorer or the command line.

I also have these parameters set.

ea support = no
store dos attributes = no
map archive = no
map hidden = no
map readonly = no
map system = no

 

[SweetAndLow]

You should try seeing max protocol version to smb2 and min to the default. I doubt it will change anything but smb negotiates the highest protocol supported so minimum level doesn't matter. Then for max smb3 is still is a strange supported/unsupported state and should probably be avoided unless you know why you have it enabled.

 

[Jason Emery]

That did seem to help some on the copy speed and deleting when you only have a few larger files. It still seems to struggle when you have many smaller files.

 

[cyberjock]

How much storage do you have Jason Emery? 32GB of RAM doesn't seem like very much RAM if you have 45 drives...

 

[Jason Emery]

180TB.

 

[Darren Myers]

I found the issue:

180TB.

How much storage do you have Jason Emery? 32GB of RAM doesn't seem like very much RAM if you have 45 drives...

I think i found the answer, 32Gb isnt enough for 180Tb.....at all

 

[Jason Emery]

Interesting, this is one of the vendor configs they felt was plenty. That is raw capacity. This CIFS volume is only about 1TiB.

 

[Jason Emery]

Also, other protocols perform much better, iSCSI for instance.

 

[Darren Myers]

Also, other protocols perform much better, iSCSI for instance.

Oh man.....now youre throwing all kinds of other wrenches into the works..... for the iSCSI instance(s) are any of the pools over 50% usage? and as for the RAM....i think youre deff gonna need more for 180Tb, @cyberjock is using 32Gb of RAM for ~40Tb with no issues, but hes a single user....

 

[Jason Emery]

I know :). How much do you think RAM wise?

 

[Darren Myers]

I know :). How much do you think RAM wise?

Rule of thumb is 1Gb of RAM per 1Tb....in theory you should have 188Gb of RAM, 180+8 for OS.....but since thats a odd number, 192Gb is where you'd wanna sit....but maybe @cyberjock or @SweetAndLow can weigh in more

 

[cyberjock]

If performance is a consideration, I'd strongly recommend you upgrade to at least 128GB of RAM. If you could upgrade to 128GB of RAM and have some ram slots left vacant in case you need more, that would be better. The reality is that slow ZFS performance is usually fixed with more RAM. How much more depends on many factors and is hard to judge from a couple of forum posts. But I'd try 128GB of RAM as a good start if money is tight, if you can afford to do 196GB or 256GB you'd be much better off. Especially if performance is important.

 

[Jason Emery]

One more wrinkle. NFS is much much faster. So I ended up making an NFS mount point which I attached to my VMware hosts and created a Windows file server and shared it that way. The performance is more than 10x what it is with the FreeNAS box doing the CIFS directly.

 

[SweetAndLow]

Ran some tests just for fun to see delete performance from OSX 10.10.4 to FreeNAS 9.2.1.9 using cifs over a wifi network. I know wifi is unpredictable and I don't trust the numbers but something is better than nothing. Note the files size was 0 bytes, i just used 'touch' to create the files. I was just looking at IOPS. CPU was around 10-15% during creation and 22-26% during deletion.

Code:

$ ./delete-perf.sh 
Created 1000 files
Deleted 1000 files in:

real 0m47.685s
user 0m0.012s
sys 0m0.649s

$ ./delete-perf.sh 
Created 3900 files
Deleted 3900 files in:

real 5m15.619s
user 0m0.040s
sys 0m3.172s

 

[anodos]

Ran some tests just for fun to see delete performance from OSX 10.10.4 to FreeNAS 9.2.1.9 using cifs over a wifi network. I know wifi is unpredictable and I don't trust the numbers but something is better than nothing. Note the files size was 0 bytes, i just used 'touch' to create the files. I was just looking at IOPS. CPU was around 10-15% during creation and 22-26% during deletion.

Code:

$ ./delete-perf.sh
Created 1000 files
Deleted 1000 files in:

real 0m47.685s
user 0m0.012s
sys 0m0.649s

$ ./delete-perf.sh
Created 3900 files
Deleted 3900 files in:

real 5m15.619s
user 0m0.040s
sys 0m3.172s

Higher CPU can indicate that delete is operating more efficiently / faster than create (i.e. not bottlenecking as much at another place). I think the OP should look at samba logs to see if samba is unhappy about something during delete.