Slow CIFS performance (small files)

[Fireball81]

Hey guys,

i ve build myself a new freenas machine and everything is up and running so far.
The write performance for big files is pretty solid
(from the ssd of a win10 client machine to a cifs share on the freenas server over 1Gbit ethernet)
I can get around 100-110MB/s, pretty much saturating the 1Gbit ethernet link.

But the performance drops heavily when i try to copy directories with a lot of small files.
(e.g 540MB directory containing 13.200 Files or a 1,44GB directory containing 25.600 files)
The performance fluctuates between 5MB/s and 15MB/s.

Maybe you can help me to figure out what the bottleneck is and if i can make some tweaks or adjustments
to improve the performance because most of my workload is actually working with these files directly over ethernet on two client machines.

The specs of my freenas machine is as follows:


Pentium G4400
SuperMicro X11SSMF
2x8GB Kingston DDR4 ECC-RAM
2x4TB Hitachi Desktar NAS (mirror vdev)
FreeNAS 9.10 stable & latest updates

I appreicate any help you can provide.
Thank you very much in advance for your time.

Dennis

 

[anodos]

Performance is always going to suffer when you're dealing with tons of small files. There aren't really any 'go fast' tweaks you can do in samba. Throwing more RAM at it can help things. Having more disks may help too. Can you give a detailed explanation of your workflow?

 

[SweetAndLow]

more vdevs will help with small files.

 

[Fireball81]

Performance is always going to suffer when you're dealing with tons of small files. There aren't really any 'go fast' tweaks you can do in samba. Throwing more RAM at it can help things. Having more disks may help too. Can you give a detailed explanation of your workflow?

Of course i was aware of a performance penalty when it comes to working with lots of small files, but it came as a surprise when i figured out how much of a performance penatly it finally was.
Here is a brief description of our workflow:
We are a small office with just 2-3 client PCs. Those PCs are mainly working with a special engineering software that creates all those small files for different projects. All those project files are saved on our NAS storage of course, so the project data is accessible from the client machines.
The client machines are on windows 10 and we mount the cifs shares directly (as a drive letter) from there.
So the bottleneck in your opinion is the RAM or amount of VDevs? Surely i could throw a little bit more hardware on the problem but i was assuming that CIFS or the nature of how the operating system deals with lots of small files is the limiting factor here.

more vdevs and more memory will help with small files.

Lets say i would increase the amount of RAM from 16GB to 32GB can you give me a rough picture in % of how much of a performance increase that would be? (i know hard to answer, just to get a rough idea)
Same thing with vdevs, before i add more vdevs to the pool (adding another 2x4TB harddrives) i must know in how much of a performance benefit that would result. The 4TB storage we have right now if more than enough for what we need so adding more vdevs would be entirely for performance reasons.

Thank you for all your help guys.

 

[anodos]

Of course i was aware of a performance penalty when it comes to working with lots of small files, but it came as a surprise when i figured out how much of a performance penatly it finally was.
Here is a brief description of our workflow:
We are a small office with just 2-3 client PCs. Those PCs are mainly working with a special engineering software that creates all those small files for different projects. All those project files are saved on our NAS storage of course, so the project data is accessible from the client machines.
The client machines are on windows 10 and we mount the cifs shares directly (as a drive letter) from there.
So the bottleneck in your opinion is the RAM or amount of VDevs? Surely i could throw a little bit more hardware on the problem but i was assuming that CIFS or the nature of how the operating system deals with lots of small files is the limiting factor here.



Lets say i would increase the amount of RAM from 16GB to 32GB can you give me a rough picture in % of how much of a performance increase that would be? (i know hard to answer, just to get a rough idea)
Same thing with vdevs, before i add more vdevs to the pool (adding another 2x4TB harddrives) i must know in how much of a performance benefit that would result. The 4TB storage we have right now if more than enough for what we need so adding more vdevs would be entirely for performance reasons.

Thank you for all your help guys.

If you don't mind, post a debug file. "system" -> "advanced" -> "save debug".

We can also try to reduce the amount of things that samba is doing in the background when you're using a share (even a small delay starts adding up when you're dealing with thousands of little files). Try disabling DOS attributes globally on the server by adding the following auxiliary parameters to 'services' -> 'CIFS':

Code:

ea support = no
store dos attributes = no
map archive = no
map hidden = no
map readonly = no
map system = no

I believe @jgreco has written various posts about ARC size, transaction groups, write performance, etc. He might even comment on this thread if properly summoned.

The fundamental problem you're probably experiencing is with the mechanics of how hard drives work. They only spin so fast, and eventually you have to write the data to the disk. ZFS does some neat things like group lots of tiny writes and sequentially throw them on your spinning rust, but at the end of the day you're still limited by the input-output operations per second (IOPS) performance of your spindles (and you don't have very many of them). By the way, ZFS performs better if you have lots of free space, and so having extra disks isn't really a bad thing.

How much storage space do you actually need? Is having a second flash-based storage pool for current work in progress an option?

I'm asking this because if your current active projects only consist of 200GB, then might be able to just pick up a couple 500GB SSDs to host the active projects and put everything else on the 4TB zpool. SSDs are often the cheapest and easiest way to throw IOPS at a problem.

 

[Fireball81]

If you don't mind, post a debug file. "system" -> "advanced" -> "save debug".

We can also try to reduce the amount of things that samba is doing in the background when you're using a share (even a small delay starts adding up when you're dealing with thousands of little files). Try disabling DOS attributes globally on the server by adding the following auxiliary parameters to 'services' -> 'CIFS':

Sure, anything that helps to find out where i might be able to squeeze a little bit of extra performance (without affecting stability of course)
http://www.file-upload.net/download-11517228/debug-freenas-20160425154143.tgz.html

How much storage space do you actually need? Is having a second flash-based storage pool for current work in progress an option?

Not so much to be honest. The thing is, we purchased the server just a few days ago and i wouldn`t like to buy a bunch of ssds to solve a problem that we thought could be improved upon by purchasing the whole thing in the first place.
For the moment i have to make sure to get the best performance as possible out of the machine.
I ll give the samba adjustments you provided a shot 1st, thank you very much. I figures those out from the sticky thread but thought theyonly help with browsing issues.
You re right, 500GB of fast ssd storage would probably suffice for the projects we have to work on within a year.
Afterwards we could move those projects to a archive dataset/share on the slower pool but that is something we might consider later on.

 

[anodos]

Sure, anything that helps to find out where i might be able to squeeze a little bit of extra performance (without affecting stability of course)
http://www.file-upload.net/download-11517228/debug-freenas-20160425154143.tgz.html



Not so much to be honest. The thing is, we purchased the server just a few days ago and i wouldn`t like to buy a bunch of ssds to solve a problem that we thought could be improved upon by purchasing the whole thing in the first place.
For the moment i have to make sure to get the best performance as possible out of the machine.
I ll give the samba adjustments you provided a shot 1st, thank you very much. I figures those out from the sticky thread but thought theyonly help with browsing issues.
You re right, 500GB of fast ssd storage would probably suffice for the projects we have to work on within a year.
Afterwards we could move those projects to a archive dataset/share on the slower pool but that is something we might consider later on.

Well, one oddity I noticed is that your samba debug file lists about 65,000 active sessions from 192.168.100.153. This may reflect either (1) reflect a bug in the debug script or (2) indicate some underlying problem with samba or your freenas config. Post the output of "smbstatus" enclosed in [ code ] tags. (If it legitimately has 65000 lines, then use head to grab the first 100 lines or so). I recommend enabling SSH and using putty to run the "smbstatus" command. You will need to run smbstatus as 'root'.

Also, if you don't mind, please provide the name of the engineering software (if it's a common commercial product). I'll take a look at the samba mailing lists to see if anything relevant shakes out there.

 

[Fireball81]

Well, one oddity I noticed is that your samba debug file lists about 65,000 active sessions from 192.168.100.153. This may reflect either (1) reflect a bug in the debug script or (2) indicate some underlying problem with samba or your freenas config. Post the output of "smbstatus" enclosed in [ code ] tags. (If it legitimately has 65000 lines, then use head to grab the first 100 lines or so). I recommend enabling SSH and using putty to run the "smbstatus" command. You will need to run smbstatus as 'root'.

Also, if you don't mind, please provide the name of the engineering software (if it's a common commercial product). I'll take a look at the samba mailing lists to see if anything relevant shakes out there.

That might be related to the fact that i was in a middle of copying lots of small files from 192.168.100.153 to a cifs share while creating the debug file.

Post the output of "smbstatus" enclosed in [ code ] tags. (If it legitimately has 65000 lines, then use head to grab the first 100 lines or so).

70822 nobody nobody 192.168.100.153 (ipv4:192.168.100.153:58310) Unknown (0x0311)
70822 nobody nobody 192.168.100.153 (ipv4:192.168.100.153:58310) Unknown (0x0311)
70822 nobody nobody 192.168.100.153 (ipv4:192.168.100.153:58310) Unknown (0x0311)
70822 nobody nobody 192.168.100.153 (ipv4:192.168.100.153:58310) Unknown (0x0311)
70822 nobody nobody 192.168.100.153 (ipv4:192.168.100.153:58310) Unknown (0x0311)
70822 nobody nobody 192.168.100.153 (ipv4:192.168.100.153:58310) Unknown (0x0311)
70822 nobody nobody 192.168.100.153 (ipv4:192.168.100.153:58310) Unknown (0x0311)
....

This pretty much repeats itself over and over again. (while not copying anything right now)
Afaik Uknown 0x0311 is no reason for concern, ambstatus is just not able to tell correctly if clients connect via SMB3_10 or SMB3_11.

Also, if you don't mind, please provide the name of the engineering software (if it's a common commercial product).

The software suite is called mbaec, the module we use most is called vicado

Thank you very much for your effort, i really appreciate it.

 

[anodos]

That might be related to the fact that i was in a middle of copying lots of small files from 192.168.100.153 to a cifs share while creating the debug file.





This pretty much repeats itself over and over again. (while not copying anything right now)
Afaik Uknown 0x0311 is no reason for concern, ambstatus is just not able to tell correctly if clients connect via SMB3_10 or SMB3_11.



The software suite is called mbaec, the module we use most is called vicado

Thank you very much for your effort, i really appreciate it.

Let's see what your file locks are like. Post output of "smbstatus -L".

If the software is really setting locks on that many files / creating that many sessions, an SSD pool may be your best short-term option. Just be sure you make it sufficiently large (you don't want to go over 80% full). It's possible that ZFS + samba is just throwing a crapton of data into ARC and evicting it just as quickly because of the way that your CAD software works. I'm not a ZFS expert and so I can't be sure.

OPTIONAL:
It's somewhat disconcerting that you appear to have sessions as 'nobody'. It's possible this represents your software connecting to the IPC$ share of your samba server. It's also possible that you have a permissions problem and your software is authenticating under the "everyone" ACE.

Here are some examples of how to set permissions:
https://forums.freenas.org/index.ph...-of-how-to-configure-share-permissions.35276/
https://www.youtube.com/watch?v=RxggaE935PM]

I'd try creating a test dataset using the guides linked above. Unmap the samba share from the client computer, clear cached credentials, map it, and try again.

Some people have problems configuring samba permissions and I'm hesitant to suggest tearing apart an apparently working setup. The probability that your problem is permissions-related is relatively slim.

 

[Fireball81]

Let's see what your file locks are like. Post output of "smbstatus -L".

If the software is really setting locks on that many files / creating that many sessions, an SSD pool may be your best short-term option. Just be sure you make it sufficiently large (you don't want to go over 80% full). It's possible that ZFS + samba is just throwing a crapton of data into ARC and evicting it just as quickly because of the way that your CAD software works. I'm not a ZFS expert and so I can't be sure.

The output looks like:

7276 1001 DENY_ALL 0x17019f RDWR EXCLUSIVE+BATCH /mnt/Z-Pool_1/Jens Filexy... Mon Apr 25 20:04:08 2016
7276 1001 DENY_ALL 0x17019f RDWR EXCLUSIVE+BATCH /mnt/Z-Pool_1/Jens Filexy... Mon Apr 25 20:04:08 2016
7276 1001 DENY_ALL 0x17019f RDWR EXCLUSIVE+BATCH /mnt/Z-Pool_1/Jens Filexy... Mon Apr 25 20:04:08 2016
7276 1001 DENY_ALL 0x17019f RDWR EXCLUSIVE+BATCH /mnt/Z-Pool_1/Jens Filexy... Mon Apr 25 20:04:08 2016
7276 1001 DENY_ALL 0x17019f RDWR EXCLUSIVE+BATCH /mnt/Z-Pool_1/Jens Filexy... Mon Apr 25 20:04:08 2016
7276 1001 DENY_ALL 0x17019f RDWR EXCLUSIVE+BATCH /mnt/Z-Pool_1/Jens Filexy... Mon Apr 25 20:04:08 2016
7276 1001 DENY_ALL 0x17019f RDWR EXCLUSIVE+BATCH /mnt/Z-Pool_1/Jens Filexy... Mon Apr 25 20:04:08 2016
7276 1001 DENY_ALL 0x17019f RDWR EXCLUSIVE+BATCH /mnt/Z-Pool_1/Jens Filexy... Mon Apr 25 20:04:08 2016
7276 1001 DENY_ALL 0x17019f RDWR EXCLUSIVE+BATCH /mnt/Z-Pool_1/Jens Filexy... Mon Apr 25 20:04:08 2016
7276 1001 DENY_ALL 0x17019f RDWR EXCLUSIVE+BATCH /mnt/Z-Pool_1/Jens Filexy... Mon Apr 25 20:04:08 2016
7276 1001 DENY_ALL 0x17019f RDWR EXCLUSIVE+BATCH /mnt/Z-Pool_1/Jens Filexy... Mon Apr 25 20:04:08 2016
7276 1001 DENY_ALL 0x17019f RDWR EXCLUSIVE+BATCH /mnt/Z-Pool_1/Jens Filexy... Mon Apr 25 20:04:08 2016
....

Keep in mind though, that this is not while using the actual program but instead copying a project folder of 1,15GB containing over 21.000 files from my local ssd to a cifs share on the freenas server.
I can`t terst it with mbaec right now cause the server is with me where it`s actually build and set up (copying the old data over to the new nas and stuff) until friday.

t's somewhat disconcerting that you appear to have sessions as 'nobody'. It's possible this represents your software connecting to the IPC$ share of your samba server. It's also possible that you have a permissions problem and your software is authenticating under the "everyone" ACE.

Here are some examples of how to set permissions:...

This is exactly how i did it.
Creating a dataset share type = windows and a cifs share afterwards while maintaining permissions on the windows client.[/QUOTE]

 

[JoGi65]

I havn't understud all here (no native Speaker), but i dont think its a iops Problem.
I have testet with a single SSD and with a Pool of 4 SSDs and the Performance dosn't go up with small files. Always nearly exactly the same.
I try to translate the Rest for me and write then again. (Post4 from anados i try)
Herre you can see the Speed on a single SSD and that the Ram Upgrade dosn't help.

https://forums.freenas.org/index.ph...alls-with-many-small-files.42843/#post-281253

 

[anodos]

A few other options to try :

OPTION 1 - Kill Case Sensitivity
If you don't need case-sensitive file names is to try the following (as reported by Jeremy Allison in the Samba mailing lists):
1) create new dataset with "Case Sensitivity" to "Insensitive".
2) create share pointing to the new dataset with the following auxiliary parameter "case sensitive = true"

The above samba parameter tells the server that the name it's receiving from the client is case sensitive. This basically gets samba to skip searching and matching file names that are passed to it, thereby speeding up operations when dealing with lots of files in a single folder.

OPTION 2 - Kill extra VFS modules
Add the following auxiliary parameter to your share "vfs objects = streams_xattr". This will make samba stop translating zfs acls (and kill the other hard-coded vfs modules for the share). You will also lose the ability to manage them from explorer and should switch dataset type to "unix", and configure permissions in other ways.

OPTION 3 - Try various AIO settings
See discussion in Samba Mailing lists here: http://marc.info/?l=samba&m=146168243800530&w=2
You can try forcing AIO in samba by setting the following parameters on your share that contains small files

Code:

aio write size = 1
aio read size = 1

The above value represents size in bytes. When file is above the value, then samba will either read or write asynchronously. Based on advice I've received in the past on the samba mailing lists, I'd avoid enabling the aio_pthread vfs object on the share (because it doesn't work on FreeBSD). Just let samba use the aio code baked into smbd.

 

[anodos]

A couple of other things that can be a performance killer:

- Programs that are supposed to make copying go more fasterer
- Security software. If your A/V is trying to scan / perform heuristics on a file transfer with thousands of teensy files, you may be in for a world of performance hurt.

 

[anodos]

Did a bit of benchmarking using the various options (1-3) above using a 1.5GB folder containing 95000 files. Killing case sensitivity may have yielded a speed increase of 5-10%. Not significant enough to justify tweaking things, but it leads me to think that I was bottle-necking elsewhere. Perhaps I hit the speed limit of my client's SSD.

 

[JoGi65]

Perhaps I hit the speed limit of my client's SSD.

I am shure that is not the Problem. When i testet with other File Systems the Difference after a while writing is up to 25:1 what the Client SSD ore other File Systems can do. I think that is the Price we must pay for this "very safe" File System. All other Systems i think, work more with including Ram in Write Process, so i saw at Windows Storage Spaces.

 

[anodos]

I am shure that is not the Problem. When i testet with other File Systems the Difference after a while writing is up to 25:1 what the Client SSD ore other File Systems can do. I think that is the Price we must pay for this "very safe" File System. All other Systems i think, work more with including Ram in Write Process, so i saw at Windows Storage Spaces.

Well, you're talking about a different SMB server there. Based on some samba mailing list posts, it appears that they acknowledge this is a problem (one user showed trace demonstrating samba choking during the write process), but it will probably be a while before the situation improves.

I may do some tracing on my own to see if I can find a choke-point on FreeNAS.

 

[anodos]

I guess I should probably write up what I did for testing so that people can make suggestions for further experimentation.

Experiment setup:
Server 1

• Supermicro X8DTH, Xeon E5645, 48GB ECC Memory, 12 * 7200 RPM Seagate Constellation drives configured in 6 mirrored vdevs.

Server 2

• Supermicro X9SCI, Xeon E3-1220, 32GB ECC Memory, 8 * 7200 RPM Seagate Constellation drives configured in single RAIDZ2 vdev.

Client

• Windows 10. 8GB RAM. Crucial BX100 SSD. No A/V. Realtek gigabit NIC.

Folder to be copied

• Size - 1.17GiB, Contains - 91,145 Files, 7,963 Folders. Average file size = 13.78KB. Unzipped copies of the samba source from various samba versions.
• Robocopy command -
  Code:

  robocopy C:\Users\anodos\Downloads\SAMBAS\ \\FREENAS\CaseInsensitive\SAMBAS /MIR /COPY:DT /FFT /LOG:C:\Users\anodos\Desktop\Robocopy_FN_CI.txt"

Network

• Gigabit. All devices on same switch.

Samba server characteristics:

• AD member servers
• Server max protocol - SMB3_00.
• Log level 1

Raw results

Code:

Server 1 - Unmodified setup
------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :      7964      7963         1         0         0         0
   Files :     91145     91011       134         0         0         0
   Bytes :   1.177 g   1.172 g    4.48 m         0         0         0
   Times :   0:10:19   0:09:12                       0:00:00   0:01:06


   Speed :             2279616 Bytes/sec.
   Speed :             130.440 MegaBytes/min.
   Ended : Monday, May 2, 2016 6:33:40 PM

Server 1 - with AIO write size = 1
------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :      7964      7963         1         0         0         0
   Files :     91145     91011       134         0         0         0
   Bytes :   1.177 g   1.172 g    4.48 m         0         0         0
   Times :   0:10:20   0:09:13                       0:00:00   0:01:07


   Speed :             2276990 Bytes/sec.
   Speed :             130.290 MegaBytes/min.
   Ended : Monday, May 2, 2016 6:53:06 PM

Server 1 - with all VFS objects turned off and AIO write size = 1
------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :      7964      7963         1         0         0         0
   Files :     91145     91011       134         0         0         0
   Bytes :   1.177 g   1.172 g    4.48 m         0         0         0
   Times :   0:10:13   0:09:06                       0:00:00   0:01:06


   Speed :             2303966 Bytes/sec.
   Speed :             131.834 MegaBytes/min.
   Ended : Monday, May 2, 2016 7:28:43 PM

Server 1 - with Case Sensitivity modifications and AIO write size = 1
------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :      7964      7963         1         0         0         0
   Files :     91145     91011       134         0         0         0
   Bytes :   1.177 g   1.172 g    4.48 m         0         0         0
   Times :   0:09:41   0:08:38                       0:00:00   0:01:03


   Speed :             2427587 Bytes/sec.
   Speed :             138.907 MegaBytes/min.
   Ended : Monday, May 2, 2016 7:48:17 PM

Server 2 - with Case Sensitivity modifcations and AIO write size = 1
------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :      7964      7963         1         0         0         0
   Files :     91145     91011       134         0         0         0
   Bytes :   1.177 g   1.172 g    4.48 m         0         0         0
   Times :   0:09:24   0:08:27                       0:00:00   0:00:57


   Speed :             2481255 Bytes/sec.
   Speed :             141.978 MegaBytes/min.
   Ended : Monday, May 2, 2016 8:32:47 PM

Server 2- with Case Sensitivity modifications and aio_pthread vfs object enabled
------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :      7964      7963         1         0         0         0
   Files :     91145     91145         0         0         0         0
   Bytes :   1.177 g   1.177 g         0         0         0         0
   Times :   0:13:00   0:12:03                       0:00:00   0:00:57


   Speed :             1747038 Bytes/sec.
   Speed :              99.966 MegaBytes/min.
   Ended : Tuesday, May 3, 2016 8:10:10 AM

Server 2 - with Case Senstivity modifications and receive buffer tuning bullsh*t
------------------------------------------------------------------------------
               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :      7964      7963         1         0         0         0
   Files :     91145     91145         0         0         0         0
   Bytes :   1.177 g   1.177 g         0         0         0         0
   Times :   0:08:42   0:07:46                       0:00:00   0:00:56


   Speed :             2711848 Bytes/sec.
   Speed :             155.173 MegaBytes/min.
   Ended : Tuesday, May 3, 2016 9:38:54 AM

Server 2 - with Case Sensitivity and Windows Defender disabled - DOH!
------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :      7964      7963         1         0         0         0
   Files :     91145     91145         0         0         0         0
   Bytes :   1.177 g   1.177 g         0         0         0         0
   Times :   0:07:53   0:06:59                       0:00:00   0:00:54


   Speed :             3015391 Bytes/sec.
   Speed :             172.542 MegaBytes/min.
   Ended : Tuesday, May 3, 2016 10:39:34 AM

Observations:

• Prior to disabling Windows Defender, it was using approx 25% CPU on the Windows 10 machine. Robocopy used approx 15%-20% CPU during copy.
• After disabling windows defender and enabling case sensitivity, htop showed CPU core consistently at 100%.

Conclusions

• I saw basically zero zil usage in all tests performed.
• Enabling AIO did not appear to have a significant impact on write speed.
• Disabling VFS modules did not appear to have a significant impact on write speed.
• Disabling oplocks did not appear to have a significant impact on write speed.
• Increasing the receive buffer in samba to 32KB may have had an impact (but it's arguably tuning for a benchmark).
• Case sensitivity modifications appear to have increased write speed by approximately 10 Megabytes/minute.
• Disabling A/V heuristics increased write speed by approximately 20 Megabytes/minute.
• With tweaks I was only able to increase speed from 130 MegaBytes / minute to 172 MegaBytes /minute.
  
• Speed appears to be CPU-bound server-side (single core of E3-1220 pegged at 100%). Possible solution - need moar CPU.

For those interested, here are some stats I pulled during the middle of the transfer:

Code:

Elapsed Times for PID 20321,

         SYSCALL          TIME (ns)
         fstatfs               7906
         madvise               8806
   getdirentries              29591
         fstatat              55799
           close             184569
           mkdir             206742
        __getcwd             744294
          pwrite             847921
       getgroups            1006371
        __sysctl            1425662
       setgroups            1551697
       ftruncate            1611680
        setregid            1824582
         getegid            2528751
         geteuid            2531561
        setreuid            2654942
       utimensat            3582704
          openat            4105631
          writev            6103947
           fstat            6241720
  __acl_get_file            6363662
         sendmsg            8157406
           readv            8811186
           fcntl            8972272
            stat            9092675
           lstat           19978389
            poll          380015963
          TOTAL:          478646429

CPU Times for PID 20321,

         SYSCALL          TIME (ns)
         fstatfs               5990
         madvise               7031
   getdirentries              21938
         fstatat              45195
           close             145129
           mkdir             200055
         geteuid             304880
       getgroups             339524
         getegid             340312
        __getcwd             513722
        __sysctl             746940
          pwrite             789925
        setregid             867182
       setgroups             914937
       ftruncate            1519549
        setreuid            1691843
       utimensat            3377230
           fstat            3622460
          openat            3977775
            poll            4262059
           fcntl            5271314
          writev            5340557
           readv            5429879
  __acl_get_file            5791786
         sendmsg            7678999
            stat            8030229
           lstat           18113941
          TOTAL:           79350381

Syscall Counts for PID 20321,

         SYSCALL              COUNT
         madvise                 11
         fstatfs                 12
           mkdir                 15
   getdirentries                 24
         fstatat                103
          pwrite                135
       ftruncate                142
           close                198
          openat                232
       utimensat                561
        __getcwd                771
         sendmsg               1101
  __acl_get_file               1476
           fstat               1640
          writev               2054
        __sysctl               2196
       setgroups               2196
       getgroups               2216
            poll               2237
            stat               2796
        setregid               3302
        setreuid               3302
           readv               4332
           lstat               5483
         getegid               7718
         geteuid               7726
           fcntl              13099
          TOTAL:              65078

 

[Fireball81]

Thank you anodos for all the effort you`ve put into this, this is some very interesting stuff right there.
Unfortunately i am currently not able to do a few tests of my own, but i will as soon as i have some time to spare.
Until then, thank you very much aain and i am sure this will be helpful for other people too.
Cheers.

 

[anodos]

Thank you anodos for all the effort you`ve put into this, this is some very interesting stuff right there.
Unfortunately i am currently not able to do a few tests of my own, but i will as soon as i have some time to spare.
Until then, thank you very much aain and i am sure this will be helpful for other people too.
Cheers.

Probably the first test I'd try in your situation is with creating a case-insensitive dataset / share as I outlined in post #12. This is pretty much the canonical answer you'd get from the samba mailing list on this problem. Just be careful not to run out of free space on your server.

 

[SteveM7]

I found this thread by searching - had exactly the same problem. FreeNAS-9.10-STABLE-201606270534 (dd17351) on N54L with 16GB EEC RAM and 4 x WD 4TB Red drives in two mirrors. (2 x NASWare version 2 drives and 2 x NASWare version 3 drives, about 2 years apart) Writing large files was very fast - almost saturate 1 gig Ethernet and reading anything was always fast and would max out 1 gig Ethernet. Issue I had was copying music to a cifs share, I tried using NFS share, turning off compression, turning off atime - nothing made much difference and write speeds were still in the 5mbit/s to 15mbit/sec region, indicating something was seriously wrong.

Then I turned off Advanced Power Management on the disks (Storage, View Disks, then select each one and turn off APM) and all of a sudden the write speed jumped up to 86mbit/s which is roughly what I'd expect for 100s of small files and folders like my mp3 music store. (15% overhead for smaller files sounds reasonable)