Samba spoiling FreeNAS?

[Sharethevibe]

I am in the process of setting-up a new NAS (data on hardware set-up and on the use-case etc below in this text).

So this is a 10Gb set-up. (2x 10Gb NIC's ; peer to peer between NAS and 1 Workstation).
Purpose is to handle mass collections.
Large transferruns, folder loading and delete-runs... so very suitable 'food for FreeNAS' I thought ;)

I have set everything up 'according to best iX-forum wisdom'.
Am getting:
- network-speed (iperf-result): ~1200 MB/s (=max 10Gb line-speed)

- diskpool-speed (dd-result): ~800-900 MB/s (the 8 disks can max do 8x200=1600 so loosing about half).

Yet, when copying/pasting from RAM-disk on WindowsPC to Samba-share on NAS, only max ~350 MB/s is what we actually get....

Also:
- loading a folder takes 2x more time than an old, single harddisk (25 files/sec vs 50 files/sec)
- deleting takes 10x more time than an old, single harddisk (100 files/sec vs 1000 files/sec)

My impression is that this is due to Samba and how this handles the files?
Is it not a pity that the good performance of FreeNAS is spoiled so badly?

I have tried various tunings on Samba. Also tried NFS as share-type. Did not help.

Does anyone have a fairly good understanding or good experiences on how we can improve this performance?
Any input would be greatly appreciated as frankly at this moment the FreeNAS-unit is not usable.

Thanks a lot in advance!

DATA:

- set-up of the FreeNAS-unit: see signature
- Windows PC: running on Windows10 (dual Xeons E5-2660 / 32 GB RAM / mobo-based 10Gb NIC (FSP+)
- data-load: mass quantities (~2 million) of audiofiles 10MB/40MB (50/50) ; critical processes are only between Workstation and FreeNAS (10Gb peer to peer). Large file-transfers/listings/deletes or tagdata-writes, by hand or by means of processing software.
- Interface: added the 'MTU 9000 up' (to use the large segment size); and also the '4096 rx' and '16384 tx' (to use the larger windows sizes)(I think this is) (on the rx I get a 'wrong value' comment in the boot-log)
- Dataset: sync off, compress on, 1MB record/blocksize; case Insensitive; Windows-share
- Samba-share:

(changed VFS-objects + added some tunables in the Aux section)
- SMB-config info: see attached picto's ('testparm' 1-5)
(added a series of general tunables in the SMB Aux section; this led to slower speeds; above 'actual speeds' are without these tunings)
- On the Windows PC, I enter the share by 'creating a network connection' and typing the IP-address/share-name; drive with a letter results and in that folder I get to write and read files.
.............................................................................................................................................................................................................................

 

[Sharethevibe]

(Data on the used Tunables)(Not all are enabled)

 

[seanm]

data-load: mass quantities (~2 million) of audiofiles 10MB/40MB (50/50)

Many small files is a common complaint of poor performance. :( I experience it too.

 

[anodos]

Try performance with FreeNAS 11.3-Beta 1. There are optimizations in the beta that aren't present in 11.2, but don't apply any of your tunings. Create new dataset with "SMB" share type. I think Beta2 will have more optimizations in samba for metadata-heavy workloads (lots of small files). I also have an experimental AIO module based on kqueue and FreeBSD's in-kernel AIO. It's probably a bit too early to throw at alpha testers though.

 

[Sharethevibe]

Thanks all on the feedback.

Main points raised are that poor performance is due too special circumstances. And that latest Beta-version might help (and Beta-version to come might even better).

• I am prepared to try and experiment lots. So I will update to the Beta –version and see what happens. But also I believe it is more functional to determine first what the problems are that cause the poor performance. And then to act on that in a focused way.
• Also I do not believe that there is ‘special circumstances’ in this use-case or set-up. So either something is wrong/missing in my settings or Samba all the time is bugging down the performance of FreeNAS. (this perhaps especially being noticed by 10Gb-users and/or people have gotten used to the slow system).

Below I elaborate on these points.
TL;DR:
it's NOT 'small' files but instead large files. So not a 'heavy case' but instead an 'easy case' for the system. So a general problem. Solving best by first identifying where the problem is.

Any input greatly appreciated.


Re ‘lots of small files’ ergo special circumstances

Re ‘small files’:

• File-sizes in general data-collections are:
  • (System-files: 4kB-128kB)
  • Database-files: 4kB-64kB
  • Data-files:
    • Mails: 4kB-100kB
    • Office-docs: 100kB-500kB
    • Pictures: 100kB-2.000kB
    • Music files: 4.000kB-40.000kB
    • Video’s/movies: 10.000kB-2.000.000kB
• Emphasis in NAS-storage is on the above orange area
• My collection as indicated consists of only music-files in either ~10.000kB (10MB) size or ~40.000kB (40MB) size. (heavy-duty MP3 or lossless files)
• So: this is NOT a case of small files; on contrary: it’s a case of fairly large files (only large video’s or movies have larger files).
• A use-case having large files makes it easier on the system to handle say 50GB of data. In my case this is about 2.000 files (avg 25MB file). In general it will be 50.000 files (avg 1MB file).
• .
• Moreover: because the files-size is that large (10MB minimum), I can use very large recordsize/blocksize. I am currently using recordsize of 1MB (have also experimented with 4MB but this did not give better performances). Having a 1MB (1024kB) recordsize in stead of the standard 128kB recordsize, means having 8x less records ergo 8x less metadata that needs to be registered. This too makes it much easier on the system to handle these files.
• .
• So this is not ‘metadata-heavy’ workload; in contrary: it’s 25 x 8 = 200x less heavy from normal load.

Re ‘lots’:

• The number of files in the storage is fairly high (~1-2 million). Also typical runs e.g. transferruns are fairly high (typically 100-200.000 files).
• Yet this does not matter for the performance: also transferring just 5.000 files gives the same slow speed/second. Also then the continuous waiting before the PC is ready and you can continue your work is not normal.

So in conclusion there is no ‘special circumstances’ here that makes it heavier on the system. In contrary: the workload makes it much easier on the system.

So if I am already having such slow performances (for file-transfers, for folder loading etc), how bad will the performance be for workloads with normal (avg 1MB) files…? And normal 128kB-records..?

Normal thinking leads to the conclusion that something must be wrong in my settings. Otherwise all FreeNAS –users using Samba would have a very very slow system. Right?


Re ‘focus on where the problem is’:

So we are seeing:

• Workload consisting of large files/large blocks; less files/MB, less records/MB
• Transferspeed:
  • Network-speed (jperf): ~1200 MB/s
  • Diskpool-speed (dd): ~800-900 MB/s
  • Actual transfer-speed: ~350 MB/s (copy-paste RAMdisk on W10-PC <-> Samba-share)
    • So: diskpool and network are OK; yet actual speed drops >50%; due to Samba-handling?
  • Batch with 40MB-files ~30% faster than 10MB-file-batch
    • So the more files per MB, the slower it goes
    • So handling per file is the problem
• Folder loading: (open a folder in MP3Tag; wait for all file-data/tag-data to be listed)
  • Diskpool (SMB Share tuned) on FreeNAS: 25 files/sec
  • Vs
  • Old, single harddisk under Windows: 50 files/sec.
  • So: very slow handling per file. Even when just loading the tagdata (not the audio/file itself)
• File-deleting:
  • Diskpool (SMB-share tuned) on FreeNAS: 100 files/sec
  • Vs.
  • Old, single harddisk under Windows: 1000 files/sec.
  • So: very, very slow handling per file. Even when just deleting the file.

What is the reason for these low speeds / this slow handling of the files?

What Samba-activity is needed per file, slowing this down?

If we know this we can recognize where the problem is? And we know what to do?



Any feedback greatly appreciated!

 

[KrisBee]

Poor SAMBA performance on large numbers of small files is a common complaint (see for example: https://lists.samba.org/archive/samba/2019-September/226164.html). For non-critical use at home, I stuck to FreeNAS for convenience.

Maybe the solaris/illumos multithreaded in-kernel SMB/CIFS server can do better on the same hardware? A brief test of the latest Ominos CE (v11 r151032 implements SMB3) on my humble home NAS showed this to be true whereas SAMBA on Ubuntu 19.10 server with zfs seems to perform around same level as FreeNAS. Adding the napp-it project to Ominos/Solaris gives a basic webui for management.

 

[Sharethevibe]

@KrisBee:
Thanks for the feedback.
1)
Please note this does NOT relate to 'small files' at all. Moreover, there are only files of 10.000kB - 40.000kB in this NAS-storage.
So this is much larger than the average files in NAS-storage that I think will be approx. 1000 kB.
So this means that this slow performance is a general problem.
And it will be worse for all those cases with avg file-size.

2)
It is my impression that the problem lies within Samba. But I dont know.
I can only suspect this, after having seen that the networks-speed proves to be super (iperf shows 1200 MB/s) and diskpool-speed also proves to be good (dd-test shows 800 MB/s), so then my logical conclusion is, that the problem lies in the conversion from NAS/ZFS to PC/NTFS.

I also see that the smaller (less large) the file is, the slower the performance gets.
So the problem clearly relates to the file-by-file handling of the files.
The handling-time per file takes too long.

3)
It really takes far too long:
- transfer-speed: knocked down to 350 MB/s (where dd on diskpool gives 800-900 MB/s)
- directory-listing: only 25 files/sec, where an old harddisk under Windows does 50 files/sec
- deleting files: only 100 files/sec, where an old harddisk under Windows does 1000 file/sec.

This cannot be a normal situation or can it??

4)
You suggest the 'Solaris/Illumos' kernel as a possible better SMB-element. I cannot judge that, but perhaps Anodos or other FreeNAS experts can.
If the better performance has to come from the fact that this is multi-thread then I suspect it will not bring anything, as I monitored the CPU-usage per thread and basically there is so little going on during Samba-activity that the CPU-threads are no more then say 10% active (and then only 3 from the total of 24 I have). So I dont think the CPU is the bottleneck.
On the other hand, using all available threads wont hurt, and, as said, I am not a good judge. Perhaps Anodos can evaluate your suggestion.


Anodos gave me the hint that there is a Beta1-version of version 11.3 and I noticed that the releasenote mentions 'improved directory-listing performance' so I will upgrade to that version and see what happens.

But:
my consistent main suggestion is to first determine where the exact problem lies (is it indeed SMB/Samba? and what exact aspect of Samba gives the problems then?).
And then to solve this performance problem in a focussed way. (I enjoy 'wrestling' with this computerstuff just as much as all users of this forum, but we very easily get stuck in an endless "try all and error all" - situation.)

Also, my impression still is that there must be something wrong with my set-up or settings.
As I cannot believe that all Samba-users have such slow performance?

 

[anodos]

As I cannot believe that all Samba-users have such slow performance?

They don't. I've seen significantly higher performance for samba on TrueNAS, but troubleshooting the particulars of performance issues goes beyond the scope of what I do here. I'll throw out general tips and recommendations, and help home users that are stuck on permissions issues. Most of the time I spend here is in an unofficial capacity (I'm not a forums admin).