Getting that new NAS running...

[Sharethevibe]

Have in last few months (after some thorough investigation into ZFS and FreeNAS) been building and testing my first FreeNAS-unit.
The unit works, but is not 'running'... the speeds for transferring files and listing folders are way too slow.
So please let me have your raw remarks on the hardware chosen and settings done etc. Dont spare me to be polite... the essence is to get this thing working so I can get ahead with using the NAS.

I have 'newbie' status (wearing that status with pride ;)). My actual status is one of being well involved and on par with general computer building, having some (ancient) programming knowledge (but nothing close to C++). I have made it my task to get a fairly thorough understanding of ZFS and FreeNAS and also learned a lot when doing testing in last months. Yet I lack FreeNAS-experience plus need to be able to discuss (detailed) tech matters with guys that do have that experience, in order to be able to get this NAS of mine here changed where necessary and running.

Use-case:

The NAS will be a dedicated unit for storing, building (processing) and sharing a large musiccollection (~2 million tracks). The files are all fairly large files of either ~10MB or ~40MB. No small files. Just folders filled 50/50 with these 10 or 40MB files. This is relevant as this is approx 25x larger than average files; also the record/blocksize can be large (i.e. 1MB or 4MB; 30x larger than the standard 128kB); this I believe makes transfer-performance and also directory-listing easier as its a lot less IO-activity. Lots of MB per file and per block that is handled. (practise unfortunately does not show this... just read on..).

Performance-needs/lack:

The main performance demands are on the activities that take place between this NAS and a single WindowsPC from which the main processingwork takes place. This is done via a peer-to-peer 10Gb-connection. Next to that there is a normal 1Gb LAN-connection (think in the order of say max 10 web-clients simultanuously).
A SMB/Samba-share is used for exchanging the data with the WindowsPC; data is accessed via a networkconnection as made in Windows Explorer.

The relevant performance and activities where the NAS untill now is way too slow are:

• File-transfer: I'd be happy to get about 500MB/s (then also on-pool file-transfers (half that speed) are OKisch):
  • actual reads: 150MB/s (copy-paste 20GB-batch of 10MB-files; file-cache nearly null (ARC claimed for Metadata); read from NAS, write to RAMdisk)
    
  • actual writes: 300 MB/s (same but vise versa)
• Directory-listing: (opening a folder in Windows Explorer... waiting for the file-data to come on screen...)
  • opening folder from FreeNAS-share: 25 files/sec (often also totally stopping/halting)
    
  • opening same data-folder, from an old 1TB harddisk, under Windows: 50 files/sec (non-stop).
• Deleting files: (right-clicking a folder in Windows Explorer and deleting it)
  • from a folder from FreeNAS-share: 100files/sec
  • 
  • same data-folder, from an old 1TB harddisk, under Windows: 1000 files/sec.

So long story short: this is way too slow for practical work. It's a waiting game with each action that you start...

Alterations/tests done:

Below there is more info on the hardware set-up and all settings, and more on tests done.
In short I obviously have tackled the aspect of IOPS and therefore VDEVs by also testing with 2 and with 4 and even with 8 VDEV's but this did not give any better transfers-speeds. Also I have destroyed this diskpool and created a NVME-SSD-pool: also here no better speeds.
But you can also see from the write-speed (300MB/s), where data is coming from superfast RAMdisk, going via 10Gb=1200MB-NICs, writing to superfast ARC, that lots of that 1200MB/s is lost somewhere.
Similar to this I also did re-re-re-reads of same testbatch (ARC freed for file-cache, to make reads come from ARC) and still got no more than that same 300MB/s.
(again: RAM does ~6000MB/s; networkspeed tested via iperf on 1200MB/s; so again performance-loss from 1200 to 300MB/s).

So to me it looks like the diskpool is not the cause for the slow speeds (even when reading from RAM its slow).
(diskpool itself gives ~700-750MB/s in dd-testing by the way).
And the line-speed is also OK (1200MB/s).

So (assuming this all is OK... we need to doublecheck that somehow), where then does the slow speed come from?
It looks like the handling per file is taking too much time.
(if you transfer very large files, so have less files/sec, the speed goes up considerably) (very large files of say 250MB give 400MB/s read; 600MB/s write).
Even when just plain deleting files, it only does this at 100-150files/sec.
So the system is 'sniffing up and down' each file first and then only processing it (sorry for the vivid expression..).

So I dived into SMB/Samba and worked with several suggested settings. Also installed the new FreeNAS testversion 11.3 Beta1, as in this version newer versions of SMB are included and there's a default group of settings now in that, initiated when choosing for 'SMB-share' in the Dataset (on good advice of Anodos).
Unfortunately none of this made the speed better. On the contrary, only slower (only the deletespeeds got a bit better).

Situ now & my request to you:

So... I am still stuck with a NAS that is not even giving me the performance of an old harddisk under Windows...

Below there is the full data on hardware set-up and all settings, and more on tests done.
Apologies for the 'textwall' and 'datadump' but you cannot really give any good advice when not knowing what the specifics of the system are.

Just dig in and let me know your suggestions on what to change and/or how to test to isolate the speedproblem.
I am available for any questions or to do any tests.

Thanks in advance!


INFO:

• Use-case:
  • Mass quantities (~2 million) of audiofiles; 10MB/40MB size (no small files; just folders filled 50/50 with these larger audio-files)
  • Critical processes are only between 1 WindowsPC and FreeNAS (10Gb peer to peer)
  • Relating to collection-building: large runs (say 100.000 files) in file-transfers/listings/deletes or tagdata-writes.
• Hardware set-up:
  • NAS:
    • FreeNAS v 11.3 Beta1
      SuperMicro X8DTL-iF
      Dual Intel Xeon X5670 CPU's (3Ghz - 6c/12t)
      4x 8GB ECC-RAM
      (L2ARC: Transcend 220s 500GB NVME-SSD (not in use yet)
      IT-flashed Dell H310 controller
      Pool of 8x 8TB WD-Red hard-disks (Z2-VDev)
      NIC: 10Gb Emulex dual-port SFP+
  • (1-on-1 connected via 10Gb direct-attach SFP+cable with 1 Windows10-PC (workstation from which most of the collection-processing is done)
  • Windows10-PC: Mobo Foxconn Intel C602 X79 / dual Xeons E5-2660 / 32 GB RAM / mobo-based 10Gb FSP+ NIC (Intel 82599)
• Settings:
  • Interfaces (NAS/WindowsPC):
    • In the NAS/Interface-setting:
      • added the 'MTU 9000' (to use the large segment size)
      • Furthermore for the NAS-NIC: added a series of settings in the Tunables (please see the screenshots ‘Tunables (1)(2)(3)’ in the attachments
    • For the WindowsPC-NIC: (settings on that adapter):
       On jumbo-frames 9000mtu
       No. of receive buffers 4096
       No. of transmit buffers 16384
       All offloading disabled
       RSS enabled; 16x no of RSS
       Interupt moderations disabled
       File/printer sharing disabled
       Flow control disabled
  • Dataset: sync disabled, compress on, 1MB record/blocksize (also tried 4MB); case Insensitive; Windows-share
  • SMB/Samba-share:
    • (1)(under previous FreeNAS 11.2-U5 install):
      • General SMB-settings: tested this series of settings (added to the SMB-services, aux section) (deleted them again as speed got worse in stead of better…)(indicated best speeds therefore are without these general settings):
        • socket options = iptos_lowdelay so_rcvbuf=131072 so_sndbuf=131072
          min receivefile size = 16384
          use sendfile = yes
          aio read size = 16384
          aio write size = 16384
          server signing = no
          strict allocate = yes
          client signing = no
          client min protocol = smb2
          client max protocol = smb3_11
          server min protocol = smb2
          server max protocol = smb3_11
          log level = 1
      • Settings on Samba-Share:
         Deleted VFS-objects zfsacl and zfs_space ; added Ixnas
         Added in aux parameters:
        case sensitive = true
        mangled names = illegal
        aio write size = 16384
    • (2)(under current FreeNAS 11.3 Beta1 install:)(nice new fresh version..;-)
      • chosen for share-type 'SMB' in the Dataset; this now default gives some optimized settings (not all previous; don't know why)
      • these 'optimised SMB-settings' under 11.3 Beta1 did not improve the speeds.. got slower instead... (just the deletes got some better).
• Tests & results: (below you can see the screenshots):
  • Iperf-test gives 1200MB/s (and I also managed to get about 900MB/s accross when doing real copy-paste of very large files, using a NVME-SSD as pool), so it looks like the (10Gb) NIC to NIC speed is OK
  • dd-test gives 700MB/s write and 750MB/s read; so diskpool-speed is also OK (although the pool is 8 disks and each disk has a seq speed of 200; so 1600 reads is the potential of the diskpool; looks like the RaidZ2 results in say loosing half the bandwidth)
  • real transfer-speeds when doing copy-paste of a 20GBbatch of files, being 10MB in size: (file-cache limited by claiming almost all ARC for metadata-cache)
    • reading from NAS, writing to RAMdisk on WindowsPC: 150MB/s
    • writing to NAS, reading from RAMdisk on WindowsPC: 300MB/s.
    • .
    • especially the 150MB/s reads is troublesome, too slow for practical work
    • (I work with a SMB/Samba-share (entered on the WindowsPC via a networklocation, made in Windows Explorer)
  • Directory-listing (opening folders with lots of files and waiting for all tag-data to list):
    • 25files/sec (vs 50 files/sec for an old, simple harddisk, under Windows direct)
  • Deleting folders/lists of files:
    • 100files/sec (vs 1000 files/sec for an old, simple harddisk, under Windows direct).
• 
  
• My conclusion so far:
  • It is clear that FreeNAS, using the hardware set-up used here, and going via SMB/Samba, is able to perform at least as good/bad as an old, single harddisk under Windows. And also it is clear that just having 150MB/s read-speed is not normal.
  • It looks like there is something wrong in hardware or settings that is making the handling of files go slow.
    (if you transfer really big files (say 250MBfiles)(so very few files per sec), speed is OK (400MB/s read); but if you transfer 10MBfiles (still fairly big!) speed drops quite a lot to that 150MB/s) (even plain deleting of files goes slow; 10x slower than an old harddisk).
• What helps if you can please let me know:
  • What suggestions you have or ideas on where the problem can sit
  • How I could do tests in isolating where the problem is.

 

[Sharethevibe]

Last screenshots showing:

• Waiting-experiences when loading folders, for file/tag-data to show on screen etc.
• Delete-speeds NAS vs simple HDD
• List of set Tunables (mostly aimed at the NAS-NIC)

 

[Jessep]

You don't have enough RAM to think about adding L2ARC, and likely your CPUs are holding you back, they are 5-6 generations old now.
For 2 million tracks I would suggest 128-256GB RAM.

Normally today you wouldn't want to go further back than E5 V2 and for performance E5 V3/V4 with higher clock speed rather than more cores.

For cost I would suggest a new MB/CPU E5 v2 with as much RAM as you can afford, DDR3 is still fairly cheap. Something like this:
https://www.ebay.com/itm/UXS-Server-Supermicro-4U-36-Bay-2x-Xeon-E5-2670-V2-10-Core-192GB-9271-8i-RAID/153185932993?hash=item23aa97d6c1:g:eZMAAOSw-ypbmt~O

NOTE: No idea about the seller, just an example

 

[Jessep]

Good post from @jgreco

Not able to saturate 10Gb connection

Good morning I have the following server: MB: Supermicro X9SRL-F CPU: E5-2650 v2 (8C/16T) RAM: 64GB 1600MHz DDR3 HBAs: 2x LSI 9211-8i flashed to IT-mode Drives: 10x10TB WD Red (white label, shucked from WD External drives) in one ZFS2 pool NIC: Intel X540-T2 2x10Gb RJ45 Other relevant HW...

www.ixsystems.com

 

[Sharethevibe]

Well, I wish it were that easy as just putting in faster CPU or more RAM, Jessup... :

Re the CPU
This is the CPU-load when transferring 10MBfiles (or doing deletes or anything)...

Just max 1% CPU-load... so CPU-load is not at all the bottleneck.

(this also figures as doing just 150-300MB/s and having files of 10MB means just 15-30 actions per sec).

Re the RAM

There is no shortage of RAM at all. So RAM is not the bottleneck.

(another discussion is, in situ when having fully loaded NAS, how much RAM is needed then;
but that's not applicable now:
the topic now is to locate the bottleneck that it has now).

TL;DR:

• system with only 10MB and 40MB files on it; large 1MB (or 4MB) blocksize
• 10Gb peer-to-peer, SFP+ direct-attach cabled
• sync disabled, compression on, case Insensitive, Windows share
• SMB/Samba-share (tried various tunables)(perhaps not the right ones)
• iperf 1200MB/s ; dd-test 750MB/s
• Very slow transfers (reads 150MB/s), dir-listing (half of old harddisk); deletes (1/10 of old harddisk).
  • (also when using all 8 disks set in stripe)(8 Vdevs)
  • (also when using pool on NVME-SSD)

What is bottlenecking this system?
Please let me have your ideas on this. Or on with which test to isolate the problem.

(system shows much better speeds when handling extremely large files (less files/sec);
so it must be related to a process that deals with file-handling)