SMB performance good, NFS bad

[DD4711]

Hello,

I'm currently testing my HP Microserver Gen10 with encrypted drives, currently performance test.

I made an NFS and an SMB share. When I copy a large file from my Linux Mint to the SMB share I get about 90-100 MBps. When I do the same to the NFS share I get less then 20 MBps. These 20 MBps I get exactly from a Synology NAS when copying to the FreeNAS NFS share.

Is this normal? I don't think so. Does anyone have some advice what to check or edit?

Thanks for help :)

 

[kdragon75]

Well, I have a few guess but we need to know what your hardware is before we can say... Please review the forum rules to see what information should be included.

 

[KrisBee]

Well, I have a few guess but we need to know what your hardware is before we can say... Please review the forum rules to see what information should be included.

@DD4711 did say they were using a HP Microserver Gen10 with encrypted drives, but not what CPU was in the machine, nor how much memory, nor what the disk topology is in use, e.g. 4 drives as a pair of mirrors, or raid-z2. But nothing about the synology device - does it even have 1GB nic?

@DD4711 needs to prepare for a barage of info on sync writes etc.

 

[kdragon75]

As @KrisBee hinted, this is likely due to NFS defaulting to synchronous writes. This means that Linux Mint (your client) only gets the ok to write more data once a write is safe on your pool. Further, the default location of ZFSs transaction log (the ZIL) in on the mail pool, its essently writing to slow disks twice before the write is confirmed.

You have two options to address sync writes. The first and safest option is to get something like an Optane NVMe drive and configure it as SLOG (Separate zfs intent LOG) device. This can be almost as small as you like because its only storing data until it saves to the pool. Optane is ideal as its write endurance is extremely high and has extremely low latency.

The second option is to disable sync writes to the data set housing your NFS share. This leaves you open to data loss in the event of a server crash or power outage. This is more than "Shucks, I didn't save that file!" because the writing application *thinks* that data WAS saved but now its gone. This may still be an option if the data is replaceable, the application is tolerant to such faults or you just don't care. Even in the event that you don't care about the application or data, still be sure to have a resilient snapshot and battery backup plan in place. Nothing sucks more than unnecessary down time.

 

[KrisBee]

Possibly need to take a step back here before delving into SLOG devices. @DD4711 needs to give full hardware details of both device on either end of the NFS xfer. Also needs to do a basic iperf test for network xfer speed and determine at least a rough idea of read/write speed on FreeNAS pool and on source. Learning to use ssh and tux on their FreeNAS to connect to spilt pane terminal screen would help them observe the simultaneous output of commands to watch disk writes, network transfer rates and zil usage with gstat , systat -ifstat 1 and zilstat commands. Having said all that, 20Mbps is low.

Samba server operates in async mode unless the client application enforces sync writes. As such on a pool with no SLOG you'd expect smb share xfer rates to be higher than nfs, Data transferred via NFS is likely to appear to happen in bursts, with a lower average speed..

The FreeNAS nfs server operates in sync mode, irrespective of Linux NFS mounts defaulting to async mode. @DD4711 needs to understand at least the basics of how zfs serialises writes into transaction groups (flushed to disk approx very 5 secs by default), and the difference between async writes which bypass the zil and sync writes which makes use of the zil as indicated by monitoring the output of the zilstat command.

For a 4 disk device like the HP Microserver Gen10, is using a SLOG device even possible?

 

[DD4711]

Hello at all:

I thought it is enough to say that with SMB I have great performance ;) So basically it runs with encryption very good.

It's an AMD Opteron X3216 Microserver with 8 GB ECC RAM and 2x4 TB Mirror with WD Red. And I use one GB NIC on a GB Network.

My Laptop is a XMG C405 with a i7 4720, 16 GB RAM and also GB NIC and SSD. Installed is Mint 19.

Is the sync write story not independent of NFS and SMB? Both are on the same Storage Dataset. I thought it is.

Need to read all your posts and get into it!

Thank you first for your answers!

 

[DD4711]

OK, just to clarify it: it's no business network, it's my home network ;) But nevertheless 20 MBps is really slow.

@kdragon75 : I think an Optane NVMe drive is therefore little oversized... So second option should be OK.

@KrisBee : as said SMB wokrs with 5x speed of NFS, Network Problems do not exist. Also r/w speed on FreeNAS pool and on source are therefore no problem.

I checked all possible NFS settings in FreeNAS GUI but I can't find any options I can set to get maybe better performance. It's a bit hard to understand that SMB works with 5x the speed on the not baad Hardware and NFS runs bad without dip deep into the technic.

 

[kdragon75]

https://www.ixsystems.com/documentation/freenas/11.2/storage.html#adding-datasets
Look for "Sync" in the table. This does not need to be a new dataset. You can edit the existing one. Also note this is a live setting. As soon as you select the setting and click save, it will take effect without interruption to any ongoing connections or transfers. Feel free to play with it and see the affect.

 

[DD4711]

OK, I set up "zfs set sync=disabled Storage" and ran another test: speed > 100 MBps

@kdragon75 : OK, I can now see in GUI that it is disabled with my command.

I think I'll let it set off. As told, it's my home network...

THANKS ALL FOR YOUR HELP!