Slow random write on 4x SSD RAIDZ2 pool expected?

[sotiris.bos]

Hey everyone, I have been investigating my slow performance and need help.

The system is a Dell R730 with 2x E5-2630L v3 CPUs and 64GB of RAM, 11.3-U4.1

I have a RAIDZ2 pool with 4x 1TB Samsung 860 EVO SSDs.

My random 4K writes seem bad. Running an fio test from the Freenas shell I get 74.5MiB/s, 19.1K IOPS.

Running the fio test from a Linux server to an NFS share on Freenas I get ~42MiB/s. That is with sync=disabled.
With sync=enabled, I get 1MiB/s from the NFS share.

I am using 16K as recordsize for the dataset.

Is my random 4K performance normal?

fio:

Show

fio --randrepeat=1 --direct=1 --gtod_reduce=1 --name=test --filename=test1 --bs=4k --iodepth=64 --siz
e=4G --readwrite=randwrite --ramp_time=4

test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=64
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [w(1)][83.1%][w=53.8MiB/s][w=13.8k IOPS][eta 00m:10s]
test: (groupid=0, jobs=1): err= 0: pid=10350: Mon Dec 28 14:58:36 2020
write: IOPS=19.1k, BW=74.5MiB/s (78.1MB/s)(3337MiB/44817msec)
bw ( KiB/s): min=13693, max=265181, per=99.41%, avg=75806.60, stdev=53013.15, samples=89
iops : min= 3423, max=66295, avg=18951.33, stdev=13253.32, samples=89
cpu : usr=2.72%, sys=31.44%, ctx=401893, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,854377,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
WRITE: bw=74.5MiB/s (78.1MB/s), 74.5MiB/s-74.5MiB/s (78.1MB/s-78.1MB/s), io=3337MiB (3500MB), run=44817-44817msec

zfs get all:

Show

ssd/vm_storage type filesystem -
ssd/vm_storage creation Wed May 27 18:07 2020 -
ssd/vm_storage used 824G -
ssd/vm_storage available 326G -
ssd/vm_storage referenced 820G -
ssd/vm_storage compressratio 1.03x -
ssd/vm_storage mounted yes -
ssd/vm_storage quota none default
ssd/vm_storage reservation none default
ssd/vm_storage recordsize 16K local
ssd/vm_storage mountpoint /mnt/ssd/vm_storage default
ssd/vm_storage sharenfs off default
ssd/vm_storage checksum on default
ssd/vm_storage compression lz4 inherited from ssd
ssd/vm_storage atime on default
ssd/vm_storage devices on default
ssd/vm_storage exec on default
ssd/vm_storage setuid on default
ssd/vm_storage readonly off default
ssd/vm_storage jailed off default
ssd/vm_storage snapdir hidden default
ssd/vm_storage aclmode passthrough local
ssd/vm_storage aclinherit passthrough inherited from ssd
ssd/vm_storage createtxg 1088059 -
ssd/vm_storage canmount on default
ssd/vm_storage xattr off temporary
ssd/vm_storage copies 1 local
ssd/vm_storage version 5 -
ssd/vm_storage utf8only off -
ssd/vm_storage normalization none -
ssd/vm_storage casesensitivity sensitive -
ssd/vm_storage vscan off default
ssd/vm_storage nbmand off default
ssd/vm_storage sharesmb off default
ssd/vm_storage refquota none default
ssd/vm_storage refreservation none default
ssd/vm_storage guid 10445387050048851856 -
ssd/vm_storage primarycache all default
ssd/vm_storage secondarycache all default
ssd/vm_storage usedbysnapshots 4.33G -
ssd/vm_storage usedbydataset 820G -
ssd/vm_storage usedbychildren 0 -
ssd/vm_storage usedbyrefreservation 0 -
ssd/vm_storage logbias latency default
ssd/vm_storage dedup off default
ssd/vm_storage mlslabel -
ssd/vm_storage sync disabled local
ssd/vm_storage refcompressratio 1.03x -
ssd/vm_storage written 142G -
ssd/vm_storage logicalused 584G -
ssd/vm_storage logicalreferenced 580G -
ssd/vm_storage volmode default default
ssd/vm_storage filesystem_limit none default
ssd/vm_storage snapshot_limit none default
ssd/vm_storage filesystem_count none default
ssd/vm_storage snapshot_count none default
ssd/vm_storage redundant_metadata all default

 

[Chris Moore]

I have a RAIDZ2 pool with 4x 1TB Samsung 860 EVO SSDs.

With only one vdev, you would be roughly limited to the performance of one drive.
If you are trying to use this to store VMs, you probably need to use mirrors to give you more vdevs.
Having more vdevs improves IOPS.

Running the fio test from a Linux server to an NFS share on Freenas I get ~42MiB/s. That is with sync=disabled.

If you are looking at the difference between local performance and network performance, it is possible there is a network bottleneck of some kind.

With sync=enabled, I get 1MiB/s from the NFS share.

That doesn't look right, but having sync enabled ensures all data is committed to stable storage before the acknowledgement is sent. So something is slowing you down. You might need a SLOG device as they do with iSCSI. Here are some reading suggestions:

The ZFS ZIL and SLOG Demystified
https://www.ixsystems.com/blog/zfs-zil-and-slog-demystified/

Some insights into SLOG/ZIL with ZFS on FreeNAS
https://www.ixsystems.com/community/threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/

Testing the benefits of SLOG using a RAM disk!
https://www.ixsystems.com/community/threads/testing-the-benefits-of-slog-using-a-ram-disk.56561/

SLOG benchmarking and finding the best SLOG
https://www.ixsystems.com/community/threads/slog-benchmarking-and-finding-the-best-slog.63521/

How to relocate swap to an SSD or other partition - [not needed in v 11.1 and newer systems]
https://www.ixsystems.com/community/resources/how-to-relocate-swap-to-an-ssd-or-other-partition.68/

Why iSCSI often requires more resources for the same result (block storage)
https://www.ixsystems.com/community...res-more-resources-for-the-same-result.28178/

Some differences between RAIDZ and mirrors, and why we use mirrors for block storage (iSCSI)
https://www.ixsystems.com/community...and-why-we-use-mirrors-for-block-storage.112/