SSD-only RAIDZ2-Pool: Extremly low read performance

[md79]

Hi folks!

I have built a new NAS, based on TrueNAS Scale. First some specs:

• MB: ASUS Pro WS W680-Ace IPMI
• CPU: Intel Core i7-13700 (E-Cores deactivated)
• RAM: 4x Kingston Server Premier DIMM 32GB, DDR5-4800, CL40-39-39, ECC (-> 128 GB RAM)
• HBA: Broadcom HBA 9405W-16i, PCIe 3.1 x16
• OS disks: 2x Micron 7400 MAX - 3DWPD Mixed Use 800GB
• Storage disks: 6x Samsung Enterprise SSD PM1643 7.68TB, SAS 12Gb/s (formated to 4kn)

The storage disks are connected to the HBA and configured in a RAIDZ2 pool:

Code:

pool: storage1
 state: ONLINE
  scan: scrub repaired 0B in 10:39:25 with 0 errors on Tue Aug 29 11:39:27 2023
config:

        NAME                                      STATE     READ WRITE CKSUM
        storage1                                  ONLINE       0     0     0
          raidz2-0                                ONLINE       0     0     0
            6a9ce000-78bc-418c-a457-1bf89a91239f  ONLINE       0     0     0
            93ed437a-1c35-47bf-8c20-783bd0ecc546  ONLINE       0     0     0
            f1a88f56-5bee-4d1e-a488-18e36ed56d06  ONLINE       0     0     0
            ba4a92c4-17ee-4e93-8f61-c357dd574081  ONLINE       0     0     0
            3619ede7-4622-42c4-8f66-a47b60bb7b40  ONLINE       0     0     0
            f74a3e48-3389-47a4-b751-64168ed7416f  ONLINE       0     0     0

Code:

NAME                                       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
storage1                                  41.9T  12.2T  29.7T        -         -     0%    29%  1.00x    ONLINE  /mnt
  raidz2-0                                41.9T  12.2T  29.7T        -         -     0%  29.2%      -    ONLINE
    6a9ce000-78bc-418c-a457-1bf89a91239f  6.98T      -      -        -         -      -      -      -    ONLINE
    93ed437a-1c35-47bf-8c20-783bd0ecc546  6.98T      -      -        -         -      -      -      -    ONLINE
    f1a88f56-5bee-4d1e-a488-18e36ed56d06  6.98T      -      -        -         -      -      -      -    ONLINE
    ba4a92c4-17ee-4e93-8f61-c357dd574081  6.98T      -      -        -         -      -      -      -    ONLINE
    3619ede7-4622-42c4-8f66-a47b60bb7b40  6.98T      -      -        -         -      -      -      -    ONLINE
    f74a3e48-3389-47a4-b751-64168ed7416f  6.98T      -      -        -         -      -      -      -    ONLINE

Record Size: 1M

Write speeds are okay:

Code:

admin@tilikum:$ fio --filename=test --ioengine=posixaio --rw=write --bs=1M --numjobs=1 --iodepth=16 --group_reporting --name=write_test --filesize=50G --direct=1 --runtime=120
write_test: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=posixaio, iodepth=16
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [W(1)][97.5%][w=1199MiB/s][w=1199 IOPS][eta 00m:01s]
write_test: (groupid=0, jobs=1): err= 0: pid=237538: Tue Aug 29 14:42:24 2023
  write: IOPS=1324, BW=1325MiB/s (1389MB/s)(50.0GiB/38650msec); 0 zone resets
    slat (usec): min=9, max=1164, avg=38.69, stdev=17.82
    clat (usec): min=944, max=174486, avg=11984.55, stdev=10424.66
     lat (usec): min=973, max=174536, avg=12023.24, stdev=10430.19
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[    7], 20.00th=[    7],
     | 30.00th=[    8], 40.00th=[    8], 50.00th=[    9], 60.00th=[   10],
     | 70.00th=[   12], 80.00th=[   15], 90.00th=[   22], 95.00th=[   35],
     | 99.00th=[   47], 99.50th=[   53], 99.90th=[  116], 99.95th=[  136],
     | 99.99th=[  171]
   bw (  MiB/s): min=  140, max= 5128, per=99.81%, avg=1322.18, stdev=827.38, samples=77
   iops        : min=  140, max= 5128, avg=1322.18, stdev=827.38, samples=77
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.36%, 4=5.31%, 10=58.03%, 20=25.13%, 50=10.58%
  lat (msec)   : 100=0.44%, 250=0.15%
  cpu          : usr=5.57%, sys=0.14%, ctx=25945, majf=0, minf=44
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.1%, 16=49.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.9%, 8=4.1%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,51200,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1325MiB/s (1389MB/s), 1325MiB/s-1325MiB/s (1389MB/s-1389MB/s), io=50.0GiB (53.7GB), run=38650-38650msec

But read speeds...

Code:

admin@tilikum:$ fio --filename=test --ioengine=posixaio --rw=read --bs=1M --numjobs=1 --iodepth=16 --group_reporting --name=write_test --filesize=50G --direct=1 --runtime=120
write_test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=posixaio, iodepth=16
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=176MiB/s][r=176 IOPS][eta 00m:00s]
write_test: (groupid=0, jobs=1): err= 0: pid=250436: Tue Aug 29 14:47:11 2023
  read: IOPS=220, BW=221MiB/s (232MB/s)(25.9GiB/120081msec)
    slat (nsec): min=58, max=288905, avg=859.40, stdev=2056.97
    clat (msec): min=15, max=165, avg=72.43, stdev=30.34
     lat (msec): min=15, max=165, avg=72.43, stdev=30.34
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   17], 10.00th=[   17], 20.00th=[   21],
     | 30.00th=[   84], 40.00th=[   86], 50.00th=[   87], 60.00th=[   88],
     | 70.00th=[   88], 80.00th=[   89], 90.00th=[   91], 95.00th=[  107],
     | 99.00th=[  113], 99.50th=[  114], 99.90th=[  132], 99.95th=[  138],
     | 99.99th=[  155]
   bw (  KiB/s): min=159744, max=978944, per=100.00%, avg=226151.73, stdev=166843.54, samples=240
   iops        : min=  156, max=  956, avg=220.85, stdev=162.93, samples=240
  lat (msec)   : 20=19.56%, 50=3.13%, 100=71.14%, 250=6.17%
  cpu          : usr=0.13%, sys=0.08%, ctx=13288, majf=0, minf=43
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.0%, 16=50.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=0.0%, 16=4.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=26521,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=221MiB/s (232MB/s), 221MiB/s-221MiB/s (232MB/s-232MB/s), io=25.9GiB (27.8GB), run=120081-120081msec

Code:

admin@tilikum:~$ sudo zpool iostat -ylv storage1 10
                                            capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                      alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
storage1                                  12.3T  29.6T  1.72K      0   180M      0    1ms      -    1ms      -  872ns      -      -      -      -      -
  raidz2-0                                12.3T  29.6T  1.72K      0   180M      0    1ms      -    1ms      -  872ns      -      -      -      -      -
    6a9ce000-78bc-418c-a457-1bf89a91239f      -      -    372      0  44.8M      0    2ms      -    2ms      -  829ns      -      -      -      -      -
    93ed437a-1c35-47bf-8c20-783bd0ecc546      -      -    372      0  44.8M      0    2ms      -    2ms      -  996ns      -      -      -      -      -
    f1a88f56-5bee-4d1e-a488-18e36ed56d06      -      -    332      0  44.7M      0    2ms      -    2ms      -  808ns      -      -      -      -      -
    ba4a92c4-17ee-4e93-8f61-c357dd574081      -      -    332      0  44.6M      0    2ms      -    2ms      -  873ns      -      -      -      -      -
    3619ede7-4622-42c4-8f66-a47b60bb7b40      -      -    156      0   626K      0  124us      -  123us      -  904ns      -      -      -      -      -
    f74a3e48-3389-47a4-b751-64168ed7416f      -      -    195      0   784K      0  269us      -  269us      -  804ns      -      -      -      -      -
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                            capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                      alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
storage1                                  12.3T  29.6T  1.74K      0   182M      0    1ms      -    1ms      -  885ns      -      -      -      -      -
  raidz2-0                                12.3T  29.6T  1.74K      0   182M      0    1ms      -    1ms      -  885ns      -      -      -      -      -
    6a9ce000-78bc-418c-a457-1bf89a91239f      -      -    355      0  45.1M      0    2ms      -    2ms      -  834ns      -      -      -      -      -
    93ed437a-1c35-47bf-8c20-783bd0ecc546      -      -    355      0  45.1M      0    2ms      -    2ms      -    1us      -      -      -      -      -
    f1a88f56-5bee-4d1e-a488-18e36ed56d06      -      -    355      0  45.1M      0    2ms      -    2ms      -  836ns      -      -      -      -      -
    ba4a92c4-17ee-4e93-8f61-c357dd574081      -      -    355      0  45.1M      0    2ms      -    2ms      -  875ns      -      -      -      -      -
    3619ede7-4622-42c4-8f66-a47b60bb7b40      -      -    177      0   711K      0  112us      -  112us      -  930ns      -      -      -      -      -
    f74a3e48-3389-47a4-b751-64168ed7416f      -      -    177      0   711K      0  221us      -  221us      -  805ns      -      -      -      -      -
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                            capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                      alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
storage1                                  12.3T  29.6T  1.85K      0   194M      0  943us      -  943us      -  869ns      -      -      -      -      -
  raidz2-0                                12.3T  29.6T  1.85K      0   194M      0  943us      -  943us      -  869ns      -      -      -      -      -
    6a9ce000-78bc-418c-a457-1bf89a91239f      -      -    545      0  48.8M      0    1ms      -    1ms      -  842ns      -      -      -      -      -
    93ed437a-1c35-47bf-8c20-783bd0ecc546      -      -    392      0  10.6M      0  502us      -  501us      -    1us      -      -      -      -      -
    f1a88f56-5bee-4d1e-a488-18e36ed56d06      -      -     60      0  9.27M      0    2ms      -    2ms      -  803ns      -      -      -      -      -
    ba4a92c4-17ee-4e93-8f61-c357dd574081      -      -    213      0  47.6M      0    1ms      -    1ms      -  831ns      -      -      -      -      -
    3619ede7-4622-42c4-8f66-a47b60bb7b40      -      -    177      0  38.4M      0  685us      -  685us      -  869ns      -      -      -      -      -
    f74a3e48-3389-47a4-b751-64168ed7416f      -      -    508      0  39.7M      0  285us      -  285us      -  807ns      -      -      -      -      -
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                            capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                      alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
storage1                                  12.3T  29.6T  1.80K      0   188M      0    1ms      -    1ms      -  873ns      -      -      -      -      -
  raidz2-0                                12.3T  29.6T  1.80K      0   188M      0    1ms      -    1ms      -  873ns      -      -      -      -      -
    6a9ce000-78bc-418c-a457-1bf89a91239f      -      -    552      0  47.4M      0  328us      -  328us      -  833ns      -      -      -      -      -
    93ed437a-1c35-47bf-8c20-783bd0ecc546      -      -    547      0  46.2M      0  291us      -  291us      -  955ns      -      -      -      -      -
    f1a88f56-5bee-4d1e-a488-18e36ed56d06      -      -      0      0      0      0      -      -      -      -      -      -      -      -      -      -
    ba4a92c4-17ee-4e93-8f61-c357dd574081      -      -      4      0  1.20M      0    1ms      -    1ms      -  800ns      -      -      -      -      -
    3619ede7-4622-42c4-8f66-a47b60bb7b40      -      -    183      0  46.0M      0    4ms      -    4ms      -  876ns      -      -      -      -      -
    f74a3e48-3389-47a4-b751-64168ed7416f      -      -    551      0  47.4M      0    1ms      -    1ms      -  830ns      -      -      -      -      -
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                            capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                      alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
storage1                                  12.3T  29.6T  4.24K      0   445M      0  480us      -  477us      -  560ns      -      -      -      -      -
  raidz2-0                                12.3T  29.6T  4.24K      0   445M      0  480us      -  477us      -  560ns      -      -      -      -      -
    6a9ce000-78bc-418c-a457-1bf89a91239f      -      -    597      0  30.0M      0  195us      -  195us      -  613ns      -      -      -      -      -
    93ed437a-1c35-47bf-8c20-783bd0ecc546      -      -    919      0   111M      0  328us      -  325us      -  634ns      -      -      -      -      -
    f1a88f56-5bee-4d1e-a488-18e36ed56d06      -      -    706      0  82.0M      0  297us      -  294us      -  469ns      -      -      -      -      -
    ba4a92c4-17ee-4e93-8f61-c357dd574081      -      -    706      0  82.0M      0  299us      -  297us      -  478ns      -      -      -      -      -
    3619ede7-4622-42c4-8f66-a47b60bb7b40      -      -    818      0   110M      0  874us      -  870us      -  552ns      -      -      -      -      -
    f74a3e48-3389-47a4-b751-64168ed7416f      -      -    597      0  30.0M      0  888us      -  887us      -  607ns      -      -      -      -      -
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----

Real world copy (dataset@storage1 -> dataset@nvme pool; sequential/one big file):

Code:

admin@tilikum:~$ sudo zpool iostat -ylv storage1 10
                                            capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                      alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
storage1                                  12.3T  29.6T  1.26K      0   321M      0  102ms      -    5ms      -  224ms      -   71ms      -      -      -
  raidz2-0                                12.3T  29.6T  1.26K      0   321M      0  102ms      -    5ms      -  224ms      -   71ms      -      -      -
    6a9ce000-78bc-418c-a457-1bf89a91239f      -      -    176      0  44.0M      0  167ms      -   10ms      -  218ms      -   95ms      -      -      -
    93ed437a-1c35-47bf-8c20-783bd0ecc546      -      -    274      0  68.4M      0  199ms      -    8ms      -  216ms      -  136ms      -      -      -
    f1a88f56-5bee-4d1e-a488-18e36ed56d06      -      -    147      0  36.7M      0   64ms      -    5ms      -      -      -   51ms      -      -      -
    ba4a92c4-17ee-4e93-8f61-c357dd574081      -      -    147      0  36.7M      0   51ms      -    5ms      -      -      -   51ms      -      -      -
    3619ede7-4622-42c4-8f66-a47b60bb7b40      -      -    320      0  79.9M      0   46ms      -    3ms      -  201ms      -   42ms      -      -      -
    f74a3e48-3389-47a4-b751-64168ed7416f      -      -    222      0  55.5M      0   68ms      -    3ms      -  301ms      -   42ms      -      -      -
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                            capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                      alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
storage1                                  12.3T  29.6T    960      0   239M      0  237ms      -   12ms      -  283ms      -  194ms      -      -      -
  raidz2-0                                12.3T  29.6T    960      0   239M      0  237ms      -   12ms      -  283ms      -  194ms      -      -      -
    6a9ce000-78bc-418c-a457-1bf89a91239f      -      -      0      0  2.80K      0  196us      -  196us      -      -      -    1us      -      -      -
    93ed437a-1c35-47bf-8c20-783bd0ecc546      -      -      0      0  2.80K      0  196us      -  196us      -      -      -    2us      -      -      -
    f1a88f56-5bee-4d1e-a488-18e36ed56d06      -      -    237      0  59.3M      0  220ms      -   12ms      -  301ms      -  188ms      -      -      -
    ba4a92c4-17ee-4e93-8f61-c357dd574081      -      -    238      0  59.5M      0  188ms      -   12ms      -  335ms      -  188ms      -      -      -
    3619ede7-4622-42c4-8f66-a47b60bb7b40      -      -    241      0  60.2M      0  203ms      -   12ms      -  230ms      -  201ms      -      -      -
    f74a3e48-3389-47a4-b751-64168ed7416f      -      -    241      0  60.3M      0  337ms      -   12ms      -  301ms      -  201ms      -      -      -
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                            capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                      alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
storage1                                  12.3T  29.6T    962      0   240M      0  252ms      -   12ms      -  295ms      -  201ms      -      -      -
  raidz2-0                                12.3T  29.6T    962      0   240M      0  252ms      -   12ms      -  295ms      -  201ms      -      -      -
    6a9ce000-78bc-418c-a457-1bf89a91239f      -      -      0      0  1.60K      0    6ms      -    6ms      -    1us      -    1us      -      -      -
    93ed437a-1c35-47bf-8c20-783bd0ecc546      -      -      0      0  1.20K      0  196us      -  196us      -      -      -    2us      -      -      -
    f1a88f56-5bee-4d1e-a488-18e36ed56d06      -      -    240      0  60.0M      0  343ms      -   12ms      -  320ms      -  201ms      -      -      -
    ba4a92c4-17ee-4e93-8f61-c357dd574081      -      -    240      0  59.9M      0  207ms      -   12ms      -  308ms      -  201ms      -      -      -
    3619ede7-4622-42c4-8f66-a47b60bb7b40      -      -    240      0  60.0M      0  201ms      -   12ms      -  316ms      -  201ms      -      -      -
    f74a3e48-3389-47a4-b751-64168ed7416f      -      -    240      0  60.0M      0  256ms      -   12ms      -  255ms      -  201ms      -      -      -
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                            capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                      alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
storage1                                  12.3T  29.6T    977      0   244M      0  170ms      -   10ms      -  311ms      -  122ms      -      -      -
  raidz2-0                                12.3T  29.6T    977      0   244M      0  170ms      -   10ms      -  311ms      -  122ms      -      -      -
    6a9ce000-78bc-418c-a457-1bf89a91239f      -      -     82      0  20.5M      0    7ms      -    7ms      -      -      -   77us      -      -      -
    93ed437a-1c35-47bf-8c20-783bd0ecc546      -      -     82      0  20.5M      0    8ms      -    8ms      -      -      -   91us      -      -      -
    f1a88f56-5bee-4d1e-a488-18e36ed56d06      -      -    240      0  59.9M      0  328ms      -   12ms      -  301ms      -  201ms      -      -      -
    ba4a92c4-17ee-4e93-8f61-c357dd574081      -      -    163      0  40.8M      0  230ms      -   11ms      -  316ms      -  143ms      -      -      -
    3619ede7-4622-42c4-8f66-a47b60bb7b40      -      -    163      0  40.8M      0  146ms      -   11ms      -  352ms      -  143ms      -      -      -
    f74a3e48-3389-47a4-b751-64168ed7416f      -      -    245      0  61.4M      0  102ms      -    7ms      -      -      -   98ms      -      -      -
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                            capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                      alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
storage1                                  12.3T  29.6T  1.50K      0   384M      0   59ms      -    4ms      -  279ms      -   33ms      -      -      -
  raidz2-0                                12.3T  29.6T  1.50K      0   384M      0   59ms      -    4ms      -  279ms      -   33ms      -      -      -
    6a9ce000-78bc-418c-a457-1bf89a91239f      -      -     93      0  23.3M      0    1ms      -    1ms      -      -      -   10us      -      -      -
    93ed437a-1c35-47bf-8c20-783bd0ecc546      -      -     93      0  23.3M      0    1ms      -    1ms      -      -      -    9us      -      -      -
    f1a88f56-5bee-4d1e-a488-18e36ed56d06      -      -    389      0  97.1M      0   56ms      -    4ms      -  281ms      -   30ms      -      -      -
    ba4a92c4-17ee-4e93-8f61-c357dd574081      -      -    290      0  72.3M      0   15ms      -    5ms      -  402ms      -   12ms      -      -      -
    3619ede7-4622-42c4-8f66-a47b60bb7b40      -      -    290      0  72.2M      0   13ms      -    4ms      -  402ms      -   11ms      -      -      -
    f74a3e48-3389-47a4-b751-64168ed7416f      -      -    382      0  95.5M      0  157ms      -    6ms      -  216ms      -   86ms      -      -      -
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                            capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                      alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
storage1                                  12.3T  29.6T  1.20K      0   306M      0   73ms      -    5ms      -  213ms      -   41ms      -      -      -
  raidz2-0                                12.3T  29.6T  1.20K      0   306M      0   73ms      -    5ms      -  213ms      -   41ms      -      -      -
    6a9ce000-78bc-418c-a457-1bf89a91239f      -      -    172      0  43.2M      0  159ms      -    6ms      -  241ms      -   95ms      -      -      -
    93ed437a-1c35-47bf-8c20-783bd0ecc546      -      -     97      0  24.4M      0   15ms      -    2ms      -      -      -   13ms      -      -      -
    f1a88f56-5bee-4d1e-a488-18e36ed56d06      -      -    226      0  56.4M      0   16ms      -    4ms      -  301ms      -    8ms      -      -      -
    ba4a92c4-17ee-4e93-8f61-c357dd574081      -      -    226      0  56.4M      0   16ms      -    4ms      -  402ms      -   11ms      -      -      -
    3619ede7-4622-42c4-8f66-a47b60bb7b40      -      -    210      0  52.4M      0    4ms      -    3ms      -      -      -  387us      -      -      -
    f74a3e48-3389-47a4-b751-64168ed7416f      -      -    291      0  72.9M      0  181ms      -    6ms      -  150ms      -   99ms      -      -      -
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----

All tests were done on a dataset with "primarycache=none".

For a pool with 6 disks where each disk have these specs...

• Sequential Read: 2,100 MB/s
• Sequential Write: 2,000 MB/s
• Random Read (QD = 64): 400K IOPS
• Random Write (QD = 64): 70K IOPS

...these values seem much too low to me.

Any hints and suggestions?

 

[tangles]

Terrible latency.

I had issues with Samsung SATA disks connected to SAS HBAs years and years ago. Other disks worked fine on the same HBA. I wonder if you've got some conflict going on here because your fio command seems fine when I run the read test.

Do you have access to another SAS HBA to rule it out?
Assuming you have a spare, create a single disk pool and run the same tests as a baseline.

Also look at zpool history storage1 | grep create to see what attributes were set and start experimenting from there to see if you get improvement.

I ran your same read command to get an idea for you:

A pair of mirrored 1TB NVMEs I use for iSCSI:

Show : SSD output

root@TrueNAS[/mnt/scuzzy]# fio --filename=test --ioengine=posixaio --rw=read --bs=1M --numjobs=1 --iodepth=16 --group_reporting --name=write_test --filesize=50G --direct=1 --runtime=120
write_test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=posixaio, iodepth=16
fio-3.25
Starting 1 process
write_test: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=2149MiB/s][r=2149 IOPS][eta 00m:00s]
write_test: (groupid=0, jobs=1): err= 0: pid=435598: Wed Aug 30 12:21:11 2023
read: IOPS=2103, BW=2104MiB/s (2206MB/s)(50.0GiB/24338msec)
slat (nsec): min=121, max=361195, avg=1471.09, stdev=2488.66
clat (usec): min=4696, max=13806, avg=7591.33, stdev=443.66
lat (usec): min=4697, max=13810, avg=7592.80, stdev=443.78
clat percentiles (usec):
| 1.00th=[ 6521], 5.00th=[ 7111], 10.00th=[ 7242], 20.00th=[ 7373],
| 30.00th=[ 7439], 40.00th=[ 7570], 50.00th=[ 7635], 60.00th=[ 7635],
| 70.00th=[ 7701], 80.00th=[ 7767], 90.00th=[ 7832], 95.00th=[ 8029],
| 99.00th=[ 9241], 99.50th=[10290], 99.90th=[12125], 99.95th=[12125],
| 99.99th=[12256]
bw ( MiB/s): min= 1832, max= 2296, per=100.00%, avg=2104.50, stdev=65.33, samples=48
iops : min= 1832, max= 2296, avg=2104.50, stdev=65.33, samples=48
lat (msec) : 10=99.29%, 20=0.71%
cpu : usr=2.08%, sys=0.84%, ctx=25604, majf=0, minf=92
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.0%, 16=50.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.8%, 8=0.2%, 16=3.9%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=51200,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
READ: bw=2104MiB/s (2206MB/s), 2104MiB/s-2104MiB/s (2206MB/s-2206MB/s), io=50.0GiB (53.7GB), run=24338-24338msec
root@TrueNAS[/mnt/scuzzy]#

Rotational 12TB disks in a 1 x RAIDZ2 | 6 wide config connected to a LSI SAS HBA for basic SMB sharing:

Show : ROT output

root@TrueNAS[/mnt/ztank]# fio --filename=test --ioengine=posixaio --rw=read --bs=1M --numjobs=1 --iodepth=16 --group_reporting --name=write_test --filesize=50G --direct=1 --runtime=120
write_test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=posixaio, iodepth=16
fio-3.25
Starting 1 process
write_test: Laying out IO file (1 file / 51200MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=2114MiB/s][r=2114 IOPS][eta 00m:00s]
write_test: (groupid=0, jobs=1): err= 0: pid=468757: Wed Aug 30 12:25:23 2023
read: IOPS=2087, BW=2088MiB/s (2189MB/s)(50.0GiB/24523msec)
slat (nsec): min=131, max=301409, avg=1137.78, stdev=2002.05
clat (usec): min=5590, max=22112, avg=7650.08, stdev=960.82
lat (usec): min=5595, max=22124, avg=7651.22, stdev=961.07
clat percentiles (usec):
| 1.00th=[ 6587], 5.00th=[ 7046], 10.00th=[ 7242], 20.00th=[ 7308],
| 30.00th=[ 7373], 40.00th=[ 7439], 50.00th=[ 7439], 60.00th=[ 7504],
| 70.00th=[ 7570], 80.00th=[ 7701], 90.00th=[ 7963], 95.00th=[ 8586],
| 99.00th=[12125], 99.50th=[13829], 99.90th=[16581], 99.95th=[18744],
| 99.99th=[21890]
bw ( MiB/s): min= 1500, max= 2336, per=100.00%, avg=2089.12, stdev=165.38, samples=49
iops : min= 1500, max= 2336, avg=2089.10, stdev=165.44, samples=49
lat (msec) : 10=96.69%, 20=3.26%, 50=0.04%
cpu : usr=1.54%, sys=0.54%, ctx=25624, majf=0, minf=110
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.0%, 16=50.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=95.8%, 8=3.9%, 16=0.3%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=51200,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
READ: bw=2088MiB/s (2189MB/s), 2088MiB/s-2088MiB/s (2189MB/s-2189MB/s), io=50.0GiB (53.7GB), run=24523-24523msec
root@TrueNAS[/mnt/ztank]#

Hope this helps you hone in on your issue.

 

[md79]

Terrible latency.

I had issues with Samsung SATA disks connected to SAS HBAs years and years ago. Other disks worked fine on the same HBA. I wonder if you've got some conflict going on here because your fio command seems fine when I run the read test.

Thanks for your great reply! This could take me a few steps further.

Do you have access to another SAS HBA to rule it out?

When I have some time, I'll check out some old servers here at work. Maybe I will find one. But if I remember correctly, the old ones didn't have an HBA built in, just a simple SD card slot.

Also look at zpool history storage1 | grep create to see what attributes were set and start experimenting from there to see if you get improvement.

Thank you for the hint. I will do that.

Assuming you have a spare, create a single disk pool and run the same tests as a baseline.

I have 6 more of these (PM1633a, not PM1643 btw. My fault...). So I could do some testing.

At least for the MegaRAID 94xx the Series PM1633a ist listed as compatible. Though it's not the same disk and controller model (MegaRAID vs. HBA). But I wonder if my re-formating to 4kn might have an impact in conjunction with this controller. Because all disks were testet with a sectro size of 512 bytes. MegaRAID 94xx Tri-Mode Storage Adapter Drive Compatibility

The testing will take some time. Especially when I do the re-formating.

 

[md79]

Just a little interim update. New pool, one device...

Code:

admin@tilikum:$ fio --filename=test --ioengine=posixaio --rw=read --bs=1M --numjobs=1 --iodepth=16 --group_reporting --name=read_test --filesize=50G --direct=1 --runtime=120
read_test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=posixaio, iodepth=16
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=49.0MiB/s][r=49 IOPS][eta 00m:00s]
read_test: (groupid=0, jobs=1): err= 0: pid=3781864: Wed Aug 30 20:26:52 2023
  read: IOPS=144, BW=145MiB/s (152MB/s)(17.0GiB/120280msec)
    slat (nsec): min=42, max=308731, avg=781.92, stdev=2620.24
    clat (msec): min=26, max=540, avg=110.45, stdev=117.40
     lat (msec): min=26, max=540, avg=110.45, stdev=117.40
    clat percentiles (msec):
     |  1.00th=[   29],  5.00th=[   29], 10.00th=[   29], 20.00th=[   30],
     | 30.00th=[   31], 40.00th=[   32], 50.00th=[   32], 60.00th=[   33],
     | 70.00th=[  275], 80.00th=[  284], 90.00th=[  284], 95.00th=[  288],
     | 99.00th=[  288], 99.50th=[  288], 99.90th=[  292], 99.95th=[  359],
     | 99.99th=[  523]
   bw (  KiB/s): min=40960, max=561152, per=100.00%, avg=148531.20, stdev=186471.88, samples=240
   iops        : min=   40, max=  548, avg=145.05, stdev=182.10, samples=240
  lat (msec)   : 50=68.33%, 100=0.01%, 250=0.26%, 500=31.38%, 750=0.01%
  cpu          : usr=0.08%, sys=0.04%, ctx=8731, majf=0, minf=43
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.0%, 16=50.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=0.0%, 16=4.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=17421,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=17.0GiB (18.3GB), run=120280-120280msec

Code:

admin@tilikum:~$ sudo zpool iostat -ylv read_test_pool 10
                                          capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                    alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
read_test_pool                          50.0G  6.92T    112      0  57.1M      0   10ms      -   10ms      -    1us      -      -      -      -      -
  9655cf75-b297-4710-817a-d6d4df0ef5b4  50.0G  6.92T    112      0  57.1M      0   10ms      -   10ms      -    1us      -      -      -      -      -
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                          capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                    alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
read_test_pool                          50.0G  6.92T    112      0  57.2M      0   10ms      -   10ms      -    1us      -      -      -      -      -
  9655cf75-b297-4710-817a-d6d4df0ef5b4  50.0G  6.92T    112      0  57.2M      0   10ms      -   10ms      -    1us      -      -      -      -      -
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                          capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                    alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
read_test_pool                          50.0G  6.92T    113      0  57.3M      0   10ms      -   10ms      -    1us      -      -      -      -      -
  9655cf75-b297-4710-817a-d6d4df0ef5b4  50.0G  6.92T    113      0  57.3M      0   10ms      -   10ms      -    1us      -      -      -      -      -
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                          capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                    alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
read_test_pool                          50.0G  6.92T    113      0  57.3M      0   10ms      -   10ms      -    1us      -      -      -      -      -
  9655cf75-b297-4710-817a-d6d4df0ef5b4  50.0G  6.92T    113      0  57.3M      0   10ms      -   10ms      -    1us      -      -      -      -      -
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                          capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                    alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
read_test_pool                          50.0G  6.92T    113      0  57.6M      0   10ms      -   10ms      -    1us      -      -      -      -      -
  9655cf75-b297-4710-817a-d6d4df0ef5b4  50.0G  6.92T    113      0  57.6M      0   10ms      -   10ms      -    1us      -      -      -      -      -
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                          capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                    alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
read_test_pool                          50.0G  6.92T    113      0  57.3M      0   10ms      -   10ms      -    1us      -      -      -      -      -
  9655cf75-b297-4710-817a-d6d4df0ef5b4  50.0G  6.92T    113      0  57.3M      0   10ms      -   10ms      -    1us      -      -      -      -      -
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                          capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                    alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
read_test_pool                          50.0G  6.92T    899      0   455M      0  999us      -  999us      -  392ns      -      -      -      -      -
  9655cf75-b297-4710-817a-d6d4df0ef5b4  50.0G  6.92T    899      0   455M      0  999us      -  999us      -  392ns      -      -      -      -      -
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                          capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                    alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
read_test_pool                          50.0G  6.92T  1.09K      0   531M      0  762us      -  762us      -  348ns      -      -      -      -      -
  9655cf75-b297-4710-817a-d6d4df0ef5b4  50.0G  6.92T  1.09K      0   531M      0  762us      -  762us      -  348ns      -      -      -      -      -
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                          capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                    alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
read_test_pool                          50.0G  6.92T    556      0   190M      0    1ms      -    1ms      -  608ns      -      -      -      -      -
  9655cf75-b297-4710-817a-d6d4df0ef5b4  50.0G  6.92T    556      0   190M      0    1ms      -    1ms      -  608ns      -      -      -      -      -
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                          capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                    alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
read_test_pool                          50.0G  6.92T    168      0  57.5M      0    6ms      -    6ms      -    1us      -      -      -      -      -
  9655cf75-b297-4710-817a-d6d4df0ef5b4  50.0G  6.92T    168      0  57.5M      0    6ms      -    6ms      -    1us      -      -      -      -      -
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                          capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                    alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
read_test_pool                          50.0G  6.92T    168      0  57.4M      0    6ms      -    6ms      -    1us      -      -      -      -      -
  9655cf75-b297-4710-817a-d6d4df0ef5b4  50.0G  6.92T    168      0  57.4M      0    6ms      -    6ms      -    1us      -      -      -      -      -
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----

Write...

Code:

admin@tilikum:$ fio --filename=test --ioengine=posixaio --rw=write --bs=1M --numjobs=1 --iodepth=16 --group_reporting --name=write_test --filesize=50G --direct=1 --runtime=120
write_test: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=posixaio, iodepth=16
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=489MiB/s][w=489 IOPS][eta 00m:00s]
write_test: (groupid=0, jobs=1): err= 0: pid=3769885: Wed Aug 30 20:22:51 2023
  write: IOPS=515, BW=516MiB/s (541MB/s)(50.0GiB/99291msec); 0 zone resets
    slat (usec): min=5, max=4260, avg=72.08, stdev=36.14
    clat (usec): min=409, max=46851, avg=30815.49, stdev=7716.69
     lat (usec): min=433, max=46948, avg=30887.57, stdev=7727.14
    clat percentiles (usec):
     |  1.00th=[ 2245],  5.00th=[ 3916], 10.00th=[29492], 20.00th=[31851],
     | 30.00th=[32375], 40.00th=[32900], 50.00th=[33162], 60.00th=[33817],
     | 70.00th=[33817], 80.00th=[33817], 90.00th=[34341], 95.00th=[34341],
     | 99.00th=[34866], 99.50th=[35390], 99.90th=[37487], 99.95th=[39584],
     | 99.99th=[43779]
   bw (  KiB/s): min=458752, max=5701632, per=100.00%, avg=528271.14, stdev=382367.95, samples=198
   iops        : min=  448, max= 5568, avg=515.90, stdev=373.41, samples=198
  lat (usec)   : 500=0.01%
  lat (msec)   : 2=0.83%, 4=4.20%, 10=1.39%, 20=1.67%, 50=91.90%
  cpu          : usr=4.00%, sys=0.08%, ctx=29310, majf=0, minf=47
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.0%, 16=49.9%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=2.8%, 16=1.4%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,51200,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=516MiB/s (541MB/s), 516MiB/s-516MiB/s (541MB/s-541MB/s), io=50.0GiB (53.7GB), run=99291-99291msec

Next step: 4kn -> 512e

 

[md79]

Okay, I think I got it...

One single device@512e...

Code:

admin@tilikum:$ fio --filename=test --ioengine=posixaio --rw=read --bs=1M --numjobs=1 --iodepth=16 --group_reporting --name=read_test --filesize=50G --direct=1 --runtime=120
read_test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=posixaio, iodepth=16
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [R(1)][94.3%][r=1621MiB/s][r=1621 IOPS][eta 00m:05s]
read_test: (groupid=0, jobs=1): err= 0: pid=325565: Wed Aug 30 23:14:41 2023
  read: IOPS=611, BW=612MiB/s (641MB/s)(50.0GiB/83724msec)
    slat (nsec): min=41, max=282930, avg=342.66, stdev=1311.24
    clat (usec): min=1035, max=45620, avg=26160.13, stdev=8190.52
     lat (usec): min=1035, max=45621, avg=26160.47, stdev=8190.56
    clat percentiles (usec):
     |  1.00th=[ 1352],  5.00th=[ 5276], 10.00th=[ 8979], 20.00th=[25560],
     | 30.00th=[27132], 40.00th=[28181], 50.00th=[29230], 60.00th=[29492],
     | 70.00th=[30016], 80.00th=[31589], 90.00th=[32113], 95.00th=[32375],
     | 99.00th=[34341], 99.50th=[35390], 99.90th=[39584], 99.95th=[41157],
     | 99.99th=[45876]
   bw (  KiB/s): min=493568, max=2910208, per=97.51%, avg=610586.06, stdev=245930.50, samples=167
   iops        : min=  482, max= 2842, avg=596.28, stdev=240.17, samples=167
  lat (msec)   : 2=1.80%, 4=0.56%, 10=8.01%, 20=4.86%, 50=84.76%
  cpu          : usr=0.17%, sys=0.06%, ctx=25621, majf=0, minf=45
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=50.0%, 16=50.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=95.8%, 8=0.1%, 16=4.2%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=51200,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=612MiB/s (641MB/s), 612MiB/s-612MiB/s (641MB/s-641MB/s), io=50.0GiB (53.7GB), run=83724-83724msec

Code:

admin@tilikum:~$ sudo zpool iostat -ylv read_test_pool 10
                                          capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                    alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
read_test_pool                          50.0G  6.92T  1.09K      0   550M      0  865us      -  865us      -  364ns      -      -      -      -      -
  fa51cf3e-5ce0-40fd-bcde-5b298b721d55  50.0G  6.92T  1.09K      0   550M      0  865us      -  865us      -  364ns      -      -      -      -      -
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                          capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                    alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
read_test_pool                          50.0G  6.92T  1.04K      0   534M      0  879us      -  879us      -  380ns      -      -      -      -      -
  fa51cf3e-5ce0-40fd-bcde-5b298b721d55  50.0G  6.92T  1.04K      0   534M      0  879us      -  879us      -  380ns      -      -      -      -      -
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                          capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                    alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
read_test_pool                          50.0G  6.92T  1.07K      0   546M      0  868us      -  868us      -  344ns      -      -      -      -      -
  fa51cf3e-5ce0-40fd-bcde-5b298b721d55  50.0G  6.92T  1.07K      0   546M      0  868us      -  868us      -  344ns      -      -      -      -      -
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                          capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                    alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
read_test_pool                          50.0G  6.92T  1.07K      0   540M      0  858us      -  857us      -  368ns      -      -      -      -      -
  fa51cf3e-5ce0-40fd-bcde-5b298b721d55  50.0G  6.92T  1.07K      0   540M      0  858us      -  857us      -  368ns      -      -      -      -      -
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                          capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                    alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
read_test_pool                          50.0G  6.92T  1.04K      0   529M      0  873us      -  873us      -  353ns      -      -      -      -      -
  fa51cf3e-5ce0-40fd-bcde-5b298b721d55  50.0G  6.92T  1.04K      0   529M      0  873us      -  873us      -  353ns      -      -      -      -      -
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                          capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                    alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
read_test_pool                          50.0G  6.92T  1.04K      0   526M      0  865us      -  865us      -  385ns      -      -      -      -      -
  fa51cf3e-5ce0-40fd-bcde-5b298b721d55  50.0G  6.92T  1.04K      0   526M      0  865us      -  865us      -  385ns      -      -      -      -      -
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
                                          capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                    alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
read_test_pool                          50.0G  6.92T  1.01K      0   517M      0  873us      -  873us      -  391ns      -      -      -      -      -
  fa51cf3e-5ce0-40fd-bcde-5b298b721d55  50.0G  6.92T  1.01K      0   517M      0  873us      -  873us      -  391ns      -      -      -      -      -
--------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----

Totaly different numbers. ~100-200 IOPS (+ some spikes up to 1,000 IOPS) vs. constant >1,000 IOPS.

I wonder if this is a TrueNAS/zfs/Linux thing or the controller.

In the next days I will re-format one disk after the other via the replace function. A long way to go...

 

[tangles]

I don't know enough about the low-level structure of ZFS nor the correct terms to describe your issue.

Read/Write amplification does comes to mind here for you which looks to be what's going on given that changing the ashift value improves IOPS for you.

I found https://github.com/bradfa/flashbench which has a diff latency column as it tests block sizes. Although designed for SD cards, it might help you identify the optimal block size to use with your devices if you don't want to let ZFS decide when creating your pools.

 

[NickF]

TBH when you are playing around with low level stuff like the sector size you’re gonna see large performance differences.

The file system or operating system has nothing to do with it. It also does not matter if it’s ZFS, EXT4 or NTFS.

Also, a RAIDZ2 is not a good choice from a performance perspective as an FYI. Though not your problem above, may be an issue now that you fixed the first issue. Be better to use 3 drive wide z1 and have two vdevs or use mirrors.

 

[tangles]

more reading md79...

How I Learned to Stop Worrying and Love RAIDZ

The popularity of OpenZFS has spawned a great community of users, sysadmins, architects and developers, contributing a wealth of advice, tips…

www.delphix.com

 

[md79]

changing the ashift value

Ashift was and is 12. So this should not be the problem.

TBH when you are playing around with low level stuff like the sector size you’re gonna see large performance differences.

The disks where part of a storage system and formated with a block size of 520, which leads to problems with TrueNAS. So I had to change this. I thought it would be a good idea to format the disks to 4kn. "Native" is usually better than anything emulated, I think. But...lessons learned...

Also, a RAIDZ2 is not a good choice from a performance perspective

It's a matter of "know your workload/know your demands and expectations". It's just a small homelab. Most of the time one user, sometimes one or two devices more. No iSCSI, just plain smb and nfs. So perfomance-wise there are no high requirements. The NAS is connected via 2x 25 GBit/s (LACP). Why? 80 % because I want to rule out the network as a bottleneck and 20 % because...just for fun.

And yes, I know: One endpoint can only use 25 GBit/s of this LACP bond, not 50 GBit/s. But...I have two cats...and they love cables...

For me, a RAID-Z2 is the best compromise between reliability, storage utilization and speed. Assuming everything works as it should and as I know it should. And these bad read speeds where not part of the plan.

But at the end of the day this new NAS is also something like a project to learn. My journey with ZFS began with OpenSolaris and later OpenIndiana in a VM@ESXi. It was about time for something new and to refresh and expand my knowledge.

 

[sfatula]

I'd do Raidz1 also. 2x3 ssd. Much faster, better storage utilization. Reliabiliy about the same really on SSD. When your rebuild time is measured in minutes instead of days like hard drives, it's not much of a concern. Do weekly scrubs, or even daily with ssd. If you don't need the extra space, mirrors will work too. I'd personally keep a drive on hand for a spare.