kamalov
Dabbler
- Joined
- Dec 14, 2022
- Messages
- 43
Hello,
I have a new TrueNAS Scale system I just set up.
It has the following hardware:
Motherboard: MSI X79-GD49 PLUS
CPU: I7-4930K
RAM: 24GB
SSD: 256GB
HDD's: 3 x HGST Ultrastar HUH728080ALE601 8TB
The system was connected to an ethernet switch that was connected to a TP-Link Archer AX11000 router. My PC was connected to the same ethernet switch.
When I tried transferring files I was only getting about 10.4 MBS.
Then I read thru some thread on the forum and decided to connect the system directly to the router. The ethernet cable is plugged in directly in to the motherboard. I am not using a nic. Now I am getting about 36 MBS.
From what I understand after reading the forum threads, it could be the cable, motherboard, the PC i am using, router, etc. I just don't even know where to begin and was hoping someone could assist me in the right direction.
I ran the code below and got the following message.s
I have a new TrueNAS Scale system I just set up.
It has the following hardware:
Motherboard: MSI X79-GD49 PLUS
CPU: I7-4930K
RAM: 24GB
SSD: 256GB
HDD's: 3 x HGST Ultrastar HUH728080ALE601 8TB
The system was connected to an ethernet switch that was connected to a TP-Link Archer AX11000 router. My PC was connected to the same ethernet switch.
When I tried transferring files I was only getting about 10.4 MBS.
Then I read thru some thread on the forum and decided to connect the system directly to the router. The ethernet cable is plugged in directly in to the motherboard. I am not using a nic. Now I am getting about 36 MBS.
From what I understand after reading the forum threads, it could be the cable, motherboard, the PC i am using, router, etc. I just don't even know where to begin and was hoping someone could assist me in the right direction.
I ran the code below and got the following message.s
Code:
admin@truenas[~]# sudo fio --name=seq-read --rw=read --ioengine=posixaio \ --blocksize=128k --iodepth=64 --runtime=300 --size=4g \ --direct=1 --invalidate=1 --time_based --numjobs="$(nproc)" [sudo] password for admin: seq-read: (g=0): rw=read, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=posixaio, iodepth=64 ... fio-3.25 Starting 16 processes Jobs: 16 (f=16): [R(16)][100.0%][r=550MiB/s][r=4403 IOPS][eta 00m:00s] seq-read: (groupid=0, jobs=1): err= 0: pid=15159: Sun Aug 7 09:33:49 2022 read: IOPS=254, BW=31.9MiB/s (33.4MB/s)(9560MiB/300139msec) slat (nsec): min=559, max=171251, avg=1878.01, stdev=1610.78 clat (msec): min=23, max=6039, avg=250.98, stdev=272.62 lat (msec): min=23, max=6039, avg=250.99, stdev=272.62 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 167], 10.00th=[ 171], 20.00th=[ 178], | 30.00th=[ 182], 40.00th=[ 188], 50.00th=[ 192], 60.00th=[ 199], | 70.00th=[ 209], 80.00th=[ 236], 90.00th=[ 418], 95.00th=[ 558], | 99.00th=[ 743], 99.50th=[ 1670], 99.90th=[ 5336], 99.95th=[ 6007], | 99.99th=[ 6007] bw ( KiB/s): min= 512, max=49763, per=6.59%, avg=35344.95, stdev=12518.87, samples=554 iops : min= 4, max= 388, avg=276.08, stdev=97.77, samples=554 lat (msec) : 50=0.01%, 250=82.02%, 500=11.32%, 750=5.81%, 1000=0.10% lat (msec) : 2000=0.50%, >=2000=0.25% cpu : usr=0.10%, sys=0.13%, ctx=9272, majf=3, minf=50 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=11.0%, 16=25.0%, 32=51.5%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=98.6%, 8=0.0%, 16=0.1%, 32=0.0%, 64=1.4%, >=64=0.0% issued rwts: total=76482,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 seq-read: (groupid=0, jobs=1): err= 0: pid=15160: Sun Aug 7 09:33:49 2022 read: IOPS=254, BW=31.8MiB/s (33.3MB/s)(9528MiB/300019msec) slat (nsec): min=558, max=170203, avg=1853.23, stdev=1550.34 clat (msec): min=152, max=3390, avg=251.73, stdev=221.26 lat (msec): min=152, max=3390, avg=251.73, stdev=221.26 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 167], 10.00th=[ 169], 20.00th=[ 176], | 30.00th=[ 184], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 249], 80.00th=[ 275], 90.00th=[ 305], 95.00th=[ 477], | 99.00th=[ 1720], 99.50th=[ 2022], 99.90th=[ 2601], 99.95th=[ 3373], | 99.99th=[ 3373] bw ( KiB/s): min= 1792, max=49250, per=6.66%, avg=35720.92, stdev=10551.39, samples=545 iops : min= 14, max= 384, avg=279.02, stdev=82.41, samples=545 lat (msec) : 250=70.24%, 500=25.45%, 750=2.72%, 1000=0.50%, 2000=0.59% lat (msec) : >=2000=0.50% cpu : usr=0.09%, sys=0.13%, ctx=9270, majf=4, minf=50 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=12.5%, 16=25.0%, 32=50.0%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0% issued rwts: total=76224,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 seq-read: (groupid=0, jobs=1): err= 0: pid=15161: Sun Aug 7 09:33:49 2022 read: IOPS=269, BW=33.7MiB/s (35.4MB/s)(9.88GiB/300107msec) slat (nsec): min=558, max=222095, avg=1873.28, stdev=1659.20 clat (msec): min=13, max=2976, avg=237.06, stdev=114.67 lat (msec): min=13, max=2976, avg=237.06, stdev=114.67 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 169], 10.00th=[ 176], 20.00th=[ 186], | 30.00th=[ 199], 40.00th=[ 209], 50.00th=[ 224], 60.00th=[ 236], | 70.00th=[ 253], 80.00th=[ 275], 90.00th=[ 300], 95.00th=[ 313], | 99.00th=[ 430], 99.50th=[ 523], 99.90th=[ 1989], 99.95th=[ 2970], | 99.99th=[ 2970] bw ( KiB/s): min= 512, max=49250, per=6.55%, avg=35138.20, stdev=9537.74, samples=589 iops : min= 4, max= 384, avg=274.46, stdev=74.49, samples=589 lat (msec) : 20=0.01%, 50=0.01%, 250=68.26%, 500=31.11%, 750=0.40% lat (msec) : 2000=0.16%, >=2000=0.08% cpu : usr=0.13%, sys=0.12%, ctx=9848, majf=5, minf=51 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=12.0%, 16=25.0%, 32=50.5%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0% issued rwts: total=80961,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 seq-read: (groupid=0, jobs=1): err= 0: pid=15162: Sun Aug 7 09:33:49 2022 read: IOPS=263, BW=32.9MiB/s (34.5MB/s)(9872MiB/300134msec) slat (nsec): min=559, max=176978, avg=1884.72, stdev=1571.51 clat (msec): min=26, max=3356, avg=243.04, stdev=160.88 lat (msec): min=26, max=3356, avg=243.04, stdev=160.88 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 169], 10.00th=[ 176], 20.00th=[ 184], | 30.00th=[ 194], 40.00th=[ 205], 50.00th=[ 222], 60.00th=[ 239], | 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 642], 99.50th=[ 1770], 99.90th=[ 2299], 99.95th=[ 3306], | 99.99th=[ 3306] bw ( KiB/s): min= 512, max=49250, per=6.52%, avg=34990.20, stdev=9787.95, samples=577 iops : min= 4, max= 384, avg=273.31, stdev=76.45, samples=577 lat (msec) : 50=0.01%, 250=67.37%, 500=30.68%, 750=1.30%, 1000=0.08% lat (msec) : 2000=0.33%, >=2000=0.24% cpu : usr=0.11%, sys=0.13%, ctx=9599, majf=4, minf=51 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=11.5%, 16=25.0%, 32=51.1%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0% issued rwts: total=78977,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 seq-read: (groupid=0, jobs=1): err= 0: pid=15163: Sun Aug 7 09:33:49 2022 read: IOPS=263, BW=32.0MiB/s (34.6MB/s)(9896MiB/300050msec) slat (nsec): min=558, max=238299, avg=1874.96, stdev=1712.14 clat (msec): min=25, max=3658, avg=242.38, stdev=163.77 lat (msec): min=25, max=3658, avg=242.38, stdev=163.77 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 169], 10.00th=[ 174], 20.00th=[ 182], | 30.00th=[ 188], 40.00th=[ 201], 50.00th=[ 213], 60.00th=[ 232], | 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 292], 95.00th=[ 330], | 99.00th=[ 676], 99.50th=[ 776], 99.90th=[ 1989], 99.95th=[ 3641], | 99.99th=[ 3641] bw ( KiB/s): min= 512, max=49408, per=6.56%, avg=35169.24, stdev=10163.72, samples=575 iops : min= 4, max= 386, avg=274.71, stdev=79.37, samples=575 lat (msec) : 50=0.01%, 250=67.28%, 500=30.12%, 750=1.94%, 1000=0.16% lat (msec) : 2000=0.41%, >=2000=0.08% cpu : usr=0.11%, sys=0.12%, ctx=9630, majf=7, minf=51 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=11.0%, 16=25.0%, 32=51.5%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=98.6%, 8=0.0%, 16=0.1%, 32=0.0%, 64=1.4%, >=64=0.0% issued rwts: total=79170,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 seq-read: (groupid=0, jobs=1): err= 0: pid=15164: Sun Aug 7 09:33:49 2022 read: IOPS=256, BW=32.0MiB/s (33.6MB/s)(9608MiB/300152msec) slat (nsec): min=628, max=289701, avg=1887.67, stdev=1812.02 clat (msec): min=156, max=4436, avg=249.74, stdev=203.27 lat (msec): min=156, max=4436, avg=249.74, stdev=203.27 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 167], 10.00th=[ 171], 20.00th=[ 178], | 30.00th=[ 184], 40.00th=[ 190], 50.00th=[ 199], 60.00th=[ 209], | 70.00th=[ 241], 80.00th=[ 288], 90.00th=[ 309], 95.00th=[ 460], | 99.00th=[ 1116], 99.50th=[ 1888], 99.90th=[ 1989], 99.95th=[ 3708], | 99.99th=[ 3708] bw ( KiB/s): min= 512, max=49250, per=6.55%, avg=35116.37, stdev=11339.85, samples=560 iops : min= 4, max= 384, avg=274.29, stdev=88.56, samples=560 lat (msec) : 250=71.98%, 500=23.94%, 750=2.83%, 1000=0.17%, 2000=1.00% lat (msec) : >=2000=0.08% cpu : usr=0.12%, sys=0.11%, ctx=9359, majf=8, minf=45 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=12.5%, 16=25.0%, 32=50.0%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0% issued rwts: total=76864,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 seq-read: (groupid=0, jobs=1): err= 0: pid=15165: Sun Aug 7 09:33:49 2022 read: IOPS=259, BW=32.4MiB/s (33.0MB/s)(9720MiB/300159msec) slat (nsec): min=628, max=172298, avg=1846.76, stdev=1638.97 clat (msec): min=22, max=4427, avg=246.88, stdev=209.22 lat (msec): min=22, max=4427, avg=246.88, stdev=209.22 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 167], 10.00th=[ 171], 20.00th=[ 178], | 30.00th=[ 184], 40.00th=[ 190], 50.00th=[ 197], 60.00th=[ 203], | 70.00th=[ 215], 80.00th=[ 241], 90.00th=[ 384], 95.00th=[ 531], | 99.00th=[ 667], 99.50th=[ 1552], 99.90th=[ 3004], 99.95th=[ 4396], | 99.99th=[ 4396] bw ( KiB/s): min= 256, max=49408, per=6.52%, avg=34964.02, stdev=12130.99, samples=569 iops : min= 2, max= 386, avg=273.10, stdev=94.76, samples=569 lat (msec) : 50=0.01%, 250=81.06%, 500=12.68%, 750=5.51%, 1000=0.01% lat (msec) : 2000=0.49%, >=2000=0.25% cpu : usr=0.11%, sys=0.12%, ctx=9449, majf=3, minf=50 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=11.7%, 16=25.0%, 32=50.8%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=98.6%, 8=0.0%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0% issued rwts: total=77759,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 seq-read: (groupid=0, jobs=1): err= 0: pid=15166: Sun Aug 7 09:33:49 2022 read: IOPS=266, BW=33.3MiB/s (34.9MB/s)(9984MiB/300119msec) slat (nsec): min=559, max=180051, avg=1859.75, stdev=1588.79 clat (msec): min=22, max=3138, avg=240.30, stdev=135.00 lat (msec): min=22, max=3138, avg=240.30, stdev=135.00 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 171], 10.00th=[ 178], 20.00th=[ 186], | 30.00th=[ 197], 40.00th=[ 211], 50.00th=[ 226], 60.00th=[ 241], | 70.00th=[ 257], 80.00th=[ 271], 90.00th=[ 292], 95.00th=[ 309], | 99.00th=[ 600], 99.50th=[ 667], 99.90th=[ 2056], 99.95th=[ 3138], | 99.99th=[ 3138] bw ( KiB/s): min= 1024, max=49250, per=6.54%, avg=35050.26, stdev=9098.72, samples=583 iops : min= 8, max= 384, avg=273.77, stdev=71.06, samples=583 lat (msec) : 50=0.01%, 250=66.02%, 500=32.68%, 750=0.81%, 1000=0.08% lat (msec) : 2000=0.24%, >=2000=0.16% cpu : usr=0.11%, sys=0.13%, ctx=9695, majf=4, minf=55 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=12.5%, 16=25.0%, 32=50.0%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0% issued rwts: total=79872,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 seq-read: (groupid=0, jobs=1): err= 0: pid=15167: Sun Aug 7 09:33:49 2022 read: IOPS=262, BW=32.8MiB/s (34.4MB/s)(9856MiB/300128msec) slat (nsec): min=558, max=238158, avg=1883.56, stdev=1763.24 clat (msec): min=23, max=3768, avg=243.43, stdev=172.55 lat (msec): min=23, max=3768, avg=243.43, stdev=172.55 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 169], 10.00th=[ 174], 20.00th=[ 186], | 30.00th=[ 192], 40.00th=[ 201], 50.00th=[ 211], 60.00th=[ 224], | 70.00th=[ 239], 80.00th=[ 264], 90.00th=[ 309], 95.00th=[ 393], | 99.00th=[ 701], 99.50th=[ 995], 99.90th=[ 2400], 99.95th=[ 3775], | 99.99th=[ 3775] bw ( KiB/s): min= 512, max=49250, per=6.60%, avg=35395.31, stdev=9645.17, samples=569 iops : min= 4, max= 384, avg=276.47, stdev=75.32, samples=569 lat (msec) : 50=0.01%, 250=74.99%, 500=23.31%, 750=0.97%, 1000=0.24% lat (msec) : 2000=0.24%, >=2000=0.24% cpu : usr=0.11%, sys=0.12%, ctx=9559, majf=2, minf=51 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=12.5%, 16=25.0%, 32=50.0%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0% issued rwts: total=78848,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 seq-read: (groupid=0, jobs=1): err= 0: pid=15168: Sun Aug 7 09:33:49 2022 read: IOPS=259, BW=32.4MiB/s (33.0MB/s)(9720MiB/300109msec) slat (nsec): min=559, max=195206, avg=1873.82, stdev=1560.58 clat (msec): min=31, max=2531, avg=246.82, stdev=178.39 lat (msec): min=31, max=2531, avg=246.82, stdev=178.39 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 167], 10.00th=[ 171], 20.00th=[ 180], | 30.00th=[ 188], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 226], | 70.00th=[ 245], 80.00th=[ 279], 90.00th=[ 309], 95.00th=[ 384], | 99.00th=[ 693], 99.50th=[ 1888], 99.90th=[ 2333], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 1280, max=49250, per=6.58%, avg=35277.04, stdev=10264.06, samples=563 iops : min= 10, max= 384, avg=275.55, stdev=80.18, samples=563 lat (msec) : 50=0.01%, 250=72.28%, 500=24.60%, 750=2.22%, 1000=0.08% lat (msec) : 2000=0.58%, >=2000=0.25% cpu : usr=0.10%, sys=0.13%, ctx=9442, majf=4, minf=50 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=12.5%, 16=25.0%, 32=50.0%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0% issued rwts: total=77760,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 seq-read: (groupid=0, jobs=1): err= 0: pid=15169: Sun Aug 7 09:33:49 2022 read: IOPS=256, BW=32.0MiB/s (33.6MB/s)(9616MiB/300145msec) slat (nsec): min=558, max=329651, avg=1841.20, stdev=1918.44 clat (msec): min=153, max=5285, avg=249.53, stdev=230.30 lat (msec): min=153, max=5285, avg=249.53, stdev=230.30 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 167], 10.00th=[ 171], 20.00th=[ 178], | 30.00th=[ 182], 40.00th=[ 188], 50.00th=[ 197], 60.00th=[ 203], | 70.00th=[ 213], 80.00th=[ 234], 90.00th=[ 380], 95.00th=[ 584], | 99.00th=[ 768], 99.50th=[ 1804], 99.90th=[ 2299], 99.95th=[ 5269], | 99.99th=[ 5269] bw ( KiB/s): min= 256, max=49250, per=6.59%, avg=35360.29, stdev=12158.39, samples=557 iops : min= 2, max= 384, avg=276.20, stdev=94.97, samples=557 lat (msec) : 250=83.99%, 500=8.70%, 750=5.98%, 1000=0.50%, 2000=0.50% lat (msec) : >=2000=0.33% cpu : usr=0.10%, sys=0.12%, ctx=9369, majf=4, minf=49 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=12.5%, 16=25.0%, 32=50.0%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0% issued rwts: total=76928,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 seq-read: (groupid=0, jobs=1): err= 0: pid=15170: Sun Aug 7 09:33:49 2022 read: IOPS=259, BW=32.4MiB/s (34.0MB/s)(9736MiB/300156msec) slat (nsec): min=558, max=288165, avg=1897.43, stdev=1927.94 clat (msec): min=20, max=3059, avg=246.46, stdev=182.39 lat (msec): min=20, max=3059, avg=246.46, stdev=182.39 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 169], 10.00th=[ 174], 20.00th=[ 180], | 30.00th=[ 188], 40.00th=[ 199], 50.00th=[ 209], 60.00th=[ 232], | 70.00th=[ 257], 80.00th=[ 275], 90.00th=[ 292], 95.00th=[ 326], | 99.00th=[ 760], 99.50th=[ 1888], 99.90th=[ 2265], 99.95th=[ 3037], | 99.99th=[ 3037] bw ( KiB/s): min= 256, max=49408, per=6.51%, avg=34898.95, stdev=10384.84, samples=571 iops : min= 2, max= 386, avg=272.59, stdev=81.09, samples=571 lat (msec) : 50=0.01%, 250=66.90%, 500=30.04%, 750=1.99%, 1000=0.25% lat (msec) : 2000=0.41%, >=2000=0.41% cpu : usr=0.11%, sys=0.12%, ctx=9481, majf=6, minf=52 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=11.1%, 16=25.0%, 32=51.4%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0% issued rwts: total=77889,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 seq-read: (groupid=0, jobs=1): err= 0: pid=15171: Sun Aug 7 09:33:49 2022 read: IOPS=266, BW=33.3MiB/s (34.9MB/s)(9.77GiB/300099msec) slat (nsec): min=558, max=167270, avg=1889.08, stdev=1642.16 clat (msec): min=20, max=2945, avg=239.90, stdev=139.93 lat (msec): min=20, max=2945, avg=239.90, stdev=139.93 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 169], 10.00th=[ 174], 20.00th=[ 180], | 30.00th=[ 188], 40.00th=[ 197], 50.00th=[ 211], 60.00th=[ 228], | 70.00th=[ 253], 80.00th=[ 275], 90.00th=[ 300], 95.00th=[ 317], | 99.00th=[ 625], 99.50th=[ 701], 99.90th=[ 1921], 99.95th=[ 2937], | 99.99th=[ 2937] bw ( KiB/s): min= 512, max=49250, per=6.54%, avg=35080.34, stdev=10704.25, samples=583 iops : min= 4, max= 384, avg=274.01, stdev=83.59, samples=583 lat (msec) : 50=0.01%, 250=68.88%, 500=28.21%, 750=2.51%, 2000=0.32% lat (msec) : >=2000=0.08% cpu : usr=0.11%, sys=0.13%, ctx=9734, majf=3, minf=50 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=11.4%, 16=25.0%, 32=51.1%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0% issued rwts: total=80001,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 seq-read: (groupid=0, jobs=1): err= 0: pid=15172: Sun Aug 7 09:33:49 2022 read: IOPS=270, BW=33.9MiB/s (35.5MB/s)(9.92GiB/300080msec) slat (nsec): min=559, max=187244, avg=1883.96, stdev=1572.28 clat (msec): min=22, max=3868, avg=236.10, stdev=122.49 lat (msec): min=22, max=3868, avg=236.10, stdev=122.49 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 174], 10.00th=[ 180], 20.00th=[ 190], | 30.00th=[ 203], 40.00th=[ 213], 50.00th=[ 228], 60.00th=[ 243], | 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 288], 95.00th=[ 296], | 99.00th=[ 338], 99.50th=[ 523], 99.90th=[ 2039], 99.95th=[ 3842], | 99.99th=[ 3842] bw ( KiB/s): min= 512, max=49408, per=6.55%, avg=35131.79, stdev=8883.00, samples=591 iops : min= 4, max= 386, avg=274.41, stdev=69.38, samples=591 lat (msec) : 50=0.01%, 250=66.28%, 500=33.00%, 750=0.55%, 2000=0.01% lat (msec) : >=2000=0.16% cpu : usr=0.11%, sys=0.13%, ctx=9870, majf=5, minf=50 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=11.6%, 16=25.0%, 32=51.0%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0% issued rwts: total=81281,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 seq-read: (groupid=0, jobs=1): err= 0: pid=15173: Sun Aug 7 09:33:49 2022 read: IOPS=266, BW=33.3MiB/s (34.0MB/s)(9.77GiB/300101msec) slat (nsec): min=628, max=164406, avg=1863.71, stdev=1483.51 clat (msec): min=17, max=3732, avg=239.71, stdev=149.39 lat (msec): min=17, max=3732, avg=239.71, stdev=149.39 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 169], 10.00th=[ 174], 20.00th=[ 180], | 30.00th=[ 184], 40.00th=[ 190], 50.00th=[ 199], 60.00th=[ 209], | 70.00th=[ 224], 80.00th=[ 245], 90.00th=[ 338], 95.00th=[ 527], | 99.00th=[ 676], 99.50th=[ 693], 99.90th=[ 1938], 99.95th=[ 3138], | 99.99th=[ 3138] bw ( KiB/s): min= 256, max=49408, per=6.57%, avg=35228.43, stdev=11976.08, samples=581 iops : min= 2, max= 386, avg=275.17, stdev=93.55, samples=581 lat (msec) : 20=0.01%, 50=0.01%, 250=81.11%, 500=13.29%, 750=5.20% lat (msec) : 1000=0.15%, 2000=0.16%, >=2000=0.08% cpu : usr=0.12%, sys=0.12%, ctx=9751, majf=5, minf=49 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=11.6%, 16=25.0%, 32=50.9%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=98.6%, 8=0.1%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0% issued rwts: total=80065,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 seq-read: (groupid=0, jobs=1): err= 0: pid=15174: Sun Aug 7 09:33:49 2022 read: IOPS=261, BW=32.7MiB/s (34.3MB/s)(9816MiB/300150msec) slat (nsec): min=628, max=192692, avg=1889.95, stdev=1858.62 clat (msec): min=152, max=2758, avg=244.45, stdev=164.53 lat (msec): min=152, max=2758, avg=244.45, stdev=164.53 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 169], 10.00th=[ 171], 20.00th=[ 180], | 30.00th=[ 186], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 213], | 70.00th=[ 228], 80.00th=[ 268], 90.00th=[ 368], 95.00th=[ 426], | 99.00th=[ 718], 99.50th=[ 1586], 99.90th=[ 2433], 99.95th=[ 2735], | 99.99th=[ 2735] bw ( KiB/s): min= 1280, max=49250, per=6.54%, avg=35063.18, stdev=10766.05, samples=573 iops : min= 10, max= 384, avg=273.87, stdev=84.07, samples=573 lat (msec) : 250=77.11%, 500=20.63%, 750=1.36%, 1000=0.24%, 2000=0.50% lat (msec) : >=2000=0.16% cpu : usr=0.11%, sys=0.12%, ctx=9539, majf=3, minf=52 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=12.5%, 16=25.0%, 32=50.0%, >=64=12.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=98.6%, 8=0.0%, 16=0.0%, 32=0.0%, 64=1.4%, >=64=0.0% issued rwts: total=78527,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=524MiB/s (549MB/s), 31.8MiB/s-33.9MiB/s (33.3MB/s-35.5MB/s), io=154GiB (165GB), run=300019-300159msec