ArchatParks
Dabbler
- Joined
- Feb 8, 2022
- Messages
- 28
I setup a new storage pool based on a pair of NVME drives with a pair of Optane drives in a mirrored pool. It does not perform as I had hoped.
System is running :TrueNAS-SCALE-22.12-RC.1
It is a VM to an esxi install with the controllers (and drives for the nvme) passed on to the truenas vm.
Memory: 24gb
Processor: Intel(R) Xeon(R) CPU E5-2450 0 @ 2.10GHz
Allocated 8 CPUs (2 sockets with 4 cores per socket).
[ 0.000000] Command line: BOOT_IMAGE=/ROOT/22.12-RC.1@/boot/vmlinuz-5.15.62+truenas root=ZFS=boot-pool/ROOT/22.12-RC.1 ro console=tty1 console=ttyS0,9600 libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N
[ 0.082500] smpboot: Allowing 8 CPUs, 0 hotplug CPUs
[ 0.087616] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:8 nr_cpu_ids:8 nr_node_ids:1
[ 0.136081] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[ 0.155928] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=8.
[ 0.191673] MDS: Mitigation: Clear CPU buffers
[ 0.244104] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2450 0 @ 2.10GHz (family: 0x6, model: 0x2d, stepping: 0x7)
Pool info:
pool: pool0002
state: ONLINE
scan: scrub repaired 0B in 00:00:21 with 0 errors on Sat Nov 26 11:24:13 2022
config:
NAME STATE READ WRITE CKSUM
pool0002 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c7451373-4550-4d27-8621-01baf622de0b ONLINE 0 0 0
45e2e0a4-df54-4556-ab4f-86ae2f45e609 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
36594b5b-25d1-4389-a4f2-af5157652cd9 ONLINE 0 0 0
85643d9f-9b7f-4492-aa68-c49712466d07 ONLINE 0 0 0
" nvme list -v" results in the following:
NVM Express Subsystems
Subsystem Subsystem-NQN Controllers
---------------- ------------------------------------------------------------------------------------------------ ----------------
nvme-subsys0 nqn.1994-11.com.samsung:nvme:980PRO:M.2:S6S2NS0TA04636Y nvme0
nvme-subsys1 nqn.2014.08.org.nvmexpress:80868086BTPG128200VY512A-2 INTEL HBRPEKNL0202AHO nvme1
nvme-subsys2 nqn.1994-11.com.samsung:nvme:980PRO:M.2:S6S2NS0TA04991H nvme2
nvme-subsys3 nqn.2014.08.org.nvmexpress:80868086BTPG12820206512A-2 INTEL HBRPEKNL0202AHO nvme3
NVM Express Controllers
Device SN MN FR TxPort Address Subsystem Namespaces
-------- -------------------- ---------------------------------------- -------- ------ -------------- ------------ ----------------
nvme0 S6S2NS0TA04636Y Samsung SSD 970 EVO Plus 2TB 4B2QEXM7 pcie 0000:04:00.0 nvme-subsys0 nvme0n1
nvme1 BTPG128200VY512A-2 INTEL HBRPEKNL0202AHO HPS1 pcie 0000:0c:00.0 nvme-subsys1 nvme1n1
nvme2 S6S2NS0TA04991H Samsung SSD 970 EVO Plus 2TB 4B2QEXM7 pcie 0000:14:00.0 nvme-subsys2 nvme2n1
nvme3 BTPG12820206512A-2 INTEL HBRPEKNL0202AHO HPS1 pcie 0000:1c:00.0 nvme-subsys3 nvme3n1
NVM Express Namespaces
Device NSID Usage Format Controllers
------------ -------- -------------------------- ---------------- ----------------
nvme0n1 1 229.49 GB / 2.00 TB 512 B + 0 B nvme0
nvme1n1 1 29.26 GB / 29.26 GB 512 B + 0 B nvme1
nvme2n1 1 111.34 GB / 2.00 TB 512 B + 0 B nvme2
nvme3n1 1 29.26 GB / 29.26 GB 512 B + 0 B nvme3
results in the following:
nvme list-subsys
nvme-subsys0 - NQN=nqn.1994-11.com.samsung:nvme:980PRO:M.2:S6S2NS0TA04636Y
\
+- nvme0 pcie 0000:04:00.0 live
nvme-subsys1 - NQN=nqn.2014.08.org.nvmexpress:80868086BTPG128200VY512A-2 INTEL HBRPEKNL0202AHO
\
+- nvme1 pcie 0000:0c:00.0 live
nvme-subsys2 - NQN=nqn.1994-11.com.samsung:nvme:980PRO:M.2:S6S2NS0TA04991H
\
+- nvme2 pcie 0000:14:00.0 live
nvme-subsys3 - NQN=nqn.2014.08.org.nvmexpress:80868086BTPG12820206512A-2 INTEL HBRPEKNL0202AHO
\
+- nvme3 pcie 0000:1c:00.0 live
If I issue a "lspci -vv -nn -s 0000:04:00.0|grep Lnk", I get the following:
LnkCap: Port #0, Speed 5GT/s, Width x32, ASPM L0s, Exit Latency L0s <64ns
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
LnkSta: Speed 5GT/s (ok), Width x32 (ok)
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
(and the same holds true with the rest of the drives).
Results:
Sync: DISABLED
Compression Level: Inherit (LZ4)
Enable Atime: OFF
ZFS Deduplication: OFF
Case Sensitivity: ON
Running Random write test for IOP/s test on: /mnt/pool0002/normaldata
test: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Run status group 0 (all jobs):
write: IOPS=886, BW=886MiB/s (929MB/s)(4096MiB/4621msec); 0 zone resets
WRITE: bw=886MiB/s (929MB/s), 886MiB/s-886MiB/s (929MB/s-929MB/s), io=4096MiB (4295MB), run=4621-4621msec
-----------------------------------------------------------------------
Running Random Read test for IOP/s test on: /mnt/pool0002/normaldata
test: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
read: IOPS=2196, BW=2196MiB/s (2303MB/s)(4096MiB/1865msec)
Run status group 0 (all jobs):
READ: bw=2196MiB/s (2303MB/s), 2196MiB/s-2196MiB/s (2303MB/s-2303MB/s), io=4096MiB (4295MB), run=1865-1865msec
-----------------------------------------------------------------------
Running Mixed Random Workload test on: /mnt/pool0002/normaldata
test: (g=0): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Run status group 0 (all jobs):
READ: bw=1094MiB/s (1147MB/s), 1094MiB/s-1094MiB/s (1147MB/s-1147MB/s), io=1992MiB (2089MB), run=1821-1821msec
WRITE: bw=1155MiB/s (1212MB/s), 1155MiB/s-1155MiB/s (1212MB/s-1212MB/s), io=2104MiB (2206MB), run=1821-1821msec
-----------------------------------------------------------------------
Running Sequential write test for throughput test on: /mnt/pool0002/normaldata
test: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Run status group 0 (all jobs):
WRITE: bw=769MiB/s (806MB/s), 769MiB/s-769MiB/s (806MB/s-806MB/s), io=4096MiB (4295MB), run=5326-5326msec
-----------------------------------------------------------------------
Running Sequential Read test for throughput test on: /mnt/pool0002/normaldata
test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Run status group 0 (all jobs):
READ: bw=2263MiB/s (2373MB/s), 2263MiB/s-2263MiB/s (2373MB/s-2373MB/s), io=4096MiB (4295MB), run=1810-1810msec
Sync: ALWAYS
Compression Level: Inherit (LZ4)
Enable Atime: OFF
ZFS Deduplication: OFF
Case Sensitivity: ON
Running Random write test for IOP/s test on: /mnt/pool0002/normaldata
test: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
write: IOPS=240, BW=240MiB/s (252MB/s)(4096MiB/17066msec); 0 zone resets
Run status group 0 (all jobs):
WRITE: bw=240MiB/s (252MB/s), 240MiB/s-240MiB/s (252MB/s-252MB/s), io=4096MiB (4295MB), run=17066-17066msec
-----------------------------------------------------------------------
Running Random Read test for IOP/s test on: /mnt/pool0002/normaldata
test: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
read: IOPS=2184, BW=2185MiB/s (2291MB/s)(4096MiB/1875msec)
Run status group 0 (all jobs):
READ: bw=2185MiB/s (2291MB/s), 2185MiB/s-2185MiB/s (2291MB/s-2291MB/s), io=4096MiB (4295MB), run=1875-1875msec
-----------------------------------------------------------------------
Running Mixed Random Workload test on: /mnt/pool0002/normaldata
test: (g=0): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Run status group 0 (all jobs):
READ: bw=231MiB/s (243MB/s), 231MiB/s-231MiB/s (243MB/s-243MB/s), io=1992MiB (2089MB), run=8611-8611msec
WRITE: bw=244MiB/s (256MB/s), 244MiB/s-244MiB/s (256MB/s-256MB/s), io=2104MiB (2206MB), run=8611-8611msec
-----------------------------------------------------------------------
Running Sequential write test for throughput test on: /mnt/pool0002/normaldata
test: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Run status group 0 (all jobs):
WRITE: bw=243MiB/s (254MB/s), 243MiB/s-243MiB/s (254MB/s-254MB/s), io=4096MiB (4295MB), run=16884-16884msec
-----------------------------------------------------------------------
Running Sequential Read test for throughput test on: /mnt/pool0002/normaldata
test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Run status group 0 (all jobs):
READ: bw=2079MiB/s (2180MB/s), 2079MiB/s-2079MiB/s (2180MB/s-2180MB/s), io=4096MiB (4295MB), run=1970-1970msec
-----------------------------------------------------------------------
It would seem it is slower than cold glue and not what I hoped to get from nvme.
If I run the fio test based on the raw NVME, I get this:
[global]
name=nvme-seq-read
time_based
ramp_time=5
runtime=30
readwrite=read
bs=256k
ioengine=libaio
direct=1
numjobs=1
iodepth=32
group_reporting=1
[nvme0]
filename=/dev/nvme0n1
results:
iops : min=13571, max=13685, avg=13657.95, stdev=17.25, samples=60
Run status group 0 (all jobs):
READ: bw=3412MiB/s (3578MB/s), 3412MiB/s-3412MiB/s (3578MB/s-3578MB/s), io=99.0GiB (107GB), run=30003-30003msec
Disk stats (read/write):
nvme0n1: ios=477602/378, merge=0/0, ticks=1103483/11, in_queue=1103526, util=99.79%
---------------------------------------------------------------------------------------------------
[global]
name=nvme-seq-write
time_based
ramp_time=5
runtime=30
readwrite=write
bs=256k
ioengine=libaio
direct=1
numjobs=1
iodepth=32
group_reporting=1
[nvme0]
filename=/dev/nvme0n1
Any ideas as to how to speed up the process?
results:
iops : min=12854, max=13210, avg=13130.10, stdev=67.78, samples=60
Run status group 0 (all jobs):
WRITE: bw=3280MiB/s (3440MB/s), 3280MiB/s-3280MiB/s (3440MB/s-3440MB/s), io=96.1GiB (103GB), run=30003-30003msec
Disk stats (read/write):
nvme0n1: ios=74/459340, merge=0/0, ticks=40/1103274, in_queue=1103348, util=99.84%
System is running :TrueNAS-SCALE-22.12-RC.1
It is a VM to an esxi install with the controllers (and drives for the nvme) passed on to the truenas vm.
Memory: 24gb
Processor: Intel(R) Xeon(R) CPU E5-2450 0 @ 2.10GHz
Allocated 8 CPUs (2 sockets with 4 cores per socket).
[ 0.000000] Command line: BOOT_IMAGE=/ROOT/22.12-RC.1@/boot/vmlinuz-5.15.62+truenas root=ZFS=boot-pool/ROOT/22.12-RC.1 ro console=tty1 console=ttyS0,9600 libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N
[ 0.082500] smpboot: Allowing 8 CPUs, 0 hotplug CPUs
[ 0.087616] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:8 nr_cpu_ids:8 nr_node_ids:1
[ 0.136081] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[ 0.155928] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=8.
[ 0.191673] MDS: Mitigation: Clear CPU buffers
[ 0.244104] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2450 0 @ 2.10GHz (family: 0x6, model: 0x2d, stepping: 0x7)
Pool info:
pool: pool0002
state: ONLINE
scan: scrub repaired 0B in 00:00:21 with 0 errors on Sat Nov 26 11:24:13 2022
config:
NAME STATE READ WRITE CKSUM
pool0002 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c7451373-4550-4d27-8621-01baf622de0b ONLINE 0 0 0
45e2e0a4-df54-4556-ab4f-86ae2f45e609 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
36594b5b-25d1-4389-a4f2-af5157652cd9 ONLINE 0 0 0
85643d9f-9b7f-4492-aa68-c49712466d07 ONLINE 0 0 0
" nvme list -v" results in the following:
NVM Express Subsystems
Subsystem Subsystem-NQN Controllers
---------------- ------------------------------------------------------------------------------------------------ ----------------
nvme-subsys0 nqn.1994-11.com.samsung:nvme:980PRO:M.2:S6S2NS0TA04636Y nvme0
nvme-subsys1 nqn.2014.08.org.nvmexpress:80868086BTPG128200VY512A-2 INTEL HBRPEKNL0202AHO nvme1
nvme-subsys2 nqn.1994-11.com.samsung:nvme:980PRO:M.2:S6S2NS0TA04991H nvme2
nvme-subsys3 nqn.2014.08.org.nvmexpress:80868086BTPG12820206512A-2 INTEL HBRPEKNL0202AHO nvme3
NVM Express Controllers
Device SN MN FR TxPort Address Subsystem Namespaces
-------- -------------------- ---------------------------------------- -------- ------ -------------- ------------ ----------------
nvme0 S6S2NS0TA04636Y Samsung SSD 970 EVO Plus 2TB 4B2QEXM7 pcie 0000:04:00.0 nvme-subsys0 nvme0n1
nvme1 BTPG128200VY512A-2 INTEL HBRPEKNL0202AHO HPS1 pcie 0000:0c:00.0 nvme-subsys1 nvme1n1
nvme2 S6S2NS0TA04991H Samsung SSD 970 EVO Plus 2TB 4B2QEXM7 pcie 0000:14:00.0 nvme-subsys2 nvme2n1
nvme3 BTPG12820206512A-2 INTEL HBRPEKNL0202AHO HPS1 pcie 0000:1c:00.0 nvme-subsys3 nvme3n1
NVM Express Namespaces
Device NSID Usage Format Controllers
------------ -------- -------------------------- ---------------- ----------------
nvme0n1 1 229.49 GB / 2.00 TB 512 B + 0 B nvme0
nvme1n1 1 29.26 GB / 29.26 GB 512 B + 0 B nvme1
nvme2n1 1 111.34 GB / 2.00 TB 512 B + 0 B nvme2
nvme3n1 1 29.26 GB / 29.26 GB 512 B + 0 B nvme3
results in the following:
nvme list-subsys
nvme-subsys0 - NQN=nqn.1994-11.com.samsung:nvme:980PRO:M.2:S6S2NS0TA04636Y
\
+- nvme0 pcie 0000:04:00.0 live
nvme-subsys1 - NQN=nqn.2014.08.org.nvmexpress:80868086BTPG128200VY512A-2 INTEL HBRPEKNL0202AHO
\
+- nvme1 pcie 0000:0c:00.0 live
nvme-subsys2 - NQN=nqn.1994-11.com.samsung:nvme:980PRO:M.2:S6S2NS0TA04991H
\
+- nvme2 pcie 0000:14:00.0 live
nvme-subsys3 - NQN=nqn.2014.08.org.nvmexpress:80868086BTPG12820206512A-2 INTEL HBRPEKNL0202AHO
\
+- nvme3 pcie 0000:1c:00.0 live
If I issue a "lspci -vv -nn -s 0000:04:00.0|grep Lnk", I get the following:
LnkCap: Port #0, Speed 5GT/s, Width x32, ASPM L0s, Exit Latency L0s <64ns
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
LnkSta: Speed 5GT/s (ok), Width x32 (ok)
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
(and the same holds true with the rest of the drives).
Results:
Sync: DISABLED
Compression Level: Inherit (LZ4)
Enable Atime: OFF
ZFS Deduplication: OFF
Case Sensitivity: ON
Running Random write test for IOP/s test on: /mnt/pool0002/normaldata
test: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Run status group 0 (all jobs):
write: IOPS=886, BW=886MiB/s (929MB/s)(4096MiB/4621msec); 0 zone resets
WRITE: bw=886MiB/s (929MB/s), 886MiB/s-886MiB/s (929MB/s-929MB/s), io=4096MiB (4295MB), run=4621-4621msec
-----------------------------------------------------------------------
Running Random Read test for IOP/s test on: /mnt/pool0002/normaldata
test: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
read: IOPS=2196, BW=2196MiB/s (2303MB/s)(4096MiB/1865msec)
Run status group 0 (all jobs):
READ: bw=2196MiB/s (2303MB/s), 2196MiB/s-2196MiB/s (2303MB/s-2303MB/s), io=4096MiB (4295MB), run=1865-1865msec
-----------------------------------------------------------------------
Running Mixed Random Workload test on: /mnt/pool0002/normaldata
test: (g=0): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Run status group 0 (all jobs):
READ: bw=1094MiB/s (1147MB/s), 1094MiB/s-1094MiB/s (1147MB/s-1147MB/s), io=1992MiB (2089MB), run=1821-1821msec
WRITE: bw=1155MiB/s (1212MB/s), 1155MiB/s-1155MiB/s (1212MB/s-1212MB/s), io=2104MiB (2206MB), run=1821-1821msec
-----------------------------------------------------------------------
Running Sequential write test for throughput test on: /mnt/pool0002/normaldata
test: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Run status group 0 (all jobs):
WRITE: bw=769MiB/s (806MB/s), 769MiB/s-769MiB/s (806MB/s-806MB/s), io=4096MiB (4295MB), run=5326-5326msec
-----------------------------------------------------------------------
Running Sequential Read test for throughput test on: /mnt/pool0002/normaldata
test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Run status group 0 (all jobs):
READ: bw=2263MiB/s (2373MB/s), 2263MiB/s-2263MiB/s (2373MB/s-2373MB/s), io=4096MiB (4295MB), run=1810-1810msec
Sync: ALWAYS
Compression Level: Inherit (LZ4)
Enable Atime: OFF
ZFS Deduplication: OFF
Case Sensitivity: ON
Running Random write test for IOP/s test on: /mnt/pool0002/normaldata
test: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
write: IOPS=240, BW=240MiB/s (252MB/s)(4096MiB/17066msec); 0 zone resets
Run status group 0 (all jobs):
WRITE: bw=240MiB/s (252MB/s), 240MiB/s-240MiB/s (252MB/s-252MB/s), io=4096MiB (4295MB), run=17066-17066msec
-----------------------------------------------------------------------
Running Random Read test for IOP/s test on: /mnt/pool0002/normaldata
test: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
read: IOPS=2184, BW=2185MiB/s (2291MB/s)(4096MiB/1875msec)
Run status group 0 (all jobs):
READ: bw=2185MiB/s (2291MB/s), 2185MiB/s-2185MiB/s (2291MB/s-2291MB/s), io=4096MiB (4295MB), run=1875-1875msec
-----------------------------------------------------------------------
Running Mixed Random Workload test on: /mnt/pool0002/normaldata
test: (g=0): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Run status group 0 (all jobs):
READ: bw=231MiB/s (243MB/s), 231MiB/s-231MiB/s (243MB/s-243MB/s), io=1992MiB (2089MB), run=8611-8611msec
WRITE: bw=244MiB/s (256MB/s), 244MiB/s-244MiB/s (256MB/s-256MB/s), io=2104MiB (2206MB), run=8611-8611msec
-----------------------------------------------------------------------
Running Sequential write test for throughput test on: /mnt/pool0002/normaldata
test: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Run status group 0 (all jobs):
WRITE: bw=243MiB/s (254MB/s), 243MiB/s-243MiB/s (254MB/s-254MB/s), io=4096MiB (4295MB), run=16884-16884msec
-----------------------------------------------------------------------
Running Sequential Read test for throughput test on: /mnt/pool0002/normaldata
test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Run status group 0 (all jobs):
READ: bw=2079MiB/s (2180MB/s), 2079MiB/s-2079MiB/s (2180MB/s-2180MB/s), io=4096MiB (4295MB), run=1970-1970msec
-----------------------------------------------------------------------
It would seem it is slower than cold glue and not what I hoped to get from nvme.
If I run the fio test based on the raw NVME, I get this:
[global]
name=nvme-seq-read
time_based
ramp_time=5
runtime=30
readwrite=read
bs=256k
ioengine=libaio
direct=1
numjobs=1
iodepth=32
group_reporting=1
[nvme0]
filename=/dev/nvme0n1
results:
iops : min=13571, max=13685, avg=13657.95, stdev=17.25, samples=60
Run status group 0 (all jobs):
READ: bw=3412MiB/s (3578MB/s), 3412MiB/s-3412MiB/s (3578MB/s-3578MB/s), io=99.0GiB (107GB), run=30003-30003msec
Disk stats (read/write):
nvme0n1: ios=477602/378, merge=0/0, ticks=1103483/11, in_queue=1103526, util=99.79%
---------------------------------------------------------------------------------------------------
[global]
name=nvme-seq-write
time_based
ramp_time=5
runtime=30
readwrite=write
bs=256k
ioengine=libaio
direct=1
numjobs=1
iodepth=32
group_reporting=1
[nvme0]
filename=/dev/nvme0n1
Any ideas as to how to speed up the process?
results:
iops : min=12854, max=13210, avg=13130.10, stdev=67.78, samples=60
Run status group 0 (all jobs):
WRITE: bw=3280MiB/s (3440MB/s), 3280MiB/s-3280MiB/s (3440MB/s-3440MB/s), io=96.1GiB (103GB), run=30003-30003msec
Disk stats (read/write):
nvme0n1: ios=74/459340, merge=0/0, ticks=40/1103274, in_queue=1103348, util=99.84%