alexhore
Explorer
- Joined
- Sep 24, 2014
- Messages
- 52
Build a new TrueNas up:
Connected the NVME drives to the second and third PCIE slots, one is gen2 then mirrored for System pool
Suck another SSD in there for boot
Noticed I can saturate Lan downloaded from RAIDZ1 pool but uploading stuck at 30MB/s. Tired another laptop and a PC, both hard wired no dice.
I asumed that writing across 10 drives one way or another would be fast. Quick google and come to the realisation that RAIDZ1 is apparently not fast?
As a test transfered a large file, way bigger than my pool of ram from RAIDZ1 to the mirrored NVMEs and then transfered it back.
SAS -> NVME SAS Read 100MB/s (10x10MB/s) NVME Write 100MB/s (x1 as mirrored)
NVME -> SAS NVME Read 200MB/s (2x100MB/s) SAS Write 200MB/s (10x20MB/s)
Considering the SAS drives should be 170MB/s each i'm guessing this is not normal? Where on earth does 30MB/s come from when uploading via SMB i'm stumped.
root@truenas[~]# pciconf -lvc mps0
mps0@pci0:1:0:0: class=0x010700 card=0x30d01000 chip=0x00641000 rev=0x02 hdr=0x00
vendor = 'Broadcom / LSI'
device = 'SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor]'
class = mass storage
subclass = SAS
cap 01[50] = powerspec 3 supports D0 D1 D2 D3 current D0
cap 10[68] = PCI-Express 2 endpoint max data 256(4096) FLR NS
link x8(x8) speed 5.0(5.0) ASPM disabled(L0s)
cap 03[d0] = VPD
cap 05[a8] = MSI supports 1 message, 64 bit
cap 11[c0] = MSI-X supports 15 messages, enabled
Table in map 0x14[0x2000], PBA in map 0x14[0x3800]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 5 corrected
ecap 0004[138] = Power Budgeting 1
ecap 0010[150] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
0 VFs configured out of 7 supported
First VF RID Offset 0x0001, VF RID Stride 0x0001
VF Device ID 0x0064
Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
ecap 000e[190] = ARI 1
root@truenas[~]# pciconf -lvc nvme0
nvme0@pci0:2:0:0: class=0x010802 card=0x50121987 chip=0x50121987 rev=0x01 hdr=0x00
vendor = 'Phison Electronics Corporation'
device = 'E12 NVMe Controller'
class = mass storage
subclass = NVM
cap 10[80] = PCI-Express 2 endpoint max data 128(256) FLR NS
link x4(x4) speed 8.0(8.0) ASPM L1(L1)
cap 11[d0] = MSI-X supports 9 messages, enabled
Table in map 0x10[0x2000], PBA in map 0x10[0x3000]
cap 05[e0] = MSI supports 8 messages, 64 bit, vector masks
cap 01[f8] = powerspec 3 supports D0 D3 current D0
ecap 0018[100] = LTR 1
ecap 001e[110] = L1 PM Substates 1
ecap 000e[128] = ARI 1
ecap 0001[200] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0019[300] = PCIe Sec 1 lane errors 0xf
root@truenas[~]# pciconf -lvc nvme1
nvme1@pci0:3:0:0: class=0x010802 card=0x50121987 chip=0x50121987 rev=0x01 hdr=0x00
vendor = 'Phison Electronics Corporation'
device = 'E12 NVMe Controller'
class = mass storage
subclass = NVM
cap 10[80] = PCI-Express 2 endpoint max data 128(256) FLR NS
link x4(x4) speed 2.5(8.0) ASPM L1(L1)
cap 11[d0] = MSI-X supports 9 messages, enabled
Table in map 0x10[0x2000], PBA in map 0x10[0x3000]
cap 05[e0] = MSI supports 8 messages, 64 bit, vector masks
cap 01[f8] = powerspec 3 supports D0 D3 current D0
ecap 0018[100] = LTR 1
ecap 001e[110] = L1 PM Substates 1
ecap 000e[128] = ARI 1
ecap 0001[200] = AER 2 0 fatal 0 non-fatal 3 corrected
ecap 0019[300] = PCIe Sec 1 lane errors 0xd
- Spermicro X9scm-f
- Intel Xeon Processor CPU SR0P4 E3-1230 v2 8 MB L3 Cache 3.3 GHz 69w LGA1155
- 32GB 4 x 8GB 2Rx8 PC3-14900E DDR3 1866MHz DIMM MAC PRO 2013
- LSI SAS 9201-16E
- 10 x Seagate Enterprise 4TB 7.2K 128MB 6Gbps 3.5" HDD ST4000NM0023
- Corsair MP510, Force Series, 240GB M.2 NVMe PCIe x4
Connected the NVME drives to the second and third PCIE slots, one is gen2 then mirrored for System pool
Suck another SSD in there for boot
Noticed I can saturate Lan downloaded from RAIDZ1 pool but uploading stuck at 30MB/s. Tired another laptop and a PC, both hard wired no dice.
I asumed that writing across 10 drives one way or another would be fast. Quick google and come to the realisation that RAIDZ1 is apparently not fast?
As a test transfered a large file, way bigger than my pool of ram from RAIDZ1 to the mirrored NVMEs and then transfered it back.
SAS -> NVME SAS Read 100MB/s (10x10MB/s) NVME Write 100MB/s (x1 as mirrored)
NVME -> SAS NVME Read 200MB/s (2x100MB/s) SAS Write 200MB/s (10x20MB/s)
Considering the SAS drives should be 170MB/s each i'm guessing this is not normal? Where on earth does 30MB/s come from when uploading via SMB i'm stumped.
root@truenas[~]# pciconf -lvc mps0
mps0@pci0:1:0:0: class=0x010700 card=0x30d01000 chip=0x00641000 rev=0x02 hdr=0x00
vendor = 'Broadcom / LSI'
device = 'SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor]'
class = mass storage
subclass = SAS
cap 01[50] = powerspec 3 supports D0 D1 D2 D3 current D0
cap 10[68] = PCI-Express 2 endpoint max data 256(4096) FLR NS
link x8(x8) speed 5.0(5.0) ASPM disabled(L0s)
cap 03[d0] = VPD
cap 05[a8] = MSI supports 1 message, 64 bit
cap 11[c0] = MSI-X supports 15 messages, enabled
Table in map 0x14[0x2000], PBA in map 0x14[0x3800]
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 5 corrected
ecap 0004[138] = Power Budgeting 1
ecap 0010[150] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
0 VFs configured out of 7 supported
First VF RID Offset 0x0001, VF RID Stride 0x0001
VF Device ID 0x0064
Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
ecap 000e[190] = ARI 1
root@truenas[~]# pciconf -lvc nvme0
nvme0@pci0:2:0:0: class=0x010802 card=0x50121987 chip=0x50121987 rev=0x01 hdr=0x00
vendor = 'Phison Electronics Corporation'
device = 'E12 NVMe Controller'
class = mass storage
subclass = NVM
cap 10[80] = PCI-Express 2 endpoint max data 128(256) FLR NS
link x4(x4) speed 8.0(8.0) ASPM L1(L1)
cap 11[d0] = MSI-X supports 9 messages, enabled
Table in map 0x10[0x2000], PBA in map 0x10[0x3000]
cap 05[e0] = MSI supports 8 messages, 64 bit, vector masks
cap 01[f8] = powerspec 3 supports D0 D3 current D0
ecap 0018[100] = LTR 1
ecap 001e[110] = L1 PM Substates 1
ecap 000e[128] = ARI 1
ecap 0001[200] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0019[300] = PCIe Sec 1 lane errors 0xf
root@truenas[~]# pciconf -lvc nvme1
nvme1@pci0:3:0:0: class=0x010802 card=0x50121987 chip=0x50121987 rev=0x01 hdr=0x00
vendor = 'Phison Electronics Corporation'
device = 'E12 NVMe Controller'
class = mass storage
subclass = NVM
cap 10[80] = PCI-Express 2 endpoint max data 128(256) FLR NS
link x4(x4) speed 2.5(8.0) ASPM L1(L1)
cap 11[d0] = MSI-X supports 9 messages, enabled
Table in map 0x10[0x2000], PBA in map 0x10[0x3000]
cap 05[e0] = MSI supports 8 messages, 64 bit, vector masks
cap 01[f8] = powerspec 3 supports D0 D3 current D0
ecap 0018[100] = LTR 1
ecap 001e[110] = L1 PM Substates 1
ecap 000e[128] = ARI 1
ecap 0001[200] = AER 2 0 fatal 0 non-fatal 3 corrected
ecap 0019[300] = PCIe Sec 1 lane errors 0xd
Last edited: