NVMe Z1 Pool - best practice ? Ultra slow writes

[Herr_Merlin]

I've setup an ESXi with FreeNas.
Now I got 4 NVMe drives, which I passed to the freenas VM.
The performance is, well I would expect the same performance out of a floppy Z1 volume..
So clearly I missed some tuning and setup.
Are there guidelines, I couldn't find any, for NVMe pools?

Using 4 of those: TS1TMTE220S mounted to a card with a PCIe Switch. So maximum bandwidth is limited to x8 for all 4.
I don't see any errors within freenas or the log.
As a little reference I've tested the 4 SSDs with the card together with windows server 2019 and build a Storage Spaces pool with one drive parity and ReFS as file system.. with that setup I was able to get nearly the bandwidth of the PCIe x8 3.0..

Thus I spend some time googling for best practice for NVMe pools but didn't find anything.

 

[sretalla]

With a Z1 pool, you're going to expect the IOPS of only a single NVME, so the way you're testing with tiny (512b) IO, is bringing that limitation to the front.

With that single NVME also in contention to write at the same time as the other 3 with the limited PCIe lanes, I'm not surprised to see IO is significantly worse than would be possible with 16 available lanes (and hence no contention).

I'm also not surprised to see nothing in the logs of FreeNAS about that, it's handled by the Chipset as far as I understand, so as long as the IO gets to the card eventually, there's nothing to report.

What do you get from a pool of 1 NVME? also for a striped pair? (and just for fun stripe all 4)

 

[Herr_Merlin]

well I would be happy with 1/4 of the IO of a single NVMe.. as you can see from the test above it's way below.
Further more I had tested the card with the drives storage spaces and ReFS... that's as close as it could come to comparable to ZFS Z1.
Additionally as you could see it varies from test to test to way slower than an single 5.9k HDD...
will test with single drive and report back

tested as suggested by @sretalla with stripes see below. no real performance impact.
while still best outcome is with 2 drives

 

[Herr_Merlin]

Digging further I noticed if I use large files with the stripe test. only tested with 4. performance will break down.
for insanity check I removed the nvme devices from the vm and attached them to the host. Than I created 4 vm datastores and put on each a single vmdk attached to the freenas. just thin provisioned for a quick test.
I've noticed to so good performance with that test but way better and more consistent than with freenas. noticed the bumps in write speed during the test from taskmgr. I assume the statement the nvme are fighting over bandwidth might be right.

I assume I need 2 x8 cards for 2 nvme each only so bandwidth is there.

 

[Herr_Merlin]

I did some changes to the ESXi and upgraded from 6.5 to 6.7.
Went for a Z1 with the 4 NVMe drives and tested prior to upgrade and later.
results are strange. As both times hardware is passed to the VM.
upgraded FreeNAS to TureNAS and run the test with 6.5 and again with 6.7

Performance is at least now consistent but slow..

 

[Patrick M. Hausen]

Don't do RAIDZn for block storage.

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

 

[Herr_Merlin]

Fun fact.
There is no difference between RAIDz1 and mirrors regarding performance.

 

[Patrick M. Hausen]

I should have read the whole thread, sorry.

 

[HoneyBadger]

Another user had some issues with interrupts using PLX-based PCIe switches with NVMe devices. The FreeBSD drivers might not be as performant as the Windows or Linux based solutions.

nvme missing interrupt

Hello! I built a all nvme freenas with the following specs in Sep 2019: SuperServer 2028U-TN24R4T+ SuperMicro X10DRU-i+ 2 x Intel Xeon E5-2620 v3 384GB (24 x 16GB) 2RX4 PC4-2133P DDR4 MEMORY 24 x Intel DC P3600 SSD 800GB NVMe PCIe 3.0 After installing freenas 11.2 I found out that after adding...

www.truenas.com

The use of PLX also impacts latency which will show up at small records as well.

QNAP QM2-2P10G1TA Adapter General Purpose Mini Review

Our mini-review of the QNAP QM2-2P10G1TA adapter which brings 10Gbase-T and two NVMe SSDs to a single PCIe slot, with a catch in terms of performance

www.servethehome.com

Adding ESXi/vSphere into the mix could certainly be adding complexity/bottleneck points too.

 

[sretalla]

Adding ESXi/vSphere into the mix could certainly be adding complexity/bottleneck points too.

Since TrueNAS runs off a USB stick, giving that a direct hardware test would be an interesting (and not costing a lot) exercise.

 

[Herr_Merlin]

Yeah can do probably later the week.

 

[Herr_Merlin]

Tested.
So it comes down to BSD having bad drivers for the PCIe switchs or something like that.
If I insert a single NVMe into a PCIe x8 Slot with an add-on card without a PLX I get the same performance as while using Windows Server with PLX between the SSDs and the OS.
Issue is this old board is unable to run PCIe bifurcation as Supermicro never released a fixed bios for this X9 Board..