Truenas core and ESXi hosts with NFS

[orddie]

Hey all.
I have a TrueNas core server, setup with a AMD Ryzen 7 3700X processor and 64GB of memory.

Disks consist of 10 500GB SSD VDEV's in mirrors (20 disks in total).
I use an LSI 12 GB/sec card and some of the motherboard 6GB SATA ports to fill the rest. Each SSD is connected at 6 GB even when connected to the LSI card.

Network connectivity is done via 10GBE non-routed subnet.
TrueNas server has a LAG, showing 20GBE total bandwidth.
Each ESXi host has a single 10 GBE network connection.

No LOG, Cache, Metadata, or Dedupe has been configured.
I see some high VM read and write latency going up as high as 2000 MS.

Would adding some NVME into the mix for LOG and Cache assist?

 

[mav@]

SSDs may have very different synchronous write latency. Somewhere here on the forum is a huge thread with benchmark results for different SSD in role of SLOG. Depending on what SSDs you have, additional SLOG may or may not improve performance.

 

[Dice]

2000ms?
Is that from Reports -> Disk Reports -> Disk Latency?
Can you post a screenshot to maybe get some clues?

 

[HoneyBadger]

Somewhere here on the forum is a huge thread with benchmark results for different SSD in role of SLOG.

You rang?

SLOG benchmarking and finding the best SLOG

I'd like to take a few minutes to talk about SLOG devices and what makes good ones versus bad ones. I have no doubt that this will be a controversial topic since this is not well understood by many people. In short, there's 3 things that you need for a "great" SLOG: 1. Fast throughput 2...

www.truenas.com

An SLOG sufficient for an all-SSD system would probably be the Optane P4800X or P5800X - you can also go with dedicated NVRAM devices such as the Radian RMS-200. There are SAS SSDs that can perform reasonably well if hotswap is a requirement, but in general NVMe will perform best.

 

[Dice]

I agree a good SLOG device is the goto-solution for this problem.
What bugs me is that I cannot see how 12 mirrors of SSDs would generate 2000ms delays due to IOPS strangling. Of course there may be a monster load on these drives, which would ...eventually be a sufficient explanation.
Can you please quantify the load in any way? What's the load ?

 

[zookeeper21]

SSDs may have very different synchronous write latency. Somewhere here on the forum is a huge thread with benchmark results for different SSD in role of SLOG. Depending on what SSDs you have, additional SLOG may or may not improve performance.

I have four SSD in my pool, I am thinking to use NVME for L2arc or slog or may be both. Will I get any benefit? I have 64gb RAM in it.

 

[mav@]

I have four SSD in my pool, I am thinking to use NVME for L2arc or slog or may be both. Will I get any benefit? I have 64gb RAM in it.

From L2ARC you likely not, since even NVMe bandwidth will not compare to the speed of the wider SSD pool, plus L2ARC needs to be filled first.

 

[kspare]

Grab an 800gb P3700 off ebay. they work amazing. I have some radian 200's I could sell you. I found them to be very unreliable in my systems...not sure why. if you go with a p4801x go large. I had troubles with the 100gb being slower than my p3700 800gb.

I know there are faster out there, but for the money right now you just cant beat the p3700!