Performance...again, again

[Ulrich Jorgensen]

Hi there everyone,

We have been trying a lot of different combinations to get good storage performance for our ESXi cluster of 4 nodes, but we seem to struggle to find the right way.

I have run multiple FreeNAS boxes, for several years, and I have numerous articles on ZFS performance and recommendations, but I cannot get it right.

We have 10Gbit/s network, Jumbo frames activated
Our test FreeNAS is a SuperMicro box with 6-core Xeon and 32GB RAM
There are 6x Samsung SM863 960GB SSD's in striped mirrors
There is 1 Intel P3700 NVMe 800GB as SLOG device.

Internal speed test gives me:

% /usr/bin/time -h dd if=/dev/zero of=/mnt/TEST/tmp bs=2048k count=10k
10240+0 records in
10240+0 record out
21474836480 bytes transferred in 7.738622 secs (2775020811 bytes/sec)
7.74s real 0.02 user 7.71s sys
% /usr/bin/time -h dd if=/mnt/TEST/tmp of=/dev/zero bs=2048k count=10k
10240+0 records in
10240+0 record out
21474836480 bytes transferred in 3.948149 secs (5439215903 bytes/sec)
3.94s real 0.00 user 3.94s sys


When I put this in test over iSCSI I only get 467MB/s write and625MB/s read
With NFS I get 775MB/s write and 779MB/s read

These numbers are with Sync = Off, with the recommended sync=always, those numbers are 20-30% lower (I forgot to write them down ;-))

Testing network with iPerf I get around 9 Gbit/s which should make it possible to climb above the numbers we are seeing.
I will set up LAG today, but as we are not saturating the 10 Gbit/s link, I do not expect to see much improvement there.

I really hate to see all these MB/s lost in iSCSI and/or NFS, does any anyone have a similar experience, and found a solution?

 

[Chris Moore]

There are 6x Samsung SM863 960GB SSD's in striped mirrors

I think it might be a limitation of the SSD pool. I would suggest adding another pair of SSDs so you have 4 vdevs instead of only 3.
I think I found the correct spec for that model SSD and it says "Up to" when talking about the read and write speed. If it is like some of the hardware I have dealt with, it is probably not quite getting there and the overhead involved in ZFS, iSCSI and such are taking enough away to make it a choke point. If I recall correctly, doing a dd to the drive is not a good test as it bypasses a lot of the software layers that are what is slowing you down. How is the processor utilization during your testing?

I will set up LAG today, but as we are not saturating the 10 Gbit/s link, I do not expect to see much improvement there.

I don't think the networking is the problem.

 

[kdragon75]

If you need the RAW speed and can't lose ANYTHING to iSCSI, try Fibre Channel! You just need a full set of HBAs and a switch! ;). I only have two hosts so I can run point to point.

 

[Ulrich Jorgensen]

If you need the RAW speed and can't lose ANYTHING to iSCSI, try Fibre Channel! You just need a full set of HBAs and a switch! ;). I only have two hosts so I can run point to point.

Thank you for your reply.
Fibre Channel is not an option at this moment, but I will look into it.
It is not a question of not wanting to loose anything, but we are loosing more than 80%

I would be satisfied by saturating the 10Gbit/s link, and with LAG beeing able to go a little higher.

 

[Ulrich Jorgensen]

I think it might be a limitation of the SSD pool. I would suggest adding another pair of SSDs so you have 4 vdevs instead of only 3.
I think I found the correct spec for that model SSD and it says "Up to" when talking about the read and write speed. If it is like some of the hardware I have dealt with, it is probably not quite getting there and the overhead involved in ZFS, iSCSI and such are taking enough away to make it a choke point. If I recall correctly, doing a dd to the drive is not a good test as it bypasses a lot of the software layers that are what is slowing you down. How is the processor utilization during your testing?

I don't think the networking is the problem.

That is why we have an SLOG NVMe, to accelerate the writes. There is an 800GB Intel P3700 NVMe as SLOG device, and it should perform up to 1900MB/s write and 2800MB/s read.
Is there another better test than dd?

Agree on the network part.
Thank you.

 

[Chris Moore]

I would be satisfied by saturating the 10Gbit/s link, and with LAG beeing able to go a little higher.

There is 1 Intel P3700 NVMe 800GB as SLOG device.

That SLOG device is actually going to cap your performance at right about 10Gig.

 

[Ulrich Jorgensen]

That SLOG device is actually going to cap your performance at right about 10Gig.

How so? 1900MB/s is close to 15 Gbit/s

The issue is that I am not getting near the teoretical 1250 MB/s of the 10Gbit/s, I am only getting about 1/2 that speed, which is why I posted this.

 

[Chris Moore]

How so? 1900MB/s is close to 15 Gbit/s

The issue is that I am not getting near the teoretical 1250 MB/s of the 10Gbit/s, I am only getting about 1/2 that speed, which is why I posted this.

I am going from memory, but those numbers look wrong.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Ulrich Jorgensen]

I am going from memory, but those numbers look wrong.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

Hmm...
- The Intel P3700 PCIe 800GB writes with up to 1900 MB/s and reads with up to 2800 MB/s source:https://ark.intel.com/products/79627/Intel-SSD-DC-P3700-Series-800GB-2_5in-PCIe-3_0-20nm-MLC
- 10 Gbit/s = 1.25 GB/s or 1280 MB/s

When I get results near 600 MB/s over iSCSI and NFS, that is approx 1/2 of 1280MB/s, and about 1/3 of 1900MB/s

Edit: I forgot, 1900MB/s equals 14.84375 Gbit/s