Hyperthreading and ZFS checksums

[JoeAtWork]

Hi All,

Is there an easy way to see how hyper-threading is causing latency for the ZFS checksum process? I have three copies from a NVME to a 12 disk stripe and it looks like I am hurting the CPU;s.

From this discussion it sounds like hyper-threading should be OFF for our TrueNAS storage systems :
https://www.reddit.com/r/zfs/comments/d9v1cl/some_interesting_findings_while_benchmarking/

What is your take?

Thanks,
Joe

 

[AlexGG]

Disable the hyperthreading and re-test, then compare the result?

You do have compression disabled, do you?

The checksum algorithm, are you using FL4, or are you using something else?

 

[RegularJoe]

I have not changed any of the defaults, compression is off, no atime and sync disabled on this test. This was with hyper threading off as this system has dual 10 core processors. This was the CPU when doing a copy from the nvme to 3 pools of 12 striped drives(for benchmarking only).

I accidentally have it on now and have to reboot to disable it. my benchmark copies 250 100 gig files and then I scrub. The device with much fewer blocks of data is the runt and goes in the runt bin or back to the vendor.

It is a Dell R720 with 192 gig of RAM and it is NOT running in NUMA mode but the intel hack mode.

 

[AlexGG]

I had several considerations but I am not able to come to any conclusion or make any prediction. If you ever do an actual measurement, pretty please post the result, I'm intrigued.

 

[RegularJoe]

I have been told over the years hyperthreading increases latency and unless you see a huge benefit to having it on you should turn it off. Home users have no choice and the need all the cores so I see it on for all desktop devices.

This report shows some tweaks of the Dell settings that I will have to verify:

https://lafibre.info/images/materiel/201212_dell_bios_tuning_for_performance.pdf

 

[JoeAtWork]

with Hyper-threading Disabled :
s1 - 05:52:20
s3 - 05:44:02
s4 - 05:45:16

5 hours we will see how the results are.

 

[HoneyBadger]

I have been told over the years hyperthreading increases latency and unless you see a huge benefit to having it on you should turn it off. Home users have no choice and the need all the cores so I see it on for all desktop devices.

This report shows some tweaks of the Dell settings that I will have to verify:

https://lafibre.info/images/materiel/201212_dell_bios_tuning_for_performance.pdf

Relevant excerpt:

"Customers running extremely low-latency environments may prefer to disable Logical Processors for the system latency improvement offered, but again this is a special case and is not recommended for most computing environments."

Normally this is intended to apply to things like HPC with things like RDMA interconnects, but those using TrueNAS as a pure storage solution (no jails/VMs/not using SCALE) and therefore with CPU to spare would likely benefit from the edge-case improvements on latency.

 

[JoeAtWork]

With hyper-threading "ON" the GUI seemed to be very laggy and slow when doing the 3 scrubs.

The scrub times were about the same:
s1 - 05:25:40
s3 - 05:46:01
s4 - 05:43:56

 

[QonoS]

Do you run latest Firmware/UEFI/BIOS on your R720 ?

 

[AlexGG]

So, there is no practical difference from the filesystem side. Thanks.

 

[JoeAtWork]

So, there is no practical difference from the filesystem side. Thanks.

at least when doing 3 scrubs of random data that fills 74% of the disk. I hate the additional risk of side channel attacks that HT exposes you to.

This is the server I am using :
TrueNAS 12.0-u5
no comrepssion
no dedupe
no atime
sync disabled

PowerEdge R720xd
BIOS 2.8.0
LC Controller 2.63.60.62
2x CPU E5-2660 v2
192gb ecc ram
2 boot SATA SSD's
1 NVME Samsung Pro 970 512gb
(3) HP D2600 shelfs 3 tb dell sas 7200 rpm disks - Dell 12Gbps HBA(SAS3008)
12 internal dell sas 7200 rpm disks - attached to SAS9207-8i(SAS2308)

 

[HoneyBadger]

no comrepssion

I know you're just doing benchmarking now, but compression is basically a "free lunch" and should be on for almost every non-archival consumer workload. If you're storing your personal collection of H.265 BluRay rips then sure, leave it off - but general documents/files absolutely get wins from compression.

 

[JoeAtWork]

just for the benchmark, it should not make a difference as I made my test files from /dev/random

NFS and Compression are like peanut butter and jelly.