Poor SMB performance & high CPU load

[Volts]

opportunity to learn something new today

Perfect! Once I get that out of the way, I can go back to hitting things with hammers.

How well does the standalone lz4 executable correlate to ZFS behavior?

I thought these both seemed relevant:

A simple (real world) ZFS compression speed an compression ratio benchmark

Posted in r/zfs by u/Schmidsfeld • 166 points and 37 comments

old.reddit.com

OpenZFS: Understanding Transparent Compression

OpenZFS’ transparent compression is one of the many compelling yet misunderstood features. Read about more about its amazing potential performance benefits in our “OpenZFS in Depth” series.

klarasystems.com

This has a bunch of meaty refererences:

LZ4 vs. ZStd

I'm about to create a new pool (TrueNAS-12.0-U1). There doesn't seem to be much info about the different choices and which one to pick. The data is a mixed bag of everything. Small files, big files, source code, incompressible files, VMs, … LZ4 or ZStd? How to decide?

www.truenas.com

Including a link to this:

ZSTD Benchmarks

fio SSD ZSTD Relative Compression Performance System: Intel Xeon E5-2630 v4 @ 2.20GHz (10 cores, 20 threads), 32GB RAM, 4x 500 Samsing 860 EVO SSDs ZFS config: ashift=12 atime=off recordsize=128k arc_max=256m Workload: fio uncompressible data to SSD: bs=128k rw=randwrite numjobs=8 iodepth=16 siz...

docs.google.com

Showing that with incompressible data, on SSD, =lz4 achieved 98% the speed of =off.

So my thoughts are a combination of these factors -
A 2% worst-case penalty isn't significant
Any real-world mix of data will compress much more than 2%
Systems with spinning disks benefit greatly from avoiding IOPS

I would like to see a real-world example of =off being faster. I hypothesize that would be an unusually slow CPU, paired with disproportionally fast storage, writing incompressible data.

 

[Davvo]

Thanks for your reply, not too encouraging though :-( on running a power efficient NAS

Why would you virtualize if you want a power efficency NAS?

 

[jgreco]

How well does the standalone lz4 executable correlate to ZFS behavior?

It doesn't really matter that much. A call to a compression library is slower than no call to any compression library.

A 2% worst-case penalty isn't significant

So upon finding that I'm correct, you decide instead to declare the result insignificant. Okay.

Any real-world mix of data will compress much more than 2%

Not true. There's plenty of already incompressible data out there. It would be fine to suggest that there is a large amount of real world data that compresses readily that can save 2% or more. It's better to make true statements than make statements that appear to be slanted to advance one's arguments without being based in fact.

Systems with spinning disks benefit greatly from avoiding IOPS

Sometimes. But it's also fair to say that systems with slow I/O (including SATA SSD) benefit greatly from being able to exploit extremely fast CPU; compression is often useful in such scenarios which is why ZFS will compare its attempt at compression against the uncompressed block. It potentially saves storage space, and upon read, data typically decompresses at a rate faster than a slow I/O device (such as SATA SSD) is capable of feeding it to the host in an uncompressed state.

I would like to see a real-world example of =off being faster. I hypothesize that would be an unusually slow CPU, paired with disproportionally fast storage, writing incompressible data.

I would think it would be fairly easy to show this for any one of those variables, and as far as I'm aware, a number of people have done so.

 

[Whattteva]

Any real-world mix of data will compress much more than 2%

Not true. There's plenty of already incompressible data out there. It would be fine to suggest that there is a large amount of real world data that compresses readily that can save 2% or more.

Just to illustrate this point, I took a look at one of my pools. As you can see from the screenshot below, the vast majority of the data only has 2-4% compression savings. There are only 2 datasets that have more significant savings and those are "vm" and "pve". "vm" is a zvol for a Windows VM, which is mostly empty, while "pve" is just hosts various Linux ISO images and those constitute for a small fraction of my entire "real-world mix" of data resulting in the pool's entire saving to just 2%.

 

[jgreco]

I've been working with these kinds of things from the days of DoubleSpace and Stacker under DOS, and while they may reclaim some space, they always seem to do so at some cost. It's a classic pick-any-two problem. ZFS is nice in that it is pretty efficient, allows you to pick yer poison, and knows how to back off on incompressible data. That said, if you have a really fast I/O subsystem like NVMe and a really fast network like 25G, 40G, or faster, CPU based compression is going to be highly noticeable when inserted into the chain. There is already one company trying to advocate for replacing CPU-based RAID computations with GPU (see graidtech.com) but this doesn't really seem to be taking the industry by storm. I think it's reasonable to note that the act of opportunistic compression may pay off in many cases, but that turning it off is a good way to eliminate any chance of a CPU based bottleneck. We still have people showing up with old, slowish, hot and heavy Westmere and Nehalem gear trying to turn them into NAS.

 

[Whattteva]

We still have people showing up with old, slowish, hot and heavy Westmere and Nehalem gear trying to turn them into NAS.

It's almost old enough to hold its own driver's license!

 

[mrpasc]

Anyone tried to run SLOG on a Samsung Pro 980? I mostly saw enterprise SSDs being used...(not in my budget currently)

Client Optanes like p1600x, 900p/905p are in the same price range (or even lower for the p1600x) than a Pro 980. eBay is full of 900p 280GB cards for around 80-90$, p1600x 118GB at newegg.com or Amazon for around 60$, 905p 380GB I have seen at the STH sale threat for 150$.