Notes on Performance, Benchmarks and Cache.

Juan Manuel Palacios

Contributor
Joined
May 29, 2017
Messages
144
Just found this post, this is the test on my main NAS with compression:ON

Having compression on is detrimental to benchmarking, because for example in this case when you write out a bunch of consecutive zeros to the filesystem, LZ4 compression is going to very efficiently reduce them down to a considerably smaller number of bytes that actually get flushed out to the media (as opposed to the reported uncompressed file size), and therefore the numbers you get will be heavily skewed toward an unrealistic performance that you wont get with real life data where such uniform byte patterns will surely be far from common, and therefore compression will never be able to do such a good job.
 
Joined
May 9, 2017
Messages
11
Ok, this certainly is not very realistic (I blame both compression as well as large ARC):
Write:
Code:
root@jpfserver:~ # dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 36.758513 secs (2921069776 bytes/sec)

Read:
Code:
root@jpfserver:~ # dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 19.041018 secs (5639098752 bytes/sec)


Specs:
- 2x Xeon on a Supermicro MB&chassis
- 36 disks x 2TB, stripe of 3-side mirrors
- 256GB RAM
- 3x 400GB NVMe's for SLOG / Cache
- FreeNas 11.1-U4

I'll try this a bit later also on our backup machine, same OS version, but 4x 8TB in raidz-1, Atom platform.
 
Last edited:
Joined
May 9, 2017
Messages
11
Agree. Then again, depending on your real life data sets, compression can be of enormous use. In 17 months, we’ve burned through 5.5TB of storage, at 1.6x compression rate. That’s a lot of TB saved, although we compress for speed rather than space. This pushes our projected pool growth by a year, which means buying the 6/8TB replacements (all 36+2 spares) a few thousands $$ cheaper. But yes, real life aggregated traffic wil, be much lower. Not that it matters, the NIC’s would cap that anyway.

Having compression on is detrimental to benchmarking, because for example in this case when you write out a bunch of consecutive zeros to the filesystem, LZ4 compression is going to very efficiently reduce them down to a considerably smaller number of bytes that actually get flushed out to the media (as opposed to the reported uncompressed file size), and therefore the numbers you get will be heavily skewed toward an unrealistic performance that you won't get with real life data where such uniform byte patterns will surely be far from common, and therefore compression will never be able to do such a good job.
..
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Having compression on is detrimental to benchmarking, because for example in this case when you write out a bunch of consecutive zeros to the filesystem, LZ4 compression is going to very efficiently reduce them down to a considerably smaller number of bytes that actually get flushed out to the media (as opposed to the reported uncompressed file size), and therefore the numbers you get will be heavily skewed toward an unrealistic performance that you won't get with real life data where such uniform byte patterns will surely be far from common, and therefore compression will never be able to do such a good job.
You can also use random for the data source instead of zero.

Sent from my SAMSUNG-SGH-I537 using Tapatalk
 

MrToddsFriends

Documentation Browser
Joined
Jan 12, 2015
Messages
1,338
You can also use random for the data source instead of zero.

Generally not a very good idea as in this way the benchmark result will be a mixed bag of three operations:
  1. Generation of random numbers
  2. Compression of random numbers
  3. Writing to disk
In the case of LZ4 compression step (2.) might not have much influence as LZ4 tends to bail out quickly when faced with data the algorithm considers as incompressible.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Generation of random numbers
Your CPU should generate random numbers much faster than the pool can commit data to disk. I think it would actually simulate a real workload much better than sending zero to disk in an uncompressed stream. The benchmark is not worth a lot if it doesn't tell you how the system would perform in a real working scenario.
 

MrToddsFriends

Documentation Browser
Joined
Jan 12, 2015
Messages
1,338
Your CPU should generate random numbers much faster than the pool can commit data to disk.

This is not generally true. Example from my Intel C2750 (Avoton) based machine using an uncompressed dataset:

Code:
root@blunzn:/mnt/volvar/ctest_nocompress # dd if=/dev/zero of=zero.dat bs=2048k count=5k
5120+0 records in
5120+0 records out
10737418240 bytes transferred in 82.465449 secs (130205054 bytes/sec)
root@blunzn:/mnt/volvar/ctest_nocompress # dd if=/dev/random of=random.dat bs=2048k count=5k
5120+0 records in
5120+0 records out
10737418240 bytes transferred in 185.157436 secs (57990748 bytes/sec)

I think it would actually simulate a real workload much better than sending zero to disk in an uncompressed stream. The benchmark is not worth a lot if it doesn't tell you how the system would perform in a real working scenario.

While using /dev/zero as data source results in a write performance test, using /dev/random instead not only results in a CPU bound test, the outcome of the latter is also dependent on the implementation of /dev/random. This is a thread about ZFS and storage performance, or did I miss something?
 
Last edited:

Stux

MVP
Joined
Jun 2, 2016
Messages
4,367
You can also use random for the data source instead of zero.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

On my Xeon d random only worked at a few hundred megs a sec

You can bench your random device by outputting it to dev null ;)
 
Joined
May 9, 2017
Messages
11
Alright, here's the Supermicro Mini backup appliance:
Write:
Code:
root@freenas:~ # dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 83.339665 secs (1288392294 bytes/sec)

Read:
Code:
root@freenas:~ # dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 52.352838 secs (2050971578 bytes/sec)


Specs:
- 1x Intel Atom C3558 Denverton, 4 cores
- 4x 8TB disks in raidz-1
- 16GB RAM
- consumer SSDs for cache/log (1 each)
- FreeNAS-11.1-RELEASE (it's a bit behind on updates...)

Funny thing, with both of these systems showing lab benchmark results in the GB/s range, the replication tasks between them are dog slow. This due to SSH overhead and encryption, as well as the limit to a single NIC on the client-side. It crawls to about 20MB/s, a whooping 50x less than the above results. Anyway, it works, and storage performance isn't the bottleneck.
 
Last edited:

Stux

MVP
Joined
Jun 2, 2016
Messages
4,367
Did you disable compression on the dataset you benched with dd?
 
Joined
May 9, 2017
Messages
11
Did you disable compression on the dataset you benched with dd?
Nope, both systems have compression on, hence the ludicrous results. Also, compression & encryption support in hardware, for both the Xeons, as well as the tiny Denverton (Not Avoton, as previously mentioned).
Here's the repeat test on an uncompressed dataset, on the large system (dual Xeon, 36 disks):
Code:
root@jpfserver:/mnt/jpfpool/data/nocompress # dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 97.887790 secs (1096910881 bytes/sec)

root@jpfserver:/mnt/jpfpool/data/nocompress # dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 49.175610 secs (2183484513 bytes/sec)

Write speed is 1.17x slower, read speed took a harder hit, 2.57x slower.
 
Last edited:

MrToddsFriends

Documentation Browser
Joined
Jan 12, 2015
Messages
1,338
Funny thing, with both of these systems showing lab benchmark results in the GB/s range, the replication tasks between them are dog slow. This due to SSH overhead and encryption, as well as the limit to a single NIC on the client-side. It crawls to about 20MB/s, a whooping 50x less than the above results.

Have a look at the CPU and Disk I/O graphs in the FreeNAS GUI (or top while replication is running as a next step). I would expect that you are able to identify the bottleneck.

Anyway, it works, and storage performance isn't the bottleneck.

That's the expected outcome of the storage benchmarks discussed here.
 
Last edited:
Joined
May 9, 2017
Messages
11
Have a look at the CPU and Disk I/O graphs in the FreeNAS GUI (or top while replication is running as a next step). I would expect that you are able to identify the bottleneck.



That's the expected outcome of the storage benchmarks discussed here.

Thanks a lot, I will. We'll repeat the tests in the next weeks anyway, as the mini server is freshly deployed, and a pool replication task has been running on-going for the entire week (and during all these tests). So far so good, we're very happy with the results (both tested and, most importantly, real life ones). The big machine has been running live for 18 months (ever since 9.10.1-U4), not a single hiccup on the pool side of things. Even the crappy Atom machine is proving faster and more consistent than expected, I'm happy we got rid of the Synology boxes it replaced (and I really like the new Atom line!).

We are still struggling with VM stability, as well as some issues regarding VM networking and LAGG (I'll detail that in appropriate threads), but I'm confident that these will be fixed soon(ish) in the 11 train (or we will learn how to work around the issues). FreeNAS is going in the right direction.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,367
Write speed is 1.17x slower, read speed took a harder hit, 2.57x slower.

If your big system is using mirrors you’d expect the write speed to be about half the read speed. Which it is.
 

RegularJoe

Patron
Joined
Aug 19, 2013
Messages
330
It really sounds like the guys at iXsystems need to add a native benchmark button or jail. It might want to have stats for all disks in the pool, compression on and off, sync disabled, sync standard and sync always. One that does a SMB test, NFS and AFP with changing the number of threads up to the maximum in the box.

A standard way to compare FreeNAS / TrueNAS systems would be nice, so we know if a tweak or change makes any difference. Example: almost identical HP DL380 G7 and G8 systems... i.e. did the PCIe 3.0 help? Then a G8 and G9 to compare if DDR4 helped any and if so how much?

Maybe this benchmark jail is purchased via iTunes/play store for $0.99 or something to see how many people buy it and have a way to publish your NAS statistics at iXsystems so they can brag about how many TB served by FreeNAS/TrueNAS and then who has the fastest pool. :smile:
 

wrayste

Cadet
Joined
Jun 4, 2019
Messages
5
New to FreeNAS and ZFS but enjoying it, I think I appreciate the advantages and disadvantages of the design but the performance isn’t perhaps what I was expecting so wanted to get a mark to market.

The system is a SuperMicro A2SDi-H-TF, 32 GB RAM with 4x Seagate ST8000NM0045 Exos Enterprise 8TB SATA.

I’m planning on running them with RAIDz2 but in evaluating the performance I’m getting ~40 MB/s (compression off, sync always, 1M record size). I appreciate the RAM cache and compression help significantly which is fabulous, but I was anticipating closer to the theoretical performance of the drives which is 237 MB/s (command: dd if=/dev/zero of=tmp.000 bs=2048k count=50k) as a base.

I tried the disks individually and get ~70 MB/s with the same settings. With a different OS and filesystem I saw sequential writes just over 200 MB/s for individual disks. So a hardware problem seems unlikely. I've checked the ashift value is 12 for the vdev which seems to be the critical setting for 4kn drives).

Are my expectations incorrect and these numbers are reasonable for FreeNAS and ZFS?
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
You should always have LZ4 compression on, because it improves performance even if the data is only a little compressible. Then, what kind of storage are you trying to create?
4x Seagate ST8000NM0045 Exos Enterprise 8TB SATA.
iSCSI, SMB, NFS, what are you doing?
Are my expectations incorrect and these numbers are reasonable for FreeNAS and ZFS?
How the pool needs to be configured for best performance depends entirely on how you plan to use the storage.
but in evaluating the performance I’m getting ~40 MB/s (compression off, sync always, 1M record size). I appreciate the RAM cache and compression help significantly which is fabulous, but I was anticipating closer to the theoretical performance of the drives which is 237 MB/s (command: dd if=/dev/zero of=tmp.000 bs=2048k count=50k) as a base.
These settings are entirely inappropriate for most uses and will actually make the system slower.
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
You should always have LZ4 compression on, because it improves performance even if the data is only a little compressible. Then, what kind of storage are you trying to create?

iSCSI, SMB, NFS, what are you doing?

How the pool needs to be configured for best performance depends entirely on how you plan to use the storage.

These settings are entirely inappropriate for most uses and will actually make the system slower.
When doing initial performance testing you can not have compression on because writing using dd and zero. So for testing keep it off.
 

wrayste

Cadet
Joined
Jun 4, 2019
Messages
5
You should always have LZ4 compression on, because it improves performance even if the data is only a little compressible. Then, what kind of storage are you trying to create?

Definitely for the real-world it will be enabled and looking forward to its benefits. The storage will eventually be a mixture of video, music, pdfs, etc. I need to read a little more on record size for each use case.

iSCSI, SMB, NFS, what are you doing?

For these tests they are all dd run from a shell on the box to avoid any additional problems or overhead/limitations of going over the network.

How the pool needs to be configured for best performance depends entirely on how you plan to use the storage.

These settings are entirely inappropriate for most uses and will actually make the system slower.

Understood, my thought here was to try and understand the performance of the underlying disk hardware. Compression and RAM disk help (the former a small amount, the latter a huge amount) and while this test is very synthetic, that should also mean that the results are very good. However what I think I am seeing is much worse performance in the synthetic test than I expected. If the synthetic test was coming out over 150 MB/s but real-world performance was at these levels then I could understand that it is the work-load, but in this case they appear to be very similar and slower than what the drive says it should be able to accomplish.

Through my reading, the performance of RAIDz2 should be roughly equivalent to the slowest drive in the pool which I'd be happy with. Maybe my expectations are wrong on the raw disk performance I should be seeing. Ultimately I'll enable compression and RAM disk and write performance will shoot up while I remain under the RAM limit which will be very for nearly all the time after the initial load. Before doing that though it would be good to know if there isn't an underlying mistake/fault.

Thanks for the help
 
Top