Hello all,
I am new to FreeNas and I am having some trouble getting the performance Im expecting from my setup. I'm trying to get that squared away before I put tons of data on the box.
The setup is:
8 x Western Digital WD60EZRZ drives, which are rated at 180MB/s [1]
FreeNAS Mini-XL with an Intel Atom 8 core CPU, 32GB of RAM, L2ARC SSD, SLOG SSD
My intention is to build this into a set of 4 mirrors but for my initial testing I just wanted to see how much throughput this system can get from the drives. I have created a pool using all 8 disks in stripe mode with no compression, atime off, no deduplication, and 1M record size, sync=standard, and L2ARC and SLOG are not connected to the pool. The drives are Advanced Format (512e so they report 512) and I have tried with both ashift=9 and ashift=12 (with very little effect). I have then run numerous tests including iozone, dd tests, etc. My expectation is that I would see about 8x180MB/s = 1.44 GB/s (probably a little less due to overhead). I am seeing that speed (or acceptably close) for writes, but reads are peaking at suspiciously close to 1/2 of that speed for some reason.
IOZONE:
Ran IoZone using this command:
iozone -R -l 8 -u 8 -r 1024k -s 32G -F test1 test2 test3 test4 test5 test6 test7 test8
With these results (truncated for brevity):
"Throughput report Y-axis is type of test X-axis is number of processes"
"Record size = 1024 kBytes "
"Output is in kBytes/sec"
" Initial write " 1198412.58
" Rewrite " 1199257.48 // Write speed close to expected
" Read " 677269.62 // Read speeds strangely about half!?
" Re-read " 674305.20
" Reverse Read " 317227.65
" Stride read " 296686.93
" Random read " 290240.55
" Mixed workload " 1144392.45
" Random write " 1160763.52
" Pwrite " 1207724.31
" Pread " 670862.85
" Fwrite " 1173588.23
" Fread " 683215.71
DD:
Write test, large file to blow cache:
dd if=/dev/zero of=test bs=1M count=32000
33554432000 bytes transferred in 22.794896 secs (1472015139 bytes/sec)
Read test (ARC cached files will show speeds of over 3GB/s, this run is typical for uncached reads):
% dd if=test of=/dev/zero bs=1M count=32000
33554432000 bytes transferred in 44.108189 secs (760730210 bytes/sec)
Strangely this is almost exactly 1/2 the expected speed, 180*4=720MB/s, and it is highly consistent (+/- 100MB/s).
Write speeds monitored using 'zpool iostat 1' during testing show peak throughput at about 1.4 GB/s as expected, and stay consistently high:
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
pool_01 40.3G 43.5T 0 1.41K 0 1.41G
Read speeds occasionally peak around 1GB/s, but average around 700MB/s:
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
pool_01 63.3G 43.4T 1.06K 0 738M 0
SCRUB:
Running a scrub operation on the pool and watching throughput using iostat shows the array is capable of the full 1.44GB/s and consistently gets close to this during the scrub:
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
pool_01 63.3G 43.4T 1.41K 0 1.41G 0
It looks to me like the hardware is fully capable of running at the expected 1.4 GB/s, and writes generally do get there, but reads appear to be running at half speed unless I test using scrub which does show the full expected speed.
The pool has 0% fragmentation:
% zpool list pool_01
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
pool_01 43.5T 32.0G 43.5T - - 0% 0% 1.00x ONLINE /mnt
I've been reading posts and trying everything I can think of for a week and I'm out of ideas! Any help with my methodology, troubleshooting, or correcting my expectations or understanding would be most appreciated!
Thanks!
[1] - https://documents.westerndigital.co...-sheet-wd-blue-pc-hard-drives-2879-771436.pdf
I am new to FreeNas and I am having some trouble getting the performance Im expecting from my setup. I'm trying to get that squared away before I put tons of data on the box.
The setup is:
8 x Western Digital WD60EZRZ drives, which are rated at 180MB/s [1]
FreeNAS Mini-XL with an Intel Atom 8 core CPU, 32GB of RAM, L2ARC SSD, SLOG SSD
My intention is to build this into a set of 4 mirrors but for my initial testing I just wanted to see how much throughput this system can get from the drives. I have created a pool using all 8 disks in stripe mode with no compression, atime off, no deduplication, and 1M record size, sync=standard, and L2ARC and SLOG are not connected to the pool. The drives are Advanced Format (512e so they report 512) and I have tried with both ashift=9 and ashift=12 (with very little effect). I have then run numerous tests including iozone, dd tests, etc. My expectation is that I would see about 8x180MB/s = 1.44 GB/s (probably a little less due to overhead). I am seeing that speed (or acceptably close) for writes, but reads are peaking at suspiciously close to 1/2 of that speed for some reason.
IOZONE:
Ran IoZone using this command:
iozone -R -l 8 -u 8 -r 1024k -s 32G -F test1 test2 test3 test4 test5 test6 test7 test8
With these results (truncated for brevity):
"Throughput report Y-axis is type of test X-axis is number of processes"
"Record size = 1024 kBytes "
"Output is in kBytes/sec"
" Initial write " 1198412.58
" Rewrite " 1199257.48 // Write speed close to expected
" Read " 677269.62 // Read speeds strangely about half!?
" Re-read " 674305.20
" Reverse Read " 317227.65
" Stride read " 296686.93
" Random read " 290240.55
" Mixed workload " 1144392.45
" Random write " 1160763.52
" Pwrite " 1207724.31
" Pread " 670862.85
" Fwrite " 1173588.23
" Fread " 683215.71
DD:
Write test, large file to blow cache:
dd if=/dev/zero of=test bs=1M count=32000
33554432000 bytes transferred in 22.794896 secs (1472015139 bytes/sec)
Read test (ARC cached files will show speeds of over 3GB/s, this run is typical for uncached reads):
% dd if=test of=/dev/zero bs=1M count=32000
33554432000 bytes transferred in 44.108189 secs (760730210 bytes/sec)
Strangely this is almost exactly 1/2 the expected speed, 180*4=720MB/s, and it is highly consistent (+/- 100MB/s).
Write speeds monitored using 'zpool iostat 1' during testing show peak throughput at about 1.4 GB/s as expected, and stay consistently high:
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
pool_01 40.3G 43.5T 0 1.41K 0 1.41G
Read speeds occasionally peak around 1GB/s, but average around 700MB/s:
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
pool_01 63.3G 43.4T 1.06K 0 738M 0
SCRUB:
Running a scrub operation on the pool and watching throughput using iostat shows the array is capable of the full 1.44GB/s and consistently gets close to this during the scrub:
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
pool_01 63.3G 43.4T 1.41K 0 1.41G 0
It looks to me like the hardware is fully capable of running at the expected 1.4 GB/s, and writes generally do get there, but reads appear to be running at half speed unless I test using scrub which does show the full expected speed.
The pool has 0% fragmentation:
% zpool list pool_01
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
pool_01 43.5T 32.0G 43.5T - - 0% 0% 1.00x ONLINE /mnt
I've been reading posts and trying everything I can think of for a week and I'm out of ideas! Any help with my methodology, troubleshooting, or correcting my expectations or understanding would be most appreciated!
Thanks!
[1] - https://documents.westerndigital.co...-sheet-wd-blue-pc-hard-drives-2879-771436.pdf