Notes on Performance, Benchmarks and Cache.

macnlz

Cadet
Joined
Jul 11, 2013
Messages
5
You know, my FreeNAS server with no drives.. Xeon e3-1230v2, 32GB of ECC RAM, dual GB Intel LAN, uses less power than your machine at the wall. Mine sat at a solid 35watts with powerd off. Plus, since I have AES-NI I do use encryption for one of my zpools. :p


I should note that this was not my first choice in hardware. :(

I had a nice box spec’d out, with ECC and ultra-low TDP processor. Alas, my wife put in her veto when she saw the price tag (about $1.9k), and I had to find ways to get the price tag down to about $1k. Her argument: for that price, I could buy a new hard drive once a year for 20 years in a row and copy everything I’ve currently got over… counter-arguments about how that doesn’t save my data from bit rot were destroyed with a simple “none of your data is that important”. :rolleyes:

My goals:
  • small, for home use (Mini-ITX or DTX since I decided on the Node 304 early on)
  • primarily to be used with WiFi (802.11n, later perhaps 802.11ac)
  • RaidZ2 (6 drives)
  • Enough usable space for 2 TimeMachine backups and all of my current data, preferably more (4TB available space would be cutting it very close)
  • full disk encryption
  • preferably on the fly compression
  • preferably low energy consumption
  • preferably ECC RAM
As you can see, I could have gotten closer to my goals if money hadn’t been an issue. I ended up going for a 12TB (8TB available, don’t use over 80% on ZFS = 6.4TB effective) config, and decided that $25 more for an 80plus Gold power supply was worth it, because my system wasn’t going to be drawing nearly as much power as the better power supplies provide (try finding any highly rated power supplies in the 200W range!).

At least the A4 has AES-NI (all modern processors seem to have it), so full disk encryption probably sucks by about a factor of 10 less than it would otherwise. Since WiFi is such a bandwidth bottleneck, even this little $46 processor is sufficient (though not ideal).

Most prices listed below are Newegg or Amazon.

Code:
Drive options:
 
(6x) WD Red 3 TB    6x $145.99 = $875.94 (12TB available)
(6x) WD Red 2 TB    6x $109.99 = $659.94 (8TB available)
(6x) WD Red 1 TB    6x $79.99 = $479.94 (4TB available)
 
 
High End Combo with ECC:
 
Fractal Design Node 304                      $89.99
Supermicro X9SPV-M4-3UE (Core i7 @17W TDP)   $619.99
(2x) Kingston KVR16LSE11/8 (16GB)            2x $86.68
SeaSonic SS-400FL2 (Platinum)                $119.99
 
= $1477.33 (4TB available)
= $1657.33 (8TB available)
= $1873.33 (12TB available)
 
 
Mid Range Combo ECC:
 
Fractal Design Node 304        $89.99
Intel® Server Board S1200KPR   $159.99
Intel Core i3-3220T (35W)      $125
LSI Logic SAS9211-4i           $175
Kingston KVR1333D3E9SK2 (8GB)  $88.99
SeaSonic SSR-450RM (Gold)      $74.99
 
= $1193.90 (4TB available)
= $1373.90 (8TB available)
 
 
Low End Combo (no ECC):
 
Fractal Design Node 304          $89.99
ASRock MB FM2 FM2A85X (AMD)      $99.99
AMD A4-4000 (65W)                $45.99
Corsair CMV8GX3M2A1333C9 (8GB)   $67.99
Corsair CX430M (Bronze)          $49.99
 
= $833.89 (4TB available)
= $1013.89 (8TB available)
= $1229.89 (12TB available)
 

purduephotog

Explorer
Joined
Jan 14, 2013
Messages
73
SUPERMICRO MBD-X10SLL-F-O uATX Server Motherboard LGA 1150 DDR3 1600
Intel Pentium G3220 Haswell 3.0GHz LGA 1150 54W Dual-Core Desktop Processor Intel HD Graphics BX80646G3220
24gb DDR3 ECC/unbuffered ram (KVR1333D3E9S/8G) (3x8gb, couldn't afford to drop another 90$ on a 4th stick... waiting)
Perc H310 operating in passthru mode / flashed to IT via several threads.
9x Seagate 3tb drives, 7200 RPM (one drive to mainboard SATA connector)

Write:
107374182400 bytes transferred in 217.778554 secs (493042958 bytes/sec)
=~470mb/sec
Read
107374182400 bytes transferred in 110.819065 secs (968914350 bytes/sec)
=~924mb/sec

(hrmm... seems...odd... on that one)

~XX watts idle, no drive spin down (need to test still)
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Might want to check your hard drive temps. 7200RPM hard drives are REALLY hard to keep below 40C based on prior users on the forum.
 

enemy85

Guru
Joined
Jun 10, 2011
Messages
757
FreeNas 9.1 64bit
AMD E-350 APU (1,6Ghz, Dual-Core)
4GB Ram
2 x 2TB Seagate Barracuda Green (5900 rpm) ST2000DL003-9VT166 mirrored (512byte logical/physical sector size)
Code:
PREFETCH DISABLED
 
dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in                                                         
51200+0 records out
107374182400 bytes transferred in 1264.529978 secs (84912327 bytes/sec)
 
dd if=tmp.dat of=/dev/null bs=2048k count=50k                               
51200+0 records in                                                         
51200+0 records out                                                         
107374182400 bytes transferred in 1300.226967 secs (82581107 bytes/sec)
 
dd if=/dev/zero of=tmp.dat bs=512k count=20480 conv=sync                   
20480+0 records in                                                         
20480+0 records out                                                         
10737418240 bytes transferred in 124.770277 secs (86057501 bytes/sec)
 
dd if=tmp.dat of=/dev/null bs=512k count=20480 conv=sync                   
20480+0 records in                                                         
20480+0 records out                                                         
10737418240 bytes transferred in 142.411551 secs (75397102 bytes/sec)
 
PREFETCH ENABLED
 
dd if=/dev/zero of=tmp.dat bs=2048k count=50k                               
51200+0 records in                                                         
51200+0 records out                                                         
107374182400 bytes transferred in 1276.198713 secs (84135943 bytes/sec)
 
dd if=tmp.dat of=/dev/null bs=2048k count=50k                               
51200+0 records in                                                         
51200+0 records out                                                         
107374182400 bytes transferred in 689.204832 secs (155794297 bytes/sec)
 
dd if=/dev/zero of=tmp.dat bs=512k count=20480 conv=sync                   
20480+0 records in                                                         
20480+0 records out                                                         
10737418240 bytes transferred in 133.142979 secs (80645771 bytes/sec)       
                                                
dd if=tmp.dat of=/dev/null bs=512k count=20480 conv=sync                   
20480+0 records in                                                         
20480+0 records out                                                         
10737418240 bytes transferred in 65.574658 secs (163743412 bytes/sec)


so for what i can see, even if with a short amount of Ram, is better to keep prefetch enabled, right?
 

amires

Explorer
Joined
Nov 19, 2012
Messages
66
I am trying to benchmark my freenas system using information in the first post however I am getting unrealistic results. My config is at follows :

MB : Supermicro X9SCL
CPU : Xeon E3-1240v2
HBA : IBM M1015
HDD : 3 x WD RED 3TB @ RAID-Z1
RAM : 16GB
FreeNAS 9.2.0

Now to the benchmarking :

dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 23.008162 secs (4666786608 bytes/sec)

dd of=/dev/zero if=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 10.088595 secs (10643125561 bytes/sec)

The results show that I am reading at 4.6TB/s and writing at 10TB/s which clearly are wrong and unrealistic given the fact that I have only 3 HDDs in my pool. Could someone please tell me how to correctly benchmark my system.
 

enemy85

Guru
Joined
Jun 10, 2011
Messages
757
probably u have compression set on the volume u are testing...disable it and u will find the right values! read old posts on this thread and u will find a similar situation!

EDIT
see posts #40 #41
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Disable compression or dedup.
 

amires

Explorer
Joined
Nov 19, 2012
Messages
66
probably u have compression set on the volume u are testing...disable it and u will find the right values! read old posts on this thread and u will find a similar situation!

Disable compression or dedup.

Thanks enemy85 and cyberjock. You were right I was using compression. I created a new dataset and turned off compression on it and repeated the test. The results seem to be right this time.

dd if=/dev/zero of=tmp.dat bs=2048k count=50k

51200+0 records in
51200+0 records out
107374182400 bytes transferred in 767.196707 secs (139956521 bytes/sec)

dd if=tmp.dat of=/dev/null bs=2048k count=50k

51200+0 records in
51200+0 records out
107374182400 bytes transferred in 758.289708 secs (141600474 bytes/sec)
 

Rand

Guru
Joined
Dec 30, 2013
Messages
906
Hey guys,
if anyone is interested, just ran a couple of write tests on the new drives (forgot to test read as i am primarily concerned with write speed atm).
Xeon E3-1230v3,16GB,8x Toshiba DT01ABA300 (3TB , 5940 RPM) on a Dell H310/IT

Quite a few suprises (for me, probably not for the more experienced ppl)...
-RaidZ2 Mirror en par with RaidZ3 and only 10% worse than 4x2 mirror, would have expected 4x2 to be significantly faster (although its ok, half the speed of the 8drive stripe).
O/c 4x2 Mirror has way better IOPS, about 30% quicker write on my test file set with mixed small/large files; but slower on large file copy alone for some reasons.
-3 way mirror is horribly slow - but i guess thats just 2 vdevs (ie 2 disks), matches by speed... (Read is o/c pretty fast)

dd if=/dev/zero of=a.out bs=2048k count=50000

Mirror 4x2 (via Add Device)
104857600000 bytes transferred in 191.484424 secs (547603809 bytes/sec)
104857600000 bytes transferred in 135.033563 secs (776529905 bytes/sec) (Read)
RaidZ2 2x4 (Auto for 8 drives)
104857600000 bytes transferred in 207.681178 secs (504896982 bytes/sec)
104857600000 bytes transferred in 206.143166 secs (508663964 bytes/sec)(Read)
RaidZ3 1x7
104857600000 bytes transferred in 207.856055 secs (504472193 bytes/sec)
104857600000 bytes transferred in 123.145976 secs (851490267 bytes/sec) (Read)
Stripe 1x8 (21TiB) (Added in one)
104857600000 bytes transferred in 94.661475 secs (1107711454 bytes/sec)
Stripe 8x1 (21TiB) (via Add Device)
104857600000 bytes transferred in 94.947123 secs (1104378907 bytes/sec)
Mirror 2x3 (Not Optimal warning)
104857600000 bytes transferred in 387.068334 secs (270902037 bytes/sec)
104857600000 bytes transferred in 170.492939 secs (615026057 bytes/sec) (Read)
Single Disk 1x1
104857600000 bytes transferred in 850.206175 secs (123331967 bytes/sec)
104857600000 bytes transferred in 716.095746 secs (146429581 bytes/sec) (Read)
 

IonutZ

Contributor
Joined
Aug 17, 2014
Messages
108
Can you guys help me understand why I'm getting 16s write speeds for 100gb?

dd if=/dev/zero of=./tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 16.536778 secs (6493053397 bytes/sec)
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Can you guys help me understand why I'm getting 16s write speeds for 100gb?

dd if=/dev/zero of=./tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 16.536778 secs (6493053397 bytes/sec)

It's called compression and 100GB of zeros compress REALLY REALLY well.
 

vikingboy

Explorer
Joined
Aug 3, 2014
Messages
71
These numbers are somewhat higher than I was expecting, I have compression off
X9SRL-F, e5-1620v2 (HT off), 32GB ECC, 10*4TB WD RED in RAIDZ2, 2*9211-8i

dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 127.213963 secs (844044001 bytes/sec)

dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 132.921829 secs (807799465 bytes/sec)
 
Last edited:

Starpulkka

Contributor
Joined
Apr 9, 2013
Messages
179
Well heres some results as keepin same so it would be somewhat comparable to other users (as zfs will handles zeroes..)
I remember when drives was fresh test was way much faster..
Compression off
kone6 kone6 hdd amd
amd fx4100 asus 16G ecc Seagate 6x3Tb raidz2 70%full pool
dd if=/dev/zero of=/mnt/kone6/tmp.000 bs=2048k count=50k
107374182400 bytes transferred in 305.854406 secs (351 063056 bytes/sec)
dd if=/mnt/kone6/tmp.000 of=/dev/null bs=2048k count=50k
107374182400 bytes transferred in 236.514321 secs (453 985966 bytes/sec) (fast huh?)

compression off
SuperMicro X7DBE Intel Xeon Core E5430 24G ecc (or something ram) 2x ibm Serve RAID M1015 IT
superkone kone6 hdd xeon Seagate 6x3Tb raidz2 70%full pool
dd if=/dev/zero of=/mnt/kone6/tmp.000 bs=2048k count=50k
107374182400 bytes transferred in 351.475753 secs (305 495277 bytes/sec)
dd if=/mnt/kone6/tmp.000 of=/dev/null bs=2048k count=50k
107374182400 bytes transferred in 292.498880 secs (367 092628 bytes/sec)

compression off
Supermicro x10 slm-F intel G3220 16G ecc 6x3Tb raidz2 2x ibm Serve RAID M1015 IT
superkone kone6hddt Seagate 6x3Tb raidz2 70%full pool
dd if=/dev/zero of=/mnt/kone6/tmp.000 bs=2048k count=50k
107374182400 bytes transferred in 280.482278 secs (382 819846 bytes/sec)
dd if=/mnt/kone6/tmp.000 of=/dev/null bs=2048k count=50k
107374182400 bytes transferred in 287.211526 secs (373 850534 bytes/sec)

compression off
Supermicro x10 slm-F intel G3220 16G ecc 6x3Tb raidz2
g3220 kone6 levyt Seagate 6x3Tb raidz2 70%full pool
dd if=/dev/zero of=/mnt/kone6/tmp.000 bs=2048k count=50k
107374182400 bytes transferred in 289.378767 secs (371 050660 bytes/sec)
dd if=/mnt/kone6/tmp.000 of=/dev/null bs=2048k count=50k
107374182400 bytes transferred in 278.919216 secs (384 965167 bytes/sec)
 

driver7

Cadet
Joined
Sep 27, 2014
Messages
5
My system isn't into production yet I'm waiting for a sixth disk to arrive but I am messing around with it until then.
Supermicro x10slm w/ Xeon 1230v3, 16gb Kingston ECC, 5x HGST Deskstar 4tb 7200rpm HDs in RAIDZ1, onboard disk controller. Freenas 9.2.1.7

Code:
[brandon@freenas] /mnt/testvol1/media# dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 201.686057 secs (532382774 bytes/sec)
[brandon@freenas] /mnt/testvol1/media# dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 182.023111 secs (589893129 bytes/sec)


EDIT

My LSI 9211-8i controller just came in and I popped it in, flashed it and did a fresh install of Freenas on a different USB drive, settings all the same.

Code:
[brandon@nastest] /mnt/testvol4# dd if=/dev/zero of=tmp.dat bs=2048k count=50k 
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 196.580952 secs (546208477 bytes/sec)
[brandon@nastest] /mnt/testvol4# dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 162.393074 secs (661199272 bytes/sec)
[brandon@nastest] /mnt/testvol4#
 
Last edited:

leenux_tux

Patron
Joined
Sep 3, 2011
Messages
238
Wished I had done this test before I did my upgrade. I might have had something to compare against....

Writing to 10 disk RAIDz2
dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 41.738820 secs (2572525597 bytes/sec)

Reading from file
dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 21.380713 secs (5022011309 bytes/sec)
 

mjws00

Guru
Joined
Jul 25, 2014
Messages
798
@leenux_tux you need to disable compression and try again.
 

vikingboy

Explorer
Joined
Aug 3, 2014
Messages
71
Yeah, worth reading the first post in this thread that explains how to use DD to bench.
 

leenux_tux

Patron
Joined
Sep 3, 2011
Messages
238
Ha, good point :D

Results with a newly created dataset with compression disabled.

Write data
dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 236.794444 secs (453448909 bytes/sec)

Read data
dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 155.053800 secs (692496298 bytes/sec)

Much different from the last set of results I posted

Still wish I had done this test previously though, just out of interest really. Previously I had a 4 disk raidZ1 attached via a single M1015. Now I have a 10 disk RAIDZ2, spread across two M1015's (5 disks each). I also have a 4 disk RAID10, split two disks on each M1015, however, my ESXi box isn't switch on right now so the iSCSI extent is not in use and should not have made any difference to the throughput on the HBA's ??
 

vikingboy

Explorer
Joined
Aug 3, 2014
Messages
71
Results with a newly created dataset with compression disabled.

Write data
dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 236.794444 secs (453448909 bytes/sec)

Read data
dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 155.053800 secs (692496298 bytes/sec)

Looks more accurate than previous results but its interesting to note the difference in your 10disc raidz2 to mine. Both on X9 boards, with 32GB RAM and M1015 / 9211's.

i.e X9SRL-F, e5-1620v2 (HT off), 32GB ECC, 10*4TB WD RED in RAIDZ2, 2*9211-8i

dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 127.213963 secs (844044001 bytes/sec)

dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 132.921829 secs (807799465 bytes/sec)

I didn't split my array 5x5 across the two M1015s as you did, I just ran 8 + 2.
 
Top