Notes on Performance, Benchmarks and Cache.

Joined
May 27, 2011
Messages
566
Hello everybody, I'm seeing a growing collection of posts about performance and wanted to make a few general statements about how to get Accurate results.

dd is often used to read and write to a block device and will get you results. Only when used properly will you get accurate results.

what is the basic structure of the dd command
Code:
dd if= of= bs= count=


if: is the input file, dd copies data from the input file
of: is the output file
bs: the size of each read or write.
count: the number of times to read or write.


Here are some Bad examples for Writing.

Code:
dd if=/dev/zero of=tmp.dat bs=24 count=50k


Why is this bad? the block size is way to small. normal hard disks address storage in 512 byte chunks, if we read or write in 24 byte chunks we are instead of writing a complete chunk. we write 24 bytes and leave, then we read the 512 byte chunk, and write the next 24 bytes after the first 24 bytes to another spot on the disk and continues.

So what do we do? We need to increase the block size. we could figure out exactly what the minimum block size would be for the pool, but that's a waste of time. we just need to pick a sufficiently large number such that nearly every write will be full. there may be one write per many thousands of writes that is not full, but that will result in something not statistical measurable.

Code:
dd if=/dev/random of=tmp.dat bs=2048k count=50k


Why is this bad? /dev/random is very taxing for the cpu, and this impacts performance significantly. the results will be very low as more time is spent generating the data than writing it.

So what do we do? use /dev/zero, it has extremely low overhead.

Code:
dd if=/dev/zero of=tmp.dat bs=2048k count=512


Why is this bad? this is way harder to see. 512 * 2048k is about 1 GB. sounds big right? nope. ZFS is magic, and you've got nearly all of your memory used for cache. if you have 8 GB of ram like most of us, that 1 GB is easily crammed in memory with out even touching the disk. you'll get very inacurate write results.

So what can we do? increase it to such a level that the cache will not be a large factor to the benchmark. 50k is a good number.

so where does that leave us?

Code:
dd if=/dev/zero of=tmp.dat bs=2048k count=50k


this will write 100 GB to the current directory and then report the speed (eventually)

The same is mostly true for reading, you need to read a sufficiently large file in large chunks. we can modify the command to read the file we just created and dump it in /dev/null

Code:
dd if=tmp.dat of=/dev/null bs=2048k count=50k


I really hope this helps you guys, if you have questions please let me know.

if you have compression enabled, writing all these zero's will give you bad results, i'll cover how to get around this if people need it.
 

Paktas

Dabbler
Joined
Jun 19, 2011
Messages
25
write/read speed with dd

If anyone's interested (for compare) what I get on my test platform with 4GB memory, 2x250 GB mirrored drives (not sure of the make) on ZFS. Took me ~20 minutes for writing and ~12 minutes for reading 100 GB file.

Write:
Code:
 dd if=/dev/zero of=/mnt/users/tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 1253.095241 secs (85687168 bytes/sec)


Read:
Code:
dd if=/mnt/users/tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 743.023982 secs (144509713 bytes/sec)
 

ProtoSD

MVP
Joined
Jul 1, 2011
Messages
3,348
I'll bite. 5 disk raidz2, 4GB RAM (Full specs below in signature). ~23 minutes to write, ~18 to read. I could probably tweak it, but hasn't been a priority for me right now.


Write:
Code:
dd if=/dev/zero of=/mnt/tank/tmp.000 bs=2048k count=50k
51200+0 records in
51200+0 records out

107374182400 bytes transferred in 1426.115520 secs (75291364 bytes/sec)


Read:
Code:
dd if=/mnt/tank/tmp.000 of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out

107374182400 bytes transferred in 1078.267002 secs (99580329 bytes/sec)


UPDATE: BIG improvement in READs making changes in this thread, 7 minutes vs. 18:
http://forums.freenas.org/showthread.php?1226-Hyper-threading-and-Prefetch&p=4950#post4950

Note: ATOM processor
 

survive

Behold the Wumpus
Moderator
Joined
May 28, 2011
Messages
875
Hi guys,

Here's what I'm getting with the r6929 nightly build:

Write:

filer# dd if=/dev/zero of=/mnt/zpool0/tmp.000 bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 239.880188 secs (447615884 bytes/sec)

Read:

filer# dd if=/mnt/zpool0/tmp.000 of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 202.787249 secs (529491785 bytes/sec)

Here are my system details:

FreeNAS 8.0.X - 6x Samsung F3 HD103SJ 7200RPM Drives - ZFS RAIDz
Gigabyte GA-MA790GP-DS4H - AMD Phenom II 940 Quad-Core 3.0Ghz
4 X 2GB OCZ DDR2 800 - Intel Pro 1000/PT Quad-Port Gig-E

-Will
 

Milhouse

Guru
Joined
Jun 1, 2011
Messages
564
Performance results for low-powered (Atom-class) hardware:

FreeNAS 8.0.1-BETA3
HP N36L (AMD Neo 1.3Ghz dual-core)
4GB RAM, vfs.zfs.prefetch_disable=0
LSI 9211-8i (IT firmware) and IcyDock MB994SP-4S 4-in-1 2.5" backplane
8 disks, 2 vdevs with 4K forced, 1 zpool: 4x2TB 3.5" Samsung HD204UI RAIDZ1 + 4x500GB 2.5" Samsung HM500JI RAIDZ1

Code:
freenas# dd if=/dev/zero of=/mnt/share/tmp.000 bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 505.039819 secs (212605380 bytes/sec)

Code:
freenas# dd if=/mnt/share/tmp.000 of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 373.910194 secs (287165700 bytes/sec)


Enabling read prefetch in boot/loader.conf (vfs.zfs.prefetch_disable=0) is likely to make a big impact on performance for those with 4GB or less RAM.
 

jenksdrummer

Patron
Joined
Jun 7, 2011
Messages
250
My results:

dd if=/dev/zero of=/mnt/RAIDZ1/tempfile.000 bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 729.398567 secs (147209204 bytes/sec)

dd if=/mnt/RAIDZ1/tempfile.000 of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 484.245441 secs (221735040 bytes/sec)


Not too shabby...

System spec:
HP Proliant Microserver
8GB Kingston PC10600E RAM (ECC)
4x WD20EARS, no jumpers
FreeNAS 8.0.1-BETA3 Build 6929
ZFS RAIDZ1, all 4 drives, "force 4K"
 

sjieke

Contributor
Joined
Jun 7, 2011
Messages
125
My results for an Atom D525 1.8GHz (Dual Core), 4GB RAM and 4 x WD20EARS (2TB) in raidz

dd if=/dev/zero of=/mnt/tank/test100G.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 947.502021 secs (113323434 bytes/sec)

dd of=/dev/null if=/mnt/tank/test100G.dat bs=2048k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 589.962676 secs (182001653 bytes/sec)

FTP Speeds (using 10G file)
Read: 60 MB/s
Write: 57 MB/s

If I disable hyperthreading (going from 4 threats to 2) in the BIOS FTP speeds increase:
Read: 75 MB/s
Write: 67 MB/s

But with CIFS I only get about 25MB/s both read and write. Does somebody know how I could get better CIFS speeds or is this the expected speed for my hardware?

Also I must mention that the network traffic is very bumpy for both read and write.
 

tropic

Dabbler
Joined
Jul 6, 2011
Messages
43
This was a home lab hypervisor box until I took out the RAID card and installed a few HDDs last weekend.

FreeNAS 8.0.1b4
MB: Intel DH67BL
CPU: Core i7-2600K
RAM: 16GB Kingston DDR3 1366
HDDs: 2 x HD103SJ & 3 x HD204UI
NICS: Intel 82578DC (onboard) & Intel 82574L (CT)


2 x Samsung F3 HD103SJ (ZFS striped)

vault# dd if=/dev/zero of=/mnt/public/tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 592.789290 secs (181133810 bytes/sec)

vault# dd if=/mnt/public/tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 499.530873 secs (214950043 bytes/sec)


3 x Samsung F4 HD204UI (RAIDZ)

vault# dd if=/dev/zero of=/mnt/private/tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 428.605905 secs (250519606 bytes/sec)

vault# dd if=/mnt/private/tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 497.290285 secs (215918520 bytes/sec)
 

pauldonovan

Explorer
Joined
May 31, 2011
Messages
76
Here's another Atom system. D525, 4GB RAM, 2x2TB WD20EARS in a ZFS mirror, 4k override (ashift=12):

Code:
freenas# dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 1541.201361 secs (69669146 bytes/sec)


Code:
freenas# dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 1451.980501 secs (73950155 bytes/sec)


and then with vfs.zfs.prefetch_disable=0:

Code:
freenas# sysctl vfs.zfs.prefetch_disable=0
vfs.zfs.prefetch_disable: 1 -> 0
freenas# dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 822.639096 secs (130524045 bytes/sec)


So my writes and reads are significantly slower than other similar Atom systems. My uneducated guess is that having to write to both drives in the mirror is somehow slower that writing the parity checks to a third drive in RAIDZ1. Not sure.

And I'm not convinced by enabling prefetch (turning off the disable). Yes, it boosts performance in a test like this, but I'm not sure how much time it will waste filling buffers when doing lots of random access on small files.

Paul
 

KingsX

Cadet
Joined
Jul 17, 2011
Messages
3
Asus E35M1-I Deluxe Fusion, 8GB RAM, 9x2TB Seagate Barracuda Green in RAIDZ1.

Code:
nas# dd if=/dev/zero of=/mnt/NAS-Storage/tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 390.461673 secs (274992886 bytes/sec)


Code:
nas# dd if=/mnt/NAS-Storage/tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 347.368826 secs (309107134 bytes/sec)
 
M

mirkoj

Guest
hey guys I coulduse some help... I tried same thing but I'm getting that "tmp.dat is read only file system"?
it works fine when I use shared volume frmo other comps. any ideas?
I'm trying to figure out why is it so slow. in freenas7 there were couple options to get rid of those network spikes but I can't find how to setup same thing in freensa8 and it is working a lot slower than with old version.
so now trying to figure out where is bottle neck.
 
Joined
May 27, 2011
Messages
566
hey guys I coulduse some help... I tried same thing but I'm getting that "tmp.dat is read only file system"?
it works fine when I use shared volume frmo other comps. any ideas?
I'm trying to figure out why is it so slow. in freenas7 there were couple options to get rid of those network spikes but I can't find how to setup same thing in freensa8 and it is working a lot slower than with old version.
so now trying to figure out where is bottle neck.

sounds like you're trying to write somewhere you're not allowed too. go somewhere you are allowed to write and try again.
 
M

mirkoj

Guest
hehe so obvious. anyway here are results:

read:
dd if=/dev/zero of=/mnt/users/tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 775.135699 secs (138523077 bytes/sec)

write:
dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 586.730636 secs (183004220 bytes/sec)


config:
HP proliant G5, quad Xeon 2.0Ghz, 4gbram
highpoint rocket raid card
5x1Tb WesternDigital hdd, black edition (5drivers attached on 3 different ports on card with esata), they are all setup as single JBOD in raid bios and crated raidz2 pool in freenas8.


Attached with gigabit card to gigabit switch.
But transfer rates are rather low, 60-80Mb per sec with big files and going down to 10-20Mb/sec with small files.
Any ideas what could be settings that I missed to speed things up?

btw is there raidz3 with frenas8? I had options only for raidz1 and raidz2?
Also in freenas7 there was option to force 4k, couldn't see that here.
And finally, there was some settings to help with those spiking when transferring over network. But now there is no LAN transfer graph at all, or I can't find it to see if there is similar problem now.
Thanks

P.S.
Just done some coping again and watching windows network monitor.. it is spiking all over the graph, even big big single files.
And looks like memory on server is maxed to all 4gb. Could that be a bottle neck as well?
 
Joined
May 27, 2011
Messages
566
Attached with gigabit card to gigabit switch.
But transfer rates are rather low, 60-80Mb per sec with big files and going down to 10-20Mb/sec with small files.
Any ideas what could be settings that I missed to speed things up?
small files suck. there is general overhead for any file transfer, you're seeing it when you move a lot of tiny files, you can tar them up, move them, then untar them.

btw is there raidz3 with frenas8? I had options only for raidz1 and raidz2?
nope, not yet, the version of zfs that FreeBSD uses does not support raidz3.

And finally, there was some settings to help with those spiking when transferring over network. But now there is no LAN transfer graph at all, or I can't find it to see if there is similar problem now.
Thanks

P.S.
Just done some coping again and watching windows network monitor.. it is spiking all over the graph, even big big single files.
And looks like memory on server is maxed to all 4gb. Could that be a bottle neck as well?

that's your memory filling, becoming filled and having to ask the sender to back off as it has no place to store the files. there are some ways to smooth it out but i have yet to find the time to dig into them and post about them.
 
M

mirkoj

Guest
Just wondering if adding 4more gb of ram totaling at 8 would help with transfers as well?
 
Joined
May 27, 2011
Messages
566
doesn't look like it. small files will always be bad, and your speeds otherwise are pretty decent. maybe a little but i wouldn't count on it.
 

Loofy

Cadet
Joined
Jul 25, 2011
Messages
1
My Config:

FreeNAS v8.01 beta4
[2 x (ZFS RAIDz on 3x 2TB drives) --> (~7.2TB)
ASUS P8H67-M Pro USB3 Motherboard (6x SATA ports)
Intel Core i3 2100T 2.5Ghz Low Power CPU (35W)
2 x 4GB Kingston DIMM DDR3 1333MHz]
[LDLC Danube Case + Seasonic S12II-430W Power Supply
6 x SEAGATE Green ST2000DL003 2TB 5900RPM, SATA 6.0Gb/s, 64MB Cache]
[Consumption ~80 watts]

Code:
freenas# dd if=/dev/zero of=/mnt/Public/tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 503.478824 secs (213264545 bytes/sec)


Code:
freenas# dd if=/mnt/Public/tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 458.083085 secs (234398924 bytes/sec)
 

Milhouse

Guru
Joined
Jun 1, 2011
Messages
564
My Config is in my signature :p

Consider what happens when you change your signature - all of these results then become meaningless. If you are discussing results or experience obtained with specific hardware, don't use your signature to reference that hardware as your signature can change over time while these results will not.
 

Stenull

Dabbler
Joined
Aug 22, 2011
Messages
45
FreeNas 8.0.2
Asus Blitz extreme Motherboard
3*500GB + 1*2TB HDDs in RAIDZ
8GB Memory DDR3 1333MHz
Dual-Core CPU E6600 @ 3.06GHz

Write: 198 MB/s
Read: 263 MB/s

Code:
dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 540.728841 secs (198573063 bytes/sec)


Code:
dd if=tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 408.045882 secs (263142424 bytes/sec)
 
Top