My new NAS box, running on 8.0.2

[ProtoSD]

I just wanted to let everyone know that my board, SUPERMICRO MBD-X7SPA-H-O with the D510 processor DOES work with 8GB of RAM! My BIOS is version 1.2 (not the latest). So I'm quite happy since I accidently overlooked the documented 4GB max when I bought it.

Floren, sorry, I'm not trying to hijack your thread. I thought other people with the variations of the motherboard we have might like to know.

EDIT: The memory I got is G-Skill F2-5300CL5D-8GBSQ (~$180) from Newegg.

 

[Daisuke]

No, this is actually great! Thanks for posting the info. :)
Do a test, transfer a 8GB file and see what is the wired memory usage hitting as value. Should be around 6GB if everything is OK.

 

[ProtoSD]

Hmmm, the wired has only peaked around 1024M during an 8GB copy, mostly about 800MB. I checked my loader.conf settings and don't have anything set for memory (I used to), I'm still running 8.03-p1. I'm curious to see if a scrub will run any faster.

 

[Daisuke]

Hmmm, the wired has only peaked around 1024M during an 8GB copy, mostly about 800MB. I checked my loader.conf settings and don't have anything set for memory (I used to), I'm still running 8.03-p1. I'm curious to see if a scrub will run any faster.

Ryan, I'm currently running 8.0.4 64bits. I did a 10GB file transfer through CIFS and monitored the memory usage.
On hourly basis with heavy usage, it hits at about 2.3GB:

On the weekly basis, it hits up to 5.8GB:

On 8.0.3 it used to hit 6GB constantly, so something changed. I don't use any customized settings, everything is stock into BIOS (beside AHCI + headless) and FreeNAS.

Related to scrubbing, it is pretty intense on the Atom processor:

As you can see from the screenshot, the CPU fluctuates at 80-90% usage, server load at 4.5, memory steady at around 2390M and the swap is not used at all.

It takes a little less than 3 hours to scrub all data on my RaidZ2 6 disks array.

CPU Usage (day):

Server Load (day):

I hope this helps you getting a better idea how my system performs under stress. I would appreciate if you can post your results also, so we have a nice comparison.

 

[bdacasc]

Hey ! Great setup You got !!!
I want to copy your setup.
It would be intresting to know what is the power consumption for your currrent setup

 

[ProtoSD]

Related to scrubbing, it is pretty intense on the Atom processor:

As you can see from the screenshot, the CPU fluctuates at 80-90% usage, server load at 4.5, memory steady at around 2390M and the swap is not used at all.

It takes a little less than 3 hours to scrub all data on my RaidZ2 6 disks array.

I hope this helps you getting a better idea how my system performs under stress. I would appreciate if you can post your results also, so we have a nice comparison.

Thanks for that, I'll post some details after I do some more tweaking. How much data do you have in your pool? Less than 3 hours for a scrub would be fantastic, even 6 or 8 would be better than the 12-15 hours for mine!

It seems like I messed up some setting, my CIFS transfers were a little faster and now they suck. I get about 35-40MB/s writing, and 20-25MB/s reading. I have GigE everything.

It's funny, I was expecting there to be a P1 release for 8.0.4 and was going to wait to upgrade, but last night I gave up so I could try the new Python utilities in 8.04 (http://doc.freenas.org/index.php/Arcstat) for tuning. Now today I wake up to see an announcement for 8.0.4-P1..... oh well.

 

[Daisuke]

How much data do you have in your pool?

I have about 2TB of media and Linux development files. I'm going to upgrade to 8.0.4-P1 also and see what is changed... I feel the same way, I ran all the tests at 4AM and now I have to do them again. :D

 

[Daisuke]

Hey ! Great setup You got !!!
I want to copy your setup.
It would be intresting to know what is the power consumption for your currrent setup

Thanks. No idea, just purchase power consumption meter to find out. :)
This is an Atom proc, so the consumption is very low.

 

[Daisuke]

Hmm, I upgraded to 8.0.4-p1 and the CIFS performance is now cut in half. I can read and write at about 50MB/sec, compared to the previous stats posted into OP. I re-installed Windows 7 Ultimate, just to be sure there is nothing special related to it.

When I ran a disk test as Administrator, the results were OK for my RAID1 array:

Code:

> winsat disk -drive c
> Disk  Sequential 64.0 Read                   96.62 MB/s         6.5
> Disk  Random 16.0 Read                       2.47 MB/s          4.4
> Responsiveness: Average IO Rate              2.12 ms/IO         6.9
> Responsiveness: Grouped IOs                  8.34 units         7.4
> Responsiveness: Long IOs                     5.59 units         7.7
> Responsiveness: Overall                      46.63 units        7.1
> Responsiveness: PenaltyFactor                0.0
> Disk  Sequential 64.0 Write                  111.03 MB/s        6.7
> Average Read Time with Sequential Writes     6.977 ms           5.3
> Latency: 95th Percentile                     32.720 ms          3.0
> Latency: Maximum                             112.231 ms         7.6
> Average Read Time with Random Writes         13.346 ms          3.7
> Total Run Time 00:01:39.50

 

[ProtoSD]

I wish I could get reads & writes as fast as yours! I was getting solid 80MB/s reads earlier, but I changed something and don't remember what I did. Now I'm getting about 45MB/s read & writes. I also just finished my upgrade to 8.04-p1.

 

[ProtoSD]

Hey ! Great setup You got !!!
I want to copy your setup.
It would be intresting to know what is the power consumption for your currrent setup

I have almost the same system, with the Atom D510 and 5x 2TB disks and idling my power is about 46w and during a scrub its just under 60w if I remember correctly.

 

[Daisuke]

I wish I could get reads & writes as fast as yours! I was getting solid 80MB/s reads earlier, but I changed something and don't remember what I did. Now I'm getting about 45MB/s read & writes. I also just finished my upgrade to 8.04-p1.

The reads you see there are performed on my Windows WD5000AAKX disks, set in RAID1. After the FreeNAS upgrade, if I transfer a file from local disk to network disk (mapped to FreeNAS through CIFS) I get about 50MB/sec as transfer speed. I used to get 110MB/sec before, so something changed in a negative way. Not very happy now.

Run this command as Administrator in your Windows system, to make sure the disks are not the bottleneck:

Code:

C:\Windows\system32> winsat disk -drive c

 

[ProtoSD]

I figured out that those results were from windows. I'm not really happy with my results either, especially knowing what you used to be able to get. I know your CPU is a bit faster than mine, but I should be able to get a lot closer to what you had than I am. I need to take a break for awhile, my eyes are getting blurry and my brain getting foggy ;)

 

[Daisuke]

If anyone has a fix or knows what might be the cause of data speed loss, please let us know. Since I moved from 8.0.2, I lost half of the transfer speed I used to get.

 

[ProtoSD]

See if this works for you. I'm getting about 50MB/s writes and 60MB/s reads with peaks of 70MB/s. I don't think I've got it the best it can be yet, the speeds seem to drop off a little during the transfers, but I think I'm getting closer. Let me know what you get. My laptop disk can only do ~68MB/s, so I can't do much faster than that anyway.


For anyone else reading this, these settings are for an Atom motherboard, D510 processor, 8GB of RAM, 5 disk raidz2.


CIFS Auxilary Settings (You need to enable Advance Settings checkbox for FreeNAS "System->Settings" to see this). After enabling that checkbox, go to Services->Control Services and click on the wrench icon next to CIFS.

Code:

socket options = TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE SO_RCVBUF=2097152 SO_SNDBUF=2097152
# 65536 or 131072 seem to work ok for above values also, experiment
read raw = yes
write raw = yes
max xmit = 65536
getwd cache = yes

For the loader.conf (Tunables in GUI), depending on your RAM etc. these seem to work well at the moment:
Note: Dividing some of the large numbers by 1048576 (1MB) will give you the value in Megabytes

Code:

vfs.zfs.prefetch_disable                0                        (Setting to 1 cuts my speeds in half)

vfs.zfs.arc_max                         5074190464	         (For 8GB of RAM)

kern.ipc.nmbclusters	                10000

vfs.zfs.txg.write_limit_override	1073741824               (For 8GB of RAM)

vfs.zfs.vdev.max_pending	        1		         (8 is also a recommended value)

vfs.zfs.vdev.min_pending	        1		         (2 is also a recommended value)

kern.ipc.somaxconn	                384	                 TCP listen queue	

kern.ipc.nsfbufs	                150000	                 Sendfile buffers - Used if "sendfile" is enabled for CIFS

net.inet.tcp.inflight.enable	        1	                 TCP Bandwidth Delay Product	

net.inet.tcp.recvbuf_auto	        1		

net.inet.tcp.sendbuf_auto	        1		

net.inet.tcp.delayed_ack	        0

net.inet.tcp.sendbuf_max	        2097152

kern.ipc.maxsockbuf	                2097152

net.inet.tcp.recvbuf_max	        2097152

 

[Daisuke]

You can get rid of those options, they are default values:
read raw = yes
write raw = yes
max xmit = 65536
getwd cache = yes

Now I remember adding these settings into aux params:
socket options = TCP_NODELAY IPTOS_LOWDELAY IPTOS_THROUGHPUT

while enabling the Large RW support, sendfile2 and AIO. Right now, I get 75MB/sec writes to network disk without any custom loader.conf values. Personally, I never fiddled with that file. However the 75MB/sec write starts dropping after 1 minute of active transfer and keeps dropping up to a miserable 48MB/sec, for a 10GB file. Before I used to get about 90MB/sec writes, dropping at 75MB/sec lowest.

I also get writes to local disk at 70MB/sec steady all the way to the end. Bye bye 115MB settings I used to get before...

 

[ProtoSD]

I can tell you're really unhappy about this. This is really the first time I've looked into the performance of my system in more detail.

I'm going to disable those Samba setting you mentioned, but I also have AIO disabled. I think I saw somewhere that AIO doesn't work right on FreeBSD anyway.

Like I said, I don't think I've got it pefect yet, just better than the default settings which are terrible.

I did share these settings with another user that was getting terrible speeds and he said his speeds are fantastic now, 90+GB/s.

Did you try to set your prefetch to 0? I thought with memory over 4GB it was supposed to be 1, but when I set it to one my transfers literally are about half, which is what you're saying.

My local disk speeds are ~320MB/s, so I know I should be able to do better. I also get better speeds with medium sized files, like 500-700MB.

I'll keep testing and post back if I make any significant progress.

 

[Daisuke]

Well, I get better results with AIO enabled, no idea why.
Also, do you think is good to fiddle with loader.conf file? I always tune sysctl on my Unix boxes... but I know nothing about FreeBSD. Before I was getting top results without any tuneup changes.

Remember that I have a 64GB SSD used for cache, not sure how it affects the tuneup.

 

[ProtoSD]

You can put the settings from sysctl.conf or loader.conf into either file and they will work. Since FreeNAS 8.03 you can add settings for those files from the GUI. I just put everything into the "Loaders" section except my jail stuff which goes into the Sysctl section just for clarity.

There are a couple of settings related to cache, but I don't think they're required. The nice thing about entering the settings from the GUI is there is a "Enable" checkbox so you can test stuff and turn it off or on using the Enable checkbox. The only hassle is having to reboot to get certain settings to take effect.

Are you using LAGG?

 

[Daisuke]

No, I don't use LAGG. I don't see anywhere the Loaders section, into Settings gui.