Low Read/Write Speeds on custom FreeNAS box

[Varun Chandak]

Hi there,

I am facing a very acute problem of slow read/write on my custom FreeNAS configuration box (config in signature).

I have created 2 NFS datasets, namely /mail and /archive. Both have same parameters and settings and are working properly.

Output of /proc/mounts is:

Code:

192.168.0.XX:/mnt/NAS/mail /mailstore nfs rw,noatime,nodiratime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0.XX,mountvers=3,mountport=824,mountproto=tcp,local_lock=none,addr=192.168.0.XX 0 0
192.168.0.XX:/mnt/NAS/archive /archive nfs rw,noatime,nodiratime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.0.XX,mountvers=3,mountport=824,mountproto=tcp,local_lock=none,addr=192.168.0.XX 0 0

Output of /etc/fstab is:

Code:

192.168.0.XX:/mnt/NAS/mail   /mailstore   nfs   rw,bg,vers=3,tcp,timeo=600,rsize=32768,wsize=32768,hard,intr,nodiratime,noatime     0 0
192.168.0.XX:/mnt/NAS/archive   /archive   nfs   rw,bg,vers=3,tcp,timeo=600,rsize=32768,wsize=32768,hard,intr,nodiratime,noatime     0 0

I have mounted both of these 2 partitions on 2 identical VirtualBox VMs.

Configuration of Base Machine:

Code:

Intel(R) Core(TM) i3-3210 CPU @ 3.20GHz (4 Processors)
16 GB RAM
/dev/sda: 500.1 GB (1 VM Installed)
/dev/sdb: 320.1 GB (1 VM Installed)

df -h:
Filesystem  Size  Used Avail Use% Mounted on
/dev/sda5  149G  6.9G  135G  5% /
tmpfs  7.8G  608K  7.8G  1% /dev/shm
/dev/sda3  294G  251G  29G  90% /MasterVM
/dev/sdb1  294G  251G  29G  90% /SlaveVM
/dev/sda1  485M  65M  395M  14% /boot

Configuration of VM:

Code:

MasterVM:
CPU: Intel(R) Core(TM) i3-3210 CPU @ 3.20GHz (1 Processor)
RAM: 6GB
NIC: Intel PRO/1000 MT Server (82545EM)

Code:

SlaveVM:
CPU: Intel(R) Core(TM) i3-3210 CPU @ 3.20GHz (2 Processor)
RAM: 6GB
NIC: Intel PRO/1000 MT Server (82545EM)

Whenever I am doing Read/write speeds using dd command, I am getting not more than ~24MB/s speeds (+6 to 7 seconds), which is quite low for NAS, I suppose, which is hampering my production environment.

The NAS usage is mainly for reading/writing/listing small files (mailing purposes; thousands and millions, maybe), and not more than 10-15 MB files.

On NAS I have enabled autotune as I don't want to play around the sysctls lest I crash my system.

What more should I do to increase the read/write speeds so that I can get around ~3 seconds for time for the dd command to execute.

NOTE: Every physical connection is Gigabit. Also, I have not installed VirtualBox Guest Additions on the VMs. Shall I do that ?

PS: the dd command I executed:

Code:

time dd if=/dev/zero of=/mail/testfile bs=16k count=16384

 

[dlavigne]

Did you get any headway with this?

 

[Varun Chandak]

not yet.. tried various combinations of block size, or rsize wsize etc...

 

[joeschmuck]

The one thing I noted was your drives are at 90% capacity. This is a problem for FreeNAS because it changes the way it writes data to the drives in order to optimize it. Drop below 90% and reboot, see how that works.

 

[cyberjock]

Actually, the cutoff for how ZFS works is 95% on FreeBSD. But, you shouldn't fill a pool >80% for performance reasons.

To make matters worse, the user is doing FreeNAS in a VM, and his CPU doesn't support VT-d per Intel's sheet: http://ark.intel.com/products/71053/Intel-Core-i3-3210-Processor-3M-Cache-3_20-GHz

So why is *anyone* surprised that things are not working well? Anyone? Anyone?


My recommendation is to stop and read our stickies and manual. This error should not have occurred and nobody should be the least bit surprised by the end-result.

 

[joeschmuck]

95%, I'll try to remember that. I saw all the VM references and didn't think the CPU couldn't handle it, maybe I should stop posting, not doing my diligent research before speaking.

 

[cyberjock]

Well, it was 90% for Solaris. This is one of those things that's unique to FreeBSD and if you don't look at the code you wouldn't know better. :P

 

[joeschmuck]

:p

You are too kind to me. Maybe I need to stay the night in a Holiday Inn Express :D

 

[Varun Chandak]

Actually, the cutoff for how ZFS works is 95% on FreeBSD. But, you shouldn't fill a pool >80% for performance reasons.

To make matters worse, the user is doing FreeNAS in a VM, and his CPU doesn't support VT-d per Intel's sheet: http://ark.intel.com/products/71053/Intel-Core-i3-3210-Processor-3M-Cache-3_20-GHz

So why is *anyone* surprised that things are not working well? Anyone? Anyone?


My recommendation is to stop and read our stickies and manual. This error should not have occurred and nobody should be the least bit surprised by the end-result.

sorry that i haven't mentioned... but freenas is working on a standalone machine.. these 2 servers are using the nfs mounts in their system....


anyhow.. I'll check for that processor and try to get an updated one if that doesn't support VTd

 

[jgreco]

The fact that the alg changes at 95% doesn't make 95% reasonable. Passing 80% is still going to be bad... slowly degrading.

 

[cyberjock]

The fact that the alg changes at 95% doesn't make 95% reasonable. Passing 80% is still going to be bad... slowly degrading.

Correct. Things slowly decline at 80%+, and things rapidly decline at 95%+.

 

[jgreco]

No, not slowly like that. It is a matter of pool age and existing fragmentation. A new pool will let you breeze past 80, kinda, but just try it on a production pool that's seen a lot of traffic. 80% is the point at which you will slowly see performance degrade as time passes.

A pool that is 80% full on day 1 may lose 50-75% of its performance if it sees constant traffic, even if it never hits 81%, after a year of use. That's what I mean by slowly degrading.

 

[cyberjock]

and that's what I meant too. :P

 

[joeschmuck]

Glad to see everyone is on the same page now ;)