Horrible write performance

[unseen]

My recently commissioned freenas server is running 11.0-U3. It is based on the following hardware:

Case: In Win MS04-01 with 265W PSU
Motherboard: ASRock Rack E3C224DI
CPU: i3-4150
RAM: 2 x 8GB ECC DDR3
Hard Disks: 4 x WD Red 3TB

I have both of the server's network interfaces connected to a gigabit switch and a client PC running Linux connected to another gigabit switch that is in turn, connected to a port on the same gigabit switch as the freenas box.

My four disks are placed in a RaidZ1 pool.

If I run: dd if=/dev/urandom of=a_file bs=2048 count=10000k on the NAS to create a_file in a pool share, I get just over 100MB/s write speed.

If I run iperf from the client with the freenas as the server, I get 836Mb/s transfer speed.

If I try to copy files from the client to a cifs share on the freenas, I get less than 20MB/s. It's actually slower than my old Marvel based QNAP!

What do I need to change to make this work at the speed that the NAS and network are obviously capable of?

 

[rs225]

Try setting sysctl tunable net.inet.tcp.recvspace=262144

 

[unseen]

Try setting sysctl tunable net.inet.tcp.recvspace=262144

I disabled my second network interface, just to make things simpler.

With the receive buffer sized at the default 64K, I get 18MB/s write speed.
Increasing the buffer size to 256K gave 21.5MB/s.

I'm scratching my head here...

 

[unseen]

Hmmm....

I changed the client PC to something with a bit more muscle (an i7 powered laptop). Now I'm getting about 43MB/s copying from a USB3 disk to the freenas. Still somewhat underwhelming, but at least I'll be able to restore my backups to the NAS in two days instead of a whole week.

 

[rs225]

You can also try net.inet.tcp.cc.algorithm=cubic but that was mainly for wi-fi performance, and I don't remember offhand if cubic is available by default.

 

[joeschmuck]

Are you moving lots of small files are large backup files? Your iperf results, while not stellar, they are respectable.

Also, what is the transfer rate of the USB3 hard drive to your computer? 43MB/sec sounds pretty good if it's a lot of small files, it sounds pretty bad if it's really large non-fragmented files.

 

[unseen]

Before I switched to using a more powerful laptop to restore the backup, I could copy from the USB3 drive to the laptop at over 100MB/s on a 2GB file. Once the restore has finished, I'll see how fast I can copy a large file to the internal SSD on the current laptop.

The backup consists of a mixture of files, some many GB in size, some a few KB. The current part of the backup is about 4TB and has seven more hours to go at 42.5MB/s average.

Looking at the output of top shows that the NAS is spending most of its time sleeping:

last pid: 61798; load averages: 0.50, 0.33, 0.21 up 0+19:46:29 09:31:15
67 processes: 1 running, 66 sleeping
CPU: 2.3% user, 0.0% nice, 0.8% system, 0.2% interrupt, 96.8% idle
Mem: 24M Active, 707M Inact, 14G Wired, 497M Free
ARC: 13G Total, 12M MFU, 12G MRU, 11M Anon, 36M Header, 611M Other
Swap: 8192M Total, 86M Used, 8105M Free, 1% Inuse

Something isn't right...

 

[SweetAndLow]

Don't use urandom for testing. It's slow and relies on the cpu. Next use iperf to test your network. Finally anything with the word usb is so and inconsistent so don't use that.

What are your local read and write speeds on freenas? Make sure to disable compression when doing your dd test from /dev/zero

Sent from my XT1096 using Tapatalk