[RESOLVED - by not using 3ware] Slow ZFS performance on 3ware 9650SE Controller

[Aaron Ryan]

I am new to freenas, and have been wanting to move off of some of our older Netapp's, especially for backups and development, and maybe even production if I can get the performance and reliability I need.

We have an older Supermicro system which has been freed up and so I wanted to try out Freenas

Here is the specs of the system:
Supermicro X7DCL MBD-X9SCM-O
8GB Memory
Intel Xeon L5410 E3-1230 V2 @ 3.30GHz
3Ware 9650SE-16M (all drives set to JBOD)
7x 1TB SATA WD BLACK (WD1001FALS) - as an alternative set of disks
7x 4TB SATA WD RED (WD40EFRX)


iperf tests over 1GE shows 942Mbps between the 2 servers, so network is fine.

I tried running the following performance script against the drives:
ftp://ftp.sol.net/incoming/solnet-array-test-v2.sh

Which shows moderate speed from each of the spinning disks, at about 150MB/s
and 250MB/s for SSDs.

I created a "/mnt/disk1" with the 7x 4TB with RAIDZ2, both with and without L2ARC and ZIL with same performance.

NFS test: when I try to copy files to the server via NFS, the bandwidth maxes out at about
28MB/s (= 224Mbps).

gstat during the copying I see 99% busy across the 7 disks

I'm not sure why it is so slow. I'm thinking about moving to a newer Motherboard/CPU incase there is a bottleneck in there.

Edit: Jun 16, 2015 [RESOLVED]
In the end it looks like there is a performance bottleneck with FreeNAS and 3Ware 9650 controllers
I ended up moving to a SAS2008 based controller, specifically a Dell H200 flashed to LSI 2911-8i IT mode version 16., and now I'm seeing 325MB/s transfers locally and near 1Gbps NFS over GigE.

 

[Aaron Ryan]

I enabled the rsync service, and am able to get 568Mbps transferring files via rsync.

 

[Aaron Ryan]

I just ordered a Supermicro MBD-X9SCM-O motherboard and will be using a Xeon E3-1230 V2 @ 3.30GHz and 8GB RAM.
It'll also help having the 2xSATA3 ports on the motherboard to fully use the throughput of the SSD drives.
I will see if this helps performance.

Edit: 6/14/2015
Ordered a IBM M1015 RAID controller too incase the performance issues are related to the 3ware controller.

 

[depasseg]

Hi Aaron - Welcome. You might want to do a little reading. There is a lot posted about NFS performance. There's a great sticky on the topic. Also read about the write performance of RAID-Z2. It's close to the max performance of a single drive, which is what you are seeing.

Also, since you are ordering more HW, I'd suggest either planning to buy more RAM or nixing the idea of using the L2ARC.

 

[Aaron Ryan]

I'm only seeing 28MB/s (224Mbps) on NFS that's way under a single drive write performance. I can get close to full 1Gbps writing sequentially to a single drive, 150MB/s = 1.2Gbps.
Anyway, something is definitely not right. Is there a NFS performance optimization checklist, I'm wondering if there is something really simple I'm overlooking.

I'll keep reading other posts, but any guidance on where to look would be greatly appreciated.

Thanks!

-Aaron

 

[depasseg]

What kind of write speeds can you get locally to your pool (not using NFS)?

Look into sync writes.

And slow NFS.

 

[SweetAndLow]

Run the same test with the same dataset locally on freenas as you do remotely over nfs. This will give you performance numbers that actually make sense.

 

[Aaron Ryan]

Run the same test with the same dataset locally on freenas as you do remotely over nfs. This will give you performance numbers that actually make sense.

How do you recommend I performing this test.

I tried rsyncing over a 8.5GB file from /mnt/disk2 (7x1TB RAIDZ2) to /mnt/disk1 (7x4TB RAIDZ2)
and am seeing 154MB/s (expected throughput of one drive) but then it pauses periodically to a trickle of 10MB/s. The 7x4TB disks are running at 100% according to gstat during this pause. The final average speed is 93.49MB/s which is still much better then what I'm seeing on nfs (28MB/s).

running dd on my test mkv file i get the following:

#dd if=/mnt/disk1/t.mkv bs=64k of=/dev/null
130243+1 records in
130243+1 records out
8535647639 bytes transferred in 24.116066 secs (353940300 bytes/sec)​

353MB/sec read speed which is very nice.

-Aaron

 

[SweetAndLow]

The dd test is better test then rsync because rsync is limited by cpu not disk. Now so the same dd test over the network with nfs.

 

[Aaron Ryan]

Here are the results of the dd over nfs:

### READ from freenas
# dd if=/mnt/freenas/t.mkv of=/disk1/t.mkv bs=64k
130243+1 records in
130243+1 records out
8535647639 bytes (8.5 GB) copied, 90.6753 s, 94.1 MB/s

### WRITE to freenas
# dd if=/disk1/t.mkv of=/mnt/freenas/t.mkv bs=64k
130243+1 records in
130243+1 records out
8535647639 bytes (8.5 GB) copied, 412.412 s, 20.7 MB/s
​

The READ looks fine IMO, WRITE is terrible though.

 

[SweetAndLow]

That's because your input is the bottleneck. You need to write with the input being /dev/zero.

 

[Aaron Ryan]

The bottleneck isnt the local file (which is on a 6x2TB mdadm RAID5 volume on Ubuntu 12.04):

### READ of local file on ubuntu server
# dd if=/disk1/t.mkv of=/dev/zero bs=64k
130243+1 records in
130243+1 records out
8535647639 bytes (8.5 GB) copied, 22.4466 s, 380 MB/s ​

But out of curiosity, I tried the dd of /dev/zero to freenas:

### WRITE of 8GB of /dev/zero to freenas
# dd if=/dev/zero of=/mnt/freenas/t.mkv bs=1GB count=8
8+0 records in
8+0 records out
8000000000 bytes (8.0 GB) copied, 138.239 s, 57.9 MB/s

​

 

[SweetAndLow]

Clearly reading from local filesystem affected your performance since it is 3x better not reading from local. Now you should look at iperf data.

 

[Aaron Ryan]

Yeah, already did that , got 942Mbps for iperf.

 

[Aaron Ryan]

Here is another test I hadnt tried on the freenas, doing dd of /dev/zero locally to /mnt/disk1:

### WRITE of /dev/null locally to freenas
dd if=/dev/zero of=/mnt/disk1/t.mkv count=17000000
17000000+0 records in
17000000+0 records out
8704000000 bytes transferred in 89.617260 secs (97124148 bytes/sec)
​

which results in a writing speed of 92MB/s (736Mbps). This is on the 3ware card.

@SweetAndLow, I see you have 6x WD-RED 3TB drives on your freenas setup (from your signature)
What kind of write throughput are you getting? Do you have it setup as RAIDZ2?

-Aaron

 

[SweetAndLow]

Local reads on empty pool that is a raidz2 are 475MB/s and local writes are 450MB/s. That 3ware card is very slow if you can only write at 92MB/s and this is your problem.

 

[Aaron Ryan]

Local reads on empty pool that is a raidz2 are 475MB/s and local writes are 450MB/s. That 3ware card is very slow if you can only write at 92MB/s and this is your problem.

Wow, 450MB/s on writes!

I have a IBM M1015 on order, I will get it re-flashed to IT mode and post the results when it arrives.
It's unfortunate the 3ware is so slow, because I have several of them.

 

[Aaron Ryan]

I found I had 2 Dell H200 RAID controller not in use and was able to flash them to LSI 2911-8i IT mode firmware 16.
I also have the system setup on the new Motherboard and Xeon E3-1230 V2 CPU.

NFS dd of /dev/zero to freenas I'm now getting 113MB/s (904Mbps).

LOCAL Copy on the freenas between the 2 raidz2 volumes via rsync, and I'm seeing 324MB/s (2.592Gbps) so this is really nice.
gstat only shows the drives running at 60-80% during the 324MB/s transfer.


LOCAL dd to freenas volume, /dev/zero to /mnt/disk1 gets 100 MB/s (800Mbps) , which is really slow for local throughput.
I tried dd from a 8.5GB file on/mnt/disk1 to /mnt/disk2 and only get 72MB/s.
It seems like there is a local bottleneck when using dd.

Are you able to get your 425MB/s via dd locally on your freenas?
Top shows dd running at 100% cpu during this test, which makes me think that it may be bottlenecked by single threaded on a cpu?

-Aaron

 

[SweetAndLow]

Yes local dd speeds are in the 450MB/s range. What dd command are you using?

 

[Aaron Ryan]

Yes local dd speeds are in the 450MB/s range. What dd command are you using?

I've tried a few different options, but here is the last one I ran:

# dd if=/dev/zero of=/mnt/disk1/t.mkv ibs=1024 count=8500000
8500000+0 records in
17000000+0 records out
8704000000 bytes transferred in 84.116415 secs (103475642 bytes/sec)

​

Top shows this during the dd:

last pid: 13713; load averages: 0.56, 0.23, 0.38 up 0+01:20:45 15:34:10
60 processes: 2 running, 58 sleeping
CPU: 0.2% user, 0.0% nice, 13.8% system, 0.0% interrupt, 86.0% idle
Mem: 366M Active, 192M Inact, 6684M Wired, 633M Free
ARC: 5400M Total, 451M MFU, 4813M MRU, 22M Anon, 30M Header, 84M Other
Swap: 32G Total, 32G Free

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
13666 root 1 103 0 9912K 1612K CPU7 7 0:39 100.00% dd
​