Horrible Performance with FreeNAS as VM Storage

[fc7t]

Hi there,

I have the following setup:

HP SE326M1 Server
2 x Intel Xeon QC L5220
LSI 9211-8i HBA
64 GB DDR 3 RAM
16 x 300 GB SAS 2.5 10K HP Drives as RAID 10 (2,1 TiB)
2 x Intel XF SR2 10 GbE Adapter

When working in the VM, e.g. Windows Server 2012 R2, the performance is horrible if no data is cached in the FreeNAS RAM, so that the data is read from the RAID 10.

In the picture shown below I've copied a file of about 4 GB from a virtualized openmediavault VM to the virtualized Windows Server 2012 R2 VM.

22,5 MB/s with 16 x 300GB SAS drives in RAID 10 and 10 GbE connections???

When I copy the Windows Server 2012 R2 VM and the openmediavault VM to my QNAP TS 469U-RP NAS (4 x 2 TB WD, 7200 RPM in RAID 5) and do the copying again it shows me over 100 MB/s!

I thought that the 7200 RPM in the NAS system should be much more slower than the drives in the HP326M1.

Do you have an idea why my configuration is SO slow?

 

[m0nkey_]

How are you presenting the storage to the VM? Are you using iSCSI, NFS, etc?

 

[jgreco]

Not going to go into a big spiel right now, but by all means read the stickies on SLOG, iSCSI, and my recent thread with Kelsey Cummings, etc. Also if you want people to show interest, post a more detailed description of your system, pool, protocols, what you're using for virtualization, and everything else needed for someone to analyze your problem beyond handwavey guesses.

 

[fc7t]

Hi,

sorry for the missing details. To be more concrete:

- the protocol for presenting the LUNs to the ESXi-Hosts is iSCSI
- there is one ZVOL (2.1 TiB, RAID 10 - mirror / stripe)
- Hypervisor is VMware ESXi 6.0 U1
- ESXi-Hosts are HP ProLiant DL360 G6 (each equipped with 32 GB of RAM, Intel XF SR2 10 GbE network card)

EDIT: I do not use L2ARC or any optimizations with the help of SSD drives.

 

[HoneyBadger]

I would try to do some testing via iperf between the machines to look at potential network issues, as well as testing directly on your FreeNAS machine to see if local disk performance is better.

16x300GB 10K SAS drives should be able to drive much, much more throughput than that.

 

[jgreco]

So, when you say "RAID 10", hopefully you don't ACTUALLY mean RAID10, but rather "8 ZFS vdevs of two mirrored 300GB drives each".

See, terminology matters. Because what you've SAID suggests that you're using the HBA in IR mode, with the mfi driver, which is known to suck. If you're doing that, then by all means, stop, go and read the manual, flash your card to IT mode, make an actual ZFS pool after reading @cyberjock 's newbie presentation, and then try it again and see if it works better. Because it will.

You also don't want to make a 2.1TB zvol on top of this. Fragmentation kills performance; for iSCSI, this should probably limit you to no more than about 1TB, or maybe 500GB if you want things to be reasonably zippy.

 

[fc7t]

Hey there,

I got the LSI 9211-8i already with the v. 20.00 firmware installed. I did not create any logical volumes within the controller menu. Just installed into the server, applied cables and booted the server. FreeNAS did see every single drive. So I think the card was already flashed with the right firmware to be in IT mode.

@jgreco: no, it's not a question of hardware RAID 10. I create 8 ZFS vdevs of two mirrored 300 GB drives each.

 

[jgreco]

@jgreco: no, it's not a question of hardware RAID 10. I create 8 ZFS vdevs of two mirrored 300 GB drives each.

Then, next time, I'd suggest you describe it appropriately. What sort of speeds are you getting out of the pool locally? What's the iperf test results on your network like?

 

[cyberjock]

I would add a 100GB L2ARC SSD if I were you. Faster for read performance, the better. L2ARCs, with sufficient RAM in the system (which you have the lower end, but enough to sustain a 100GB L2ARC), can really supercharge VM performance.

 

[jgreco]

I would add a 100GB L2ARC SSD if I were you. Faster for read performance, the better. L2ARCs, with sufficient RAM in the system (which you have the lower end, but enough to sustain a 100GB L2ARC), can really supercharge VM performance.

That's way premature at this point; there appear to be other substantial Issues with whatever this platform is.

 

[fc7t]

One point to mention is that I use direct point-to-point connections as following:

SE326M1 - ESXi-Host1
Card1,Connector1 - Connector1
Card2,Connector1 - Connector2

SE326M1 - ESXi-Host2
Card1,Connector2 - Connector1
Card2,Connector2 - Connector2

Multipathing for the ESXi-Hosts is enabled in vSphere (Round-Robin). Cables are LC-LC OM3 (up to 10 Gbit/s).

 

[fc7t]

I have done some iperf testing with one of the two ESXi-Hosts directly connected to the storage server. Here are the results:

Code:

[root@DE001-HYP-43-2:/tmp/iperf] /opt/iperf/bin/iperf -c 10.80.112.71 -P 4
------------------------------------------------------------
Client connecting to 10.80.112.71, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  6] local 10.80.112.22 port 27524 connected with 10.80.112.71 port 5001
[  3] local 10.80.112.22 port 14244 connected with 10.80.112.71 port 5001
[  5] local 10.80.112.22 port 22066 connected with 10.80.112.71 port 5001
[  4] local 10.80.112.22 port 24503 connected with 10.80.112.71 port 5001
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-10.0 sec  1.93 GBytes  1.66 Gbits/sec
[  3]  0.0-10.0 sec  1.94 GBytes  1.66 Gbits/sec
[  5]  0.0-10.0 sec  1.93 GBytes  1.66 Gbits/sec
[  4]  0.0-10.0 sec  1.94 GBytes  1.66 Gbits/sec
[SUM]  0.0-10.0 sec  7.74 GBytes  6.65 Gbits/sec
[root@DE001-HYP-43-2:/tmp/iperf] /opt/iperf/bin/iperf -c 10.80.114.71 -P 4
------------------------------------------------------------
Client connecting to 10.80.114.71, TCP port 5001
TCP window size: 35.0 KByte (default)
------------------------------------------------------------
[  6] local 10.80.114.22 port 59065 connected with 10.80.114.71 port 5001
[  4] local 10.80.114.22 port 35749 connected with 10.80.114.71 port 5001
[  5] local 10.80.114.22 port 60495 connected with 10.80.114.71 port 5001
[  3] local 10.80.114.22 port 40716 connected with 10.80.114.71 port 5001
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-10.0 sec  1.93 GBytes  1.66 Gbits/sec
[  4]  0.0-10.0 sec  1.93 GBytes  1.66 Gbits/sec
[  5]  0.0-10.0 sec  1.93 GBytes  1.66 Gbits/sec
[  3]  0.0-10.0 sec  1.93 GBytes  1.66 Gbits/sec
[SUM]  0.0-10.0 sec  7.73 GBytes  6.64 Gbits/sec

 

[jgreco]

One point to mention is that I use direct point-to-point connections as following:

SE326M1 - ESXi-Host1
Card1,Connector1 - Connector1
Card2,Connector1 - Connector2

SE326M1 - ESXi-Host2
Card1,Connector2 - Connector1
Card2,Connector2 - Connector2

Multipathing for the ESXi-Hosts is enabled in vSphere (Round-Robin). Cables are LC-LC OM3 (up to 10 Gbit/s).

Hopefully you're not attempting multipath just yet. Building complex systems is hard enough without layering in unnecessary complexity. The pointopoint connections are fine, of course.

 

[jgreco]

FreeNAS did see every single drive. So I think the card was already flashed with the right firmware to be in IT mode.

Upon rereading this, I see you haven't ACTUALLY flashed your card to IT mode. Check. This is exactly the sort of issue we see when people use the mfi driver and the MFI firmware. These cards come with MFI firmware by default and the only way you would have the IT firmware on there is if someone had deliberately flashed it on there at some point. Make sure that the firmware on there is 20.00.04.00, NOT 20.00.02.00, NOT 19.*, etc. It is not expected to work correctly with the wrong firmware on there.

 

[fc7t]

After flashing the LSI 9211-8i HBA Adapter to the right 20.00.04.00 IT firmware I did some testing again.

Cloning a VM with 12 GB disk size took about 5 minutes. Booting a Windows Server 2012 R2 VM with nothing else installed on the OS took over 1 minute. I think this is still to slow.

 

[fc7t]

One more thing. In my first post I mentioned that copying a file of 4 GB size is extremely slow when both VMs (the virtual NAS and the OS to which I want the file being copied to) are on the same LUN (which is presented through FreeNAS and iSCSI protocol).

Now here's a strange thing which I observed:

When the virtual disk of the virtual NAS VM is on the QNAP NAS (which is only connected to the hosts via 1 GbE iSCSI) and the Windows Server 2012 R2 VM is on the FreeNAS storage the copying rate goes up to 90+ MByte/s. So I think that the system does not have problems with write rather than with reading speed.

 

[jgreco]

Okay, so, now that I'm back on a Real Computer, two things:

1) Disable compression and run some dd tests on the pool from the NAS CLI. First create a 256GB file from /dev/zero and report speed/time to complete,

# dd if=/dev/zero of=/mnt/${poolname}/mytestfile bs=1048576 count=262144

This should take awhile. Control-T will give you status updates.

Then

# dd if=/mnt/${poolname}/mytestfile of=/dev/null bs=1048576

Again report speed/time to complete. Assuming your disks can push at least 60MB/sec (depends on age, that's a minimum I'd expect), I'd expect each test to run in the 10-15 minute range, hopefully on the lower end of it. My filer with 7 mirror vdevs of 2TB slowish laptop drives does that writing it out at 260MBytes/sec, but has a lot of fragmentation and other traffic to deal with.

2) Suggest that it'd be interesting to see iperf results with a larger (256K) window size, in both directions from the server. *guessing* this isn't a problem but sometimes it's just nice to check the easy stuff.

 

[fc7t]

Hi,

because the size you set in the command is probably going to take the whole night, I've set it to a smaller value. Here are the results:

# dd if=/dev/zero of=/mnt/VOL1/mytestfile bs=1048576 count=10214

10710155264 bytes transferred in 198.128380 secs (54056644 bytes/sec)

# dd if=/mnt/VOL1/mytestfile of=/dev/null bs=1048576

10710155264 bytes transferred in 344.650258 secs (31075431 bytes/sec)

The problem seems to be related to the installed hard disks (HP, 2.5, 300GB SAS, 10K, 6G).

 

[fc7t]

BTW.: The MTU size is set to 9000 on both sides (FreeNAS via interface options "mtu 9000" and ESXi-Host via vSwitch). Also FreeNAS system itself is installed on a slow USB 2.0 thumb drive.

 

[HoneyBadger]

Speed of the boot media shouldn't matter. I'm wondering if maybe the cache on your drives has somehow been disabled. On mobile right now so it's hard to look up the appropriate camcontrol syntax to check, but that might be a good rabbit hole to delve down.