Poor Hyper-V VM performance on ISCSI

[Mac Thompson]

Hi There,

We have a new FreeNAS implementation (FreeNAS-9.10-STABLE-201606072003) running on the following:

HP DL380 Gen 9.
- 2 x Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz (6 cores) 2xAPNIC disabled and booting to legacy BIOS.
- HP P840 12Gb SAS controller running in HBA mode
- 2 x 32GB Kingston USB3 (internal) in a mirror for the FreeNAS OS
- Onboard intel 1Gbp LAN for GUI access
- 1 x 2 port Chelsio 10Gbps in a VLAN LAGG for storage access

ARC
- 576GB DDR4 ECC RAM

L2ARC
- 2 x 1.2TB Intel 750 PCI Express SSD stripped to give 2.4TB

ZIL
- 2 x 400GB Intel 750 PCI Express SSD mirrored to give 400GB

Storage
12 x 6TB 12Gb SAS 7200rpm Seagate LFF in 6 mirrored vDev to give 36TB useable
Storage pool is a 24TB LUN shared via ISCSI with about 7TB used
LZ4 compression is on, encryption and de-duplication are off.

We are running approximately 50 VMs on it with two Windows 2012R2 Host nodes in a cluster. We are snapshoting hourly during business hours and replicating to a seconds DC over a 10Gb fibre which has a freeNAS backup server running on Hyper-V (actually this backup system runs very well and we achieve 500Mbps when it replicates at night which is rather good considering the backup FreeNAS install has none of the fancy specs of the main unit).

The problem is the main production unit for which I have given the specs above. Disk response time is atrocious when we do anything that involves copying data to or from the LUN. So moving a server on or off bogs its down, replicating at the host level (as opposed to FreeNAS replication which as described above is fine), host level backups, hyper-v snapshots and merges etc. Response times climb to 15 for even a few hundred milliseconds for the disk where as for optimum performance everything should be under 10.

Is there any suggestions for what I should be looking for or tuning to improve performance?

 

[depasseg]

Add more vdevs or switch them SSD's. (I presume the HP backplane is SAS3? There have been some weird perfomance issues over the last year related to a mismatch of HBA and backplane when it comes to SAS3).
How much free space do you have?
What is your fragmentation level?
Let's assume that your RAM to L2ARC ratio is good and that you aren't RAM starved (which is close given the 5:1 ratio).

With 6 vdevs you are looking at a max of ~900 IOPS (6*~150). The SLOG isn't going to help that. In fact, with ISCSI by default, sync writes aren't enabled. Have you configured that? If not, the SLOG is likely sitting there idle. (take a look at zilstat, arcstat.py, and arc_summary.py).

 

[Mac Thompson]

OK thanks. Yes its SAS3. Fragmentation is 13% and we have lots of free space. See below:

I configured a SLOG in the volume creation area. Here is the volume summary: How do I know if my SLOG is been used?:

I tried to run zilstat but i got a no such file or directory

 

[mjt5282]

zilstat on 9.10 is in /usr/local/bin, i believe you need to be root to run it.

 

[DarkLordSilver]

Do you have flow control and Jumbo frames enable end to end?

 

[maglin]

You have the IOPs of 6 drives for 50 VMs and 2 Windows servers. I think that is your Achilles heel right there.

I'm not the expert here but I would add 12 SSDs to run the VMs and use the current 12 spinning disks for storage and not system files. This might not be an option but I believe it might get you the performance you are looking for.

You can run a
gstat -p

To see how the array is working and see the IOPs that is being issued.

Our resident grinch would know more on this subject.

One last thing what is the CPU load when this is happening?


Sent from my iPhone using Tapatalk