Where to next? Help please!

[Shankage]

Hi all,

I have been battling some slow storage issues with my setup and wanted to reach out to get some input on ideas on how to proceed.

To start with I will detail the environment:
3 ESXi hosts in the environment - for the purpose of this thread I am using two, one is the Management host (will be referred to as FreeNAS host) and another is my compute host (referred to as remote host).

The server is used as iSCSI for VMware, nfs / smb for media and plex.

FreeNAS host spec:

• Dell PowerEdge R710
• 2 Xeon x5670's
• 240gb DDR3 ECC Ram
• LSI HBA (flashed) in passthrough
• ESXi 6.7 with vSS
• On board Qlogic NetXtreme II BCM5709 nic
• 6 WD RED 6tb EFRX's in mirror
• 2 uplinks used from host - not lacp

Remote Host:

• R720
• 2 x E5-2660's
• 128gb ram
• Onboard gigabit nic
• ESXi7
• 2 uplinks used from host - not lacp

Network:

• Access layer connecting both hosts (dell n2048)
• ports connecting to hosts are trunk ports
• Aggregation / L3 is trunked to this switch (cisco 3560)
• Cisco 3560 is connected to PFsense, installed on a Sophos XG430
• Jumbo frames is enabled on the access switch
• Jumbo frames is enabled on the cisco, however it does not support routed jumbo frames (read on to find out why this should not be an issue)
• jumbo frames enabled on pfsense
• Subnets relevant to FreeNAS
  • 192.168.63.150 - primary nic to reach FreeNAS, the gateway for this subnet is terminated on the Cisco
  • 10.55.0.0/24 - Storage network one and connected to interface one on the FreeNAS VM (terminated on the Dell)
  • 10.56.0.0/24 - Storage Network two and connected to the second interface on the FreeNAS VM (terminated on the Dell)
• Both hosts can see all storage created and presented in FreeNAS (iSCSI) and can be used
• vmkpings from the hosts to the gateway and to the IP interface of the freeNAS are successful with df-bits and an mtu of 8972
• Datastores are attached with round robin policies
• FreeNAS is configured with two interfaces 10.55.0.9 and 10.56.0.9
• Bonding and LACP is not configured anywhere on the network, hosts or VMs

Now onto the testing, I started noticing VM responsiveness was quite slow and laggy so decided to do some iperfing and crystal disk marking and the results are in the attached word doc. All disks that are being tested in the screenshots are disks created on a FreeNAS luns and the pool has 21% used.

I have attached a document with the tests and results, I am not entirely sure how to test iSCSI performance and those networks from the host.
Happy to have any feedback or general guidance to improve this as it is at times unbearable!

One thought I had was moving the 192.168.63.x gateway down into the dell, but not sure how that would improve things.



Thanks!

 

[Shankage]

Forgot to add
FreeNAS config:

• 4 vcpu
• 48 GB ram
• Os installed on nvme backed data store
• Pcie device passthrough
• 3 vmnics, one for 192.168.63.x, storage network one and two

 

[jgreco]

You should dump ESXi and run on bare metal until you have things working well.

48GB is maybe okay for iSCSI if you are doing very little with it, maybe just a handful of sleepy VM's. Generally 64GB is the floor recommended for iSCSI.

Please see https://www.ixsystems.com/community/threads/the-path-to-success-for-block-storage.81165/

 

[Shankage]

Baremetal is not an option as I have other vms running on it.

I've tried to do some research on recommended ram and have not ever seen what you have suggested, can you provide some documentation other than that link?

Do you have any troubleshooting steps to help isolate the issue ?

 

[jgreco]

Baremetal is not an option as I have other vms running on it.

Go get yourself an eval copy of vSphere and use VCSA to migrate the VM's.

I've tried to do some research on recommended ram and have not ever seen what you have suggested, can you provide some documentation other than that link?

No. I've written a crapton about iSCSI and block storage and all of that, and over the years I've distilled it into the information in that post, and articles linked to from that post, and a bunch of other related stuff in the resources sections. You're under no obligation to listen to me and if you don't believe what I've written, I'm perfectly fine with that. I don't feel obligated to provide you with alternative documentation that says what I'm saying. You will be able to find other people that say 16GB is great or even 8GB is fine. And it might be, for poor definitions of "fine."

Do you have any troubleshooting steps to help isolate the issue ?

Yes. Dump ESXi and run on bare metal until you can get things working well. This eliminates a bunch of problems that result when you layer one incredibly complex bit of software on top of another incredibly complex bit of software.

 

[Shankage]

Thanks for the response.

Go get yourself an eval copy of vSphere and use VCSA to migrate the VM's.

I don't need eval copies as I have full access to the software, however FreeNAS provides the storage to the remote host, so taking it offline will mean none of them will run on the remote host.

No. I've written a crapton about iSCSI and block storage and all of that, and over the years I've distilled it into the information in that post, and articles linked to from that post, and a bunch of other related stuff in the resources sections. You're under no obligation to listen to me and if you don't believe what I've written, I'm perfectly fine with that. I don't feel obligated to provide you with alternative documentation that says what I'm saying. You will be able to find other people that say 16GB is great or even 8GB is fine. And it might be, for poor definitions of "fine."

I apologize if my asking for further documentation was in someway misconstrued, my intent was not to personally attack your post or you in anyway shape or form. As I have been in this industry for quite some time and have been client facing, I have learnt to substantiate any claims or opinions I have with facts rather than just my own opinion (where required). It was not meant to say that I do not believe you, I just wanted to verify before I overhaul my entire lab and spend quite some time doing it.

Yes. Dump ESXi and run on bare metal until you can get things working well. This eliminates a bunch of problems that result when you layer one incredibly complex bit of software on top of another incredibly complex bit of software.

I know my hypervisor and its workings quite well, I was hoping there was some way to dig deeper into FreeNAS to see if there was something going on.. logs or what not, rather than just a blanket statement of redo, as there will be ripple effects of doing so.

I am happy to start by increasing to 64gb ram as I have the capacity to do so and can report back on that.

 

[Shankage]

Here is the second round of testing, out of curiosity what speeds should I be expecting in these for a gigabit network?

I've tested reading and writing to the NFS storage from the windows vm on the freenas host, copying down from it I got 115MB/s, writing to it ~65MB/s - 90. What should i expect here?

CrystalDiskMark on the remote host vms is averaging around 122 MB/s, what should I be expecting here?

 

[Shankage]

I have done some more testing, I am able to copy and paste to the NFS share around 105 MB/s average on the remote host and VM. I also tried mapping a network drive to the NFS share and CrystalDisk to it, getting 117MB/s read and 117MB/s write.