ESXI 6.7 - Freenas 11.1-U7 - Mellanox -MNPA19-XTR -Connectivity Issues

[mnichols1202]

Hello All,
I've worked with VMWare for Years but i'm reasonably new to Freenas. I've soured the Forums for posts related to the issue i'm having and there doesn't seem to be any existing posts for this particular issue. So apologies if I've missed something. Here's my issue.

ESXI 6.7 Whitelist system
i7
64GB Ram
boot from USB

Freenas 11.1-U7
I3
16GB Ram
4 2TB Mechanical SATA Drives (Raided)
boot from USB

1 Mellanox MNPA19-XTR 10GB NIC per system connected via 1 Foot SFP+ twinax cable (firmware updated to latest ConnectX-2 firmware - 2.9.1200

Issue: Stack will run forever...unless/until something puts a heavy/ sustained transfer load the system. (i.e. - multiple vms start backing up, several vms boot simultaneously, large copy operation.) Write operations seem to make it fail quicker more predictability than large read operations.

vmware looses connectivity to the freenas host and starts logging the following error:

Login to iSCSI target iqn.2005-10.org.freenas.ctl:iscsi-target on vmhba64 @ vmk2 failed. The iSCSI initiator could not establish a network connection to the target.

this persists until i reboot the freenas. (just rebooting ESXI server might clear as well but trying shut down vms and restart esxi host problematic with no connectivity to disks.)

used this article to set up iSCSI from FreeNAS to VMware: https://johnkeen.tech/freenas-11-iscsi-esxi-6-5-lab-setup/

i have 2 identical "stacks" (identical hardware, identical software, identical bios and firmware.) and both encounter the exact same issue.

Anyone have any ideas? I'm stuck, and if i can't figure this out I'll have to abandon this project.

Any additional information required will be happily provided, just trying to keep things short and simple.

Mike

 

[Spearfoot]

Hello All,
I've worked with VMWare for Years but i'm reasonably new to Freenas. I've soured the Forums for posts related to the issue i'm having and there doesn't seem to be any existing posts for this particular issue. So apologies if I've missed something. Here's my issue.

ESXI 6.7 Whitelist system
i7
64GB Ram
boot from USB

Freenas 11.1-U7
I3
16GB Ram
4 2TB Mechanical SATA Drives (Raided)
boot from USB

1 Mellanox MNPA19-XTR 10GB NIC per system connected via 1 Foot SFP+ twinax cable (firmware updated to latest ConnectX-2 firmware - 2.9.1200

Issue: Stack will run forever...unless/until something puts a heavy/ sustained transfer load the system. (i.e. - multiple vms start backing up, several vms boot simultaneously, large copy operation.) Write operations seem to make it fail quicker more predictability than large read operations.

vmware looses connectivity to the freenas host and starts logging the following error:

Login to iSCSI target iqn.2005-10.org.freenas.ctl:iscsi-target on vmhba64 @ vmk2 failed. The iSCSI initiator could not establish a network connection to the target.

this persists until i reboot the freenas. (just rebooting ESXI server might clear as well but trying shut down vms and restart esxi host problematic with no connectivity to disks.)

used this article to set up iSCSI from FreeNAS to VMware: https://johnkeen.tech/freenas-11-iscsi-esxi-6-5-lab-setup/

i have 2 identical "stacks" (identical hardware, identical software, identical bios and firmware.) and both encounter the exact same issue.

Anyone have any ideas? I'm stuck, and if i can't figure this out I'll have to abandon this project.

Any additional information required will be happily provided, just trying to keep things short and simple.

Mike

Hello, and welcome to the forum!

The more information you provide, the more likely it is that someone here will be able (or willing!) to help you out with your problem. Here are the forum rules, which include guidelines on what information to include about your hardware and software configurations:

https://www.ixsystems.com/community/threads/forum-guidelines.45124/

How did you configure the pool in your FreeNAS server? For your application, a pair of mirrors will give the best performance.

Your FreeNAS system may not have enough memory to provide good performance, according to the FreeNAS 11.1-U7 docs, which state: "For iSCSI, install at least 16 GB of RAM if performance is not critical, or at least 32 GB of RAM if good performance is a requirement."

What kind of results do you get using iperf to exercise your network?

FWIW, while I have tinkered with iSCSI datastores in the past, I've used NFS exclusively for a long time now... it's just easier. :)

Good luck!