Nested ESXI Dropping iSCSI connection (errno 32)

[prateeksaurabh]

Hi All, I dont have much knowledge about storage but i am windows admin and learning VMware.
I have made nested ESXi environment. I have FreeNAS 8.0.3 x 64bit with 2 GB RAM, E1000 adapter and 16GB to drive where i installed freenas and another 300GB for shared storage that i would connecting to main Host ESXI and other nested ESXIs. The 300 GB target that i made i have tried both UFS and ZFS file system and i using simple RAID0 with no redundancy at all for best performance. I have also Multipath configured for this 300GB target on all the host ESXis. I am having 2 spare NICs so the moment i introduced second NIC as multipath the performance significantly increased so i thought is bandwidth the problem? So I introduced also 3rd NIC but this time i didnt notice any better performance.
Now i am left out to believe that my hard disk could be a problem which i am trying to use as sharedstorage as it is not SSD but simple 7200 RPM WD 500GB hard drive.
All right now question arises what am i doing and when i get these errors. I have made 1 datastore of almost 1TB (960GB) joining two 500GBs drives. Then carving out 300GB LUN out of it for ISCSI. I have made a folder ISO in that 1TB drive where i have kept all my ISOs. Now when i am trying to create a VM (win server 2008 R2) with its vmdk, vmx etc file in 300GB shared storage i get this error and then this datastore (shared storage) becomes inactive

Please find the error log attached here

 

[jgreco]

Yup, the fix is don't-do-that.

The point of computing is not to make everything as complicated and clever as possible.

 

[prateeksaurabh]

Yup, the fix is don't-do-that.

The point of computing is not to make everything as complicated and clever as possible.

Hello Jgreco
Sorry i dont have much knowledge so sorry if you find this rude or dumb but when you say don't do that what does that mean? do u mean nested virtualization? or u mean i should use a SSD disk?

This is just my test lab and not a production environment cannot spend a lot of money :( so that is why such a setup with nested ESXI however if buying a SSD drive would solve my problem i would go and buy one :). Only i want to sure it is only SSD that is problem and not something else so i dont waste money buying SSD.

Thanks a lot for replying and for your help

 

[jgreco]

I appreciate not being able to spend a lot of money. Even when you have money, the goal is to spend it to best effect.

Virtualizing FreeNAS is a bad idea. Nesting ESXi is a bad idea. As long as you don't mind losing it all, then I have no specific objections to doing either or both. However, with each level of abstraction and complication will come additional latency and additional overhead, and it is going to get substantially slower. This is quite possibly the problem you're facing.

But the flip side to that is that you create a system with many interrelationships that are not always immediately obvious. You're not going to be able to get a lot of help here, because most of us who do anything with ESXi are either doing it the recommended way with a hypervisor on one host and FreeNAS running bare metal on another, which gives you a reasonable shot at correctly characterizing where a problem lies, or possibly following this guide to not completely losing your data. That involves running FreeNAS in a pseudo-bare-metal environment to provide FreeNAS with the aspects of a server that it really ought to have bare-metal access to (basically the disk I/O). I am in the apparent minority in that I feel that this can potentially be done safely and reliably in a recoverable manner. Others in the forum have gotten tired of the parade of failures where something has gone wrong for an ESXi user and they've decided FreeNAS is to blame. So my rather hard-to-see point is that you're not too likely to find a whole lot of help for a complicated nested ESXi with virtualized FreeNAS setup here.

You're really best off with an external network device for shared iSCSI storage for an ESXi cluster. They're pretty readily available these days, for example, the iomega 35887 is often on sale for $60-$70. They're not fast... 50MB/sec on a good day... but stick a pair of drives in there mirrored and you have an ESXi datastore that you can use as a safe base-of-operations, for backups, etc. Then you don't need to get all paranoid about messing up your virtualized FreeNAS, and you can experiment safely knowing that if you manage to trash the FreeNAS, you don't lose it all.

 

[gpsguy]

I second jgreco's sage advice about using a separate storage array. While a few on the forum (he included) can run both on the same box, most folks attempts at doing it, result in data loss.

In my home lab, I have separate boxes for FreeNAS and ESXi.

While nesting ESXi is a "bad idea", I can understand the desire to create a 2-3 host cluster and play with HA, DRS, vMotion, etc. I presume the nested copies of ESXi will probably be 60-day evaluation versions, so you'll be able to play with all the licensed features. Since it's only for testing, I wouldn't worry about running it that way.

 

[RegularJoe]

May 27, 2013 was several years ago, we now have VMware 6.5 and FreeNAS 11.
I am wondering if we can do it as the VMware interface for VM's is much better than FreeNAS. If we could get FreeNAS to run as a module on VMware's Console OS at the same time as VMware that would be cool too. I wonder if the guys with IBM LPAR/PPAR can do this now....

I just did a test booting FreeNAS 11u2 via UEFI and VMware ESXI in a bhyve VM booted up the install ISO file and locked after loading the VMware drivers. :-(

I have 20gb of RAM on this box so I might have to install VMware on the physical hardware and run FreeNAS as a VM with passthru SATA disks.

VMware 6.5 u2 20170104001
FreeNAS-11.0-U2 (e417d8aa5)

should I be looking at installing packages(ClonOS) on FreeNAS so I can use it as a server?

https://clonos.tekroutine.com/index.html

Thanks,
Joe