ESXi + FreeNAS basic Q

[pchmt]

I have looked around but cannot find anything which quite answers my specific questions. I have read lots about FreeNAS on VMware, and seen the changed recommendations (provided it is correctly configured), but I am a little confused about a few basic points.

1 - Is there any reason I cannot connect the drives to the SATA ports on the MoBo instead of using a LSI card in IT mode as seems to be the guidance if I read it correctly?
2 - Can I still boot FreeNAS off a (passed through) pair of USB drives or is that frowned upon when virtualising.

My objective is to have ESXi running a few VMs and have freeNAS provide the raided storage for it and a wider system at home. I looked at Corral - but I think that the VMware virtual environment is better for the VMs in as much as I can start things like DNS/DHCP and a Active Directory server without booting freeNAS first - or am I just barking up the wrong tree and I should just go with bhyve? Really appreciate some guidance here before I go too far.

Plan is to use a FreeNAS 9.10, SuperMicro X9 with an E3-1220 and 32Gb RAM and the six on board SATA ports - 4 for general raid storage and 2 for the ESXi datastores in mirrored Raid (which is plenty for my needs). TIA. If this is going to be insufficient or unrealistic, I'd like to know before I go too far.

 

[joeschmuck]

1 - Is there any reason I cannot connect the drives to the SATA ports on the MoBo instead of using a LSI card in IT mode as seems to be the guidance if I read it correctly?

It depends on the motherboard but you typically can pass through the motherboard SATA ports but you pass thought the controller not single ports, however you need to boot ESXi off of something and you need to store your VMs on something. When you factor those in, what SATA ports will you use?

2 - Can I still boot freeNAS off a (passed through) pair of USB drives or is that frowned upon when virtualising

I don't know, I'm not sure how that would work but I think it would be possible, but then why use ESXi if you are not really creating a VM. I would frown upon this.

I looked at Corral

Corral is dead. Read the announcements.

the six on board SATA ports - 4 for general raid storage and 2 for the ESXi datastores

Nope, won't work. You cannot pass through select SATA ports, you pass through the controller and all the ports associated with the controller are then no longer usable by ESXi.

As for running a VM on FreeNAS. My advice is that if you are using ESXi then run VMs on ESXi. If you desire to run FreeNAS on ESXi then fight the urge to run a VM on FreeNAS. But there may be exceptions such as well running small VMs in the Dockers. I would not run an Ubuntu VM on FreeNAS that is running on ESXi. So if you are running ESXi, run all your VMs from there is you can. ESXi is a very mature and stable product.

If you want a good long read of my adventures and conserns with running and configuring ESXi for FreeNAS use, read this thread. Lots of good information here.

Lastly, do not run a firewall on this same ESXi machine, you will earn to hate it. If you need to reboot or do something, it takes out your internet and you never want that. If you plan to run a firewall then use a small computer for it that is dedicated to the firewall. I have a second ESXi machine and it's primary role is the firewall. It's secondary role is a FreeNAS backup server. I rarely touch this machine and it just works. Learn from my pain.

 

[Dice]

I agree on all the directions @joeschmuck points to.
1. Involving motherboard SATA ports complicates things. From forum experience - it is significantly less prone to functioning reliable compared to a HBA passed through.
2. Why would you even consider this...?

Though I'd like to accentuate some conclusions you should have gathered from his reply. Repeat after me:
I need more researching and reading to better understand what ESXi can, should and cannot do.
At this point you show signs of significant confusion.

 

[pchmt]

I agree on all the directions @joeschmuck points to.
1. Involving motherboard SATA ports complicates things. From forum experience - it is significantly less prone to functioning reliable compared to a HBA passed through.
2. Why would you even consider this...?

Though I'd like to accentuate some conclusions you should have gathered from his reply. Repeat after me:
I need more researching and reading to better understand what ESXi can, should and cannot do.
At this point you show signs of significant confusion.

LOL - I am certainly confused about some points ... you are right there. Hence my Qs :) That said, I think I may have been a little less clear on one point. Obviously if I use ESXi then that will be where any VMs are located. The Q about using VMs inside FreeNAS was in ref to using Corral directly and just avoiding ESXi altogether. I am inclined to ESXi because of its proven stability though I had not heard Corral is dead ... I missed that. That was quick!

Thanks to both, it has been a big help.

Just want to add as an edit - re point 2, now you have pointed this out, I can see why you said that. I am still thinking in bare hardware terms sometimes and not yet fully Virtualising ... Thanks once more.

Oh and ... "I need to do more research and reading" (well, I realise that anyway, but there you go) :)

 

[pchmt]

It depends on the motherboard but you typically can pass through the motherboard SATA ports but you pass thought the controller not single ports, however you need to boot ESXi off of something and you need to store your VMs on something. When you factor those in, what SATA ports will you use?

Sadly, my long detailed reply and thanks has just gotten lost :-( Bottom line ... this has been really helpful although one or two of my questions were seemingly somewhat less clear than I thought the issues have non-the-less been addressed. Just one small bit I will retype ... as the board has SATA2 and SATA3 ports, I assumed they would be different controllers, so that was how I was planning to split the pass through/VM use.

Thanks again

 

[joeschmuck]

You would need to confirm that the board does actually have two separate controllers, and even then you would have to test it out. Sometimes a controller may not be listed to pass thought so you would need to update the pass-thru map to facilitate this. This is not difficult but until you know that you can do it, you just won't know.

At this point you show signs of significant confusion.

Agreed! And I've had years of experience with VMWare Workstation which is very similar to ESXi, yet different too. Discipline would be my #1 thing to state. You must be disciplined in how you run your ESXi system.

So good luck if you go down this road. I really like ESXi and it does what I want it to do very well.

 

[pchmt]

You would need to confirm that the board does actually have two separate controllers, and even then you would have to test it out. Sometimes a controller may not be listed to pass thought so you would need to update the pass-thru map to facilitate this. This is not difficult but until you know that you can do it, you just won't know.

Useful info ...

Discipline would be my #1 thing to state. You must be disciplined in how you run your ESXi system.

I think that applies to most technical endeavours, but I would be really interested to know specifically what you mean by disciplined. I have been playing with ESXi for a couple of weeks now on a spare bit of kit and I too am really liking it, but I am only just beginning to get some basic idea of what it can and cannot do - so as I said before I am only too aware of my understanding limitations right now. I am learning fast however

So good luck if you go down this road. I really like ESXi and it does what I want it to do very well.

Well, I have now bought a LSI 9211-8i controller so I am certainly taking your advice I am old enough to have been through the wringer a few times ... My father used to say two things which I try to live up to.
1) "A smart man learns from other people's mistakes so he does not have to learn from his own", and
2) "The difference between a smart man and a fool is a smart man only makes the same mistake once."

As before, thank you so much for taking the time to help me. I really do appreciate it.

 

[joeschmuck]

Disipline = Setting up the ESXi system and being very mindful of what VMs you have on it and what effects there will be as you work on the ESXi itself. This is why i moved my firewall to a different ESXi machine. Bringing down the internet in the house is no fun when you forget about it and you have a lot of complaints coming at your. Or not properly shutting down the VMs before powering off your computer or rebooting it. In an ideal world things work all the time properly but that is not always the case.

 

[Dice]

Another excellent scenario is that you somehow mess up what IPs left open for DHCP and what are set as static. Occasionally forgetting that some tweaks cause pretty significant repercussions for the environment. I ended up in a situation where I had setup hosts and ip's having it running for a solid year without any sort of problem, feeling content and what do you know, detail knowledge deteriorates over time. Then I figured I wanted to poke and prod a bit among my systems and... move around some VMs between hosts. What do you know - I forgot paying attention to a few details and a reboot a week later or so ended up with no network connectivity on any machine on the network what so ever.

I believe it is good to go through a couple of "these" first hand, all lost hair in the process included...
Doing everything "correct from guide" from the first day without mishaps won't set you up to think through system design... and improving it to alleviate the flaw.

 

[pchmt]

Another excellent scenario is that you somehow mess up what IPs left open for DHCP and what are set as static. Occasionally forgetting that some tweaks cause pretty significant repercussions for the environment. I ended up in a situation where I had setup hosts and ip's having it running for a solid year without any sort of problem, feeling content and what do you know, detail knowledge deteriorates over time. Then I figured I wanted to poke and prod a bit among my systems and... move around some VMs between hosts. What do you know - I forgot paying attention to a few details and a reboot a week later or so ended up with no network connectivity on any machine on the network what so ever.

Been there, done that ...
As for discipline, already have that planned. My existing freeNAS and test ESXi are staying exactly where they are (as discrete bits of hardware to fully isolate any potential problems). The 'new' ESXi system will be the one which run my 'production' stuff. NOTHING will get onto my production mc's until I have tested it first on the test ESXi. The freeNAS is going to be file serving only, and the existing freeNAS retained, first as a store to from which to build my new freeNAS VM pools and then as a file backup (in addition to my off site hdd rotations). The 'test' ESXi environment will also hold warm backups of my 'critical' VMs just in case I need to get something going quickly which was screwed up.

As for network config stuffups, I keep a discrete backup system configured and up to date with any changes - after they have been validated. If I screw up, I just shut down the main mc, boot up the warm standby and get myself up and running in less than 5 minutes. I then can take the offending mc away to my isolation network (an old 8 port Gb switch not attached to my live network) to 'fix' it, make sure it is properly fixed and bring it back into service (first turning off the backup so it does not auto update just in case).

For any updates that work fine, I turn on the warm backup, leave it for about 5 minutes for the cluster to sync, turn off the master, check the backups are working, restart the master to auto take control and shut down the warm backup.

While I am finding my feet with VMs, and ESXi in particular, I have been around long enough to at least know when I am still floundering.

I presently have all of two weeks messing about experience with ESXi, so I hope you will understand why I am a little fuzzy on some things yet. As I said, I am learning quickly. All that said, I am getting there, but I always welcome help and advice ... so thank you once again.

 

[joeschmuck]

Hey we all started out not knowing ESXi so we do understand the learning process.