Build for ESXi

[Snapper]

Hi

Firstly, yes it have read the post in the newbie section saying not to do this, but as an experiment I would like to try. Appreciate I'm on my own though!

I have a HP Gen 8 micro server with 16gb ECC ram and upgraded CPU that supports VT-d (can't remember the model offhand)
I would like to run Freenas on this under ESXi, along with 2 other small VM's (1gb each).
The machine already has 4 Ethernet ports so each VM will have one each, so I don't need to use vswitches for networking.
The bit I'm confused about is passing the on board SATA controller (AHCI mode) through to the Freenas VM - I don't want to do this with virtualised disks...
The Freenas disks are just 2 x 3tb wd red in a mirror, so with 14gb ram available to the VM, I don't see an issue there.

Has anyone successfully passed disks through (and got stuff like smart working) in this way?

I guess my build is slightly different in that it's primarily a freenas build, but using ESXi to allow my home automation servers on the same hardware...

Just wondering if anyone has any useful comments?

Cheers!

 

[Madd Martigan]

You're not passing the controller through to the VM. To do so would eliminate its use for anything else in the system. What you are talking about doing is a RDM (Raw Device Map). You might be able to do that in ESXi (never tried it with a SATA controller) but it won't be the equivalent of a "passthrough" connection. Also, ESXi doesn't recognize SATA RAID and will see any attempt you make at doing so still as separate disks. In order to use RAID for ESXi you have to have an approved array controller from the HCL. I'm not telling you that you can't virtualize FreeNAS but it won't work the way you think.

 

[ser_rhaegar]

The only way this works in that case is to pass all 4 bays in pass through. The bay's use a mini sas connector.

You may be able to pass the main 4 port ahci controller and then have a single drive on the CDROM port for ESXi storage. The 4 port controller is listed separately in ESXi for pass through.

The single drive can sit on top where the CDROM would go.

If that doesn't work you're left with scrapping the network PCIe cars for a m1015 or similar which can be passed through.


If you do use ESXi and FreeNAS, do not use the build in fake raid for your ESXi datastore. It will mess up pass through and scramble your FreeNAS pool.

 

[cyberjock]

http://forums.freenas.org/index.php...nas-in-production-as-a-virtual-machine.12484/

'nuff said

 

[Snapper]

'nuff said

Only if you didn't understand what I'm asking.

As said in the original question, I had already seen that thread. The only point relevant was point 6. I'm using a server, not prosumer or consumer grade kit, so I don't get what point you are trying to make?

 

[ser_rhaegar]

Even using server equipment, virtualizing is very risky and not recommended.

Though a lot of people do it, it isn't encouraged on these forums.

 

[Snapper]

The only way this works in that case is to pass all 4 bays in pass through. The bay's use a mini sas connector.

You may be able to pass the main 4 port ahci controller and then have a single drive on the CDROM port for ESXi storage. The 4 port controller is listed separately in ESXi for pass through.

The single drive can sit on top where the CDROM would go.

If that doesn't work you're left with scrapping the network PCIe cars for a m1015 or similar which can be passed through.


If you do use ESXi and FreeNAS, do not use the build in fake raid for your ESXi datastore. It will mess up pass through and scramble your FreeNAS pool.

Thanks, that's what I was thinking of, just wondered if anyone had done it with any success...

And yes, I will most definitely avoid the fake raid - waste of time IMHO, when zfs or mdadm can be used.

 

[ser_rhaegar]

Thanks, that's what I was thinking of, just wondered if anyone had done it with any success...

And yes, I will most definitely avoid the fake raid - waste of time IMHO, when zfs or mdadm can be used.

No problem. As long as you know the risks and have backups, I wish you the best. I'll answer any questions you have that I can. I virtualize on two Gen8 servers, one is the Microserver.

 

[Snapper]

Even using server equipment, virtualizing is very risky and not recommended.

Though a lot of people do it, it isn't encouraged on these forums.

Yeah, I'm just trying to understand what the risks are.

Totally understand why you shouldn't use virtualised storage, but given ECC ram on a server grad motherboard and a reputable hypervisor, where else would possible problems lie?

 

[ser_rhaegar]

Well I made one change on the host and that caused corruption in my pass through HBA even though it was unrelated. Lost my pool. I had an offline replicated backup to restore from though.

 

[ser_rhaegar]

http://forums.freenas.org/index.php...ng-FreeNAS-(Disaster-example-included).19440/

 

[Snapper]

http://forums.freenas.org/index.php...ng-FreeNAS-(Disaster-example-included).19440/

Thanks, I'll take a look through that thread.

The data that would be on the freenas VM is only time machine backups and media for plex, it's rsynced offsite overnight so loosing data isn't a huge problem...

 

[cyberjock]

Totally understand why you shouldn't use virtualised storage, but given ECC ram on a server grade motherboard and a reputable hypervisor, where else would possible problems lie?

That link has your answers... feel free to read it as many times as you want.

 

[Sir.Robin]

Well I made one change on the host and that caused corruption in my pass through HBA even though it was unrelated. Lost my pool. I had an offline replicated backup to restore from though.

Curious... what "change" were that? :)

 

[ser_rhaegar]

Curious... what "change" were that? :)

For the ESXi datastore, I was originally using a single SSD. I wanted redundancy so I turned on the motherboard's RAID controller, turning off AHCI which was how the SSD was originally being provided to ESXi. Keep in mind, FreeNAS was only looking at my M1015 which was in pass through mode in ESXi to the VM.

Turning from AHCI to RAID on the motherboard caused the M1015 card to go flaky. This flakyness caused errors in many reads on the pool, ZFS then attempted to correct these reads with writing data from more bad reads. Any file I opened was trashed. Took a while to find out that it was the AHCI to RAID change on the motherboard for ESXi as I did not do anything with FreeNAS for a while after that change. I finally found an article online stating that the HP Gen8 B120i controller, when turned on for RAID mode, caused pass through in ESXi to go haywire.

EDIT: now I use a second LSI controller for RAID1 on the ESXi main datastore (i.e. the one that hosts the FreeNAS VM files).

 

[Snapper]

That link has your answers... feel free to read it as many times as you want.

Just read it again.
Lots of info about why not to use rdm or vmdk, but very little on vt-d, which is what I asked about.
Perhaps you could tell me exactly where in the thread these answers are?

 

[Snapper]

For the ESXi datastore, I was originally using a single SSD. I wanted redundancy so I turned on the motherboard's RAID controller, turning off AHCI which was how the SSD was originally being provided to ESXi. Keep in mind, FreeNAS was only looking at my M1015 which was in pass through mode in ESXi to the VM.

Turning from AHCI to RAID on the motherboard caused the M1015 card to go flaky. This flakyness caused errors in many reads on the pool, ZFS then attempted to correct these reads with writing data from more bad reads. Any file I opened was trashed. Took a while to find out that it was the AHCI to RAID change on the motherboard for ESXi as I did not do anything with FreeNAS for a while after that change. I finally found an article online stating that the HP Gen8 B120i controller, when turned on for RAID mode, caused pass through in ESXi to go haywire.

EDIT: now I use a second LSI controller for RAID1 on the ESXi main datastore (i.e. the one that hosts the FreeNAS VM files).

Good info, thanks.
I had a similar issue with the b120i, but going the other way.
Initially had a ssd setup as raid 0, only drive attached to the system and switched to achi, caused the partition table to be wiped.
HP support couldn't explain why changing the mode would cause this issue, if it was a multiple disk array then just maybe, but not a single raid 0...

 

[Snapper]

No problem. As long as you know the risks and have backups, I wish you the best. I'll answer any questions you have that I can. I virtualize on two Gen8 servers, one is the Microserver.

Hi,

What version ESXi are you using?
I've seen issues with v5.1 and later that won't do VT-d properly on the HP?

 

[ser_rhaegar]

v5.1U1 and older work fine
HP disabled VT-d for anything except GPUs on newer ESXi versions. So 5.1U2 and newer won't allow VT-d of HBAs on HP hardware.

See:
http://forums.freenas.org/index.php...-lsi-9240-in-it-mode.17085/page-2#post-104359

 

[Snapper]

Great - thanks.
Out of interest, do you know what the HP Support work-around for 5.5 they talk about in the advisory is?

"HP does not currently have any servers certified for the VMDirectPath feature and a resolution to this problem is not available. For a subset of configurations, VMDirectPath can be enabled with a workaround. Please contact HP support to determine if the targeted solution is a candidate for the workaround. "