Dell Poweredge R730XD, H730 mini, ESXI passthrough

Dave Hob

Contributor
Joined
Jun 15, 2014
Messages
106
Hi

I picked up this unit yesterday and trying to figure out the best setup I can do with it.

My goal is/was to install ESXI on it and run FREENAS, PFsense, other VMS on it.

H730 mini has HBA mode so it doesn't need it flash and after some google search, it looks like some people were able to make it work.

Note that I need update all firmware up to date, virtualization is enabled in bios both CPU and devices

The issue I am experiencing is no matter how I set the H730 mini (raid or HBA mode), once I do a passthrough, I don't see the devices (HDDs) in ESXi, they disappear. The way it works, when I try to make passthrough active, it says it can't, but after the next reboot, it sets the passthrough to active and I can't see the HDDS.

ESXI is installed on USB to make sure it's not affecting this

Any feedback would be appreciated, can provide screenshots and more details.

Thank you in advance
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
Er .. if you passthrough the HBA it is supposed to be invisible for ESXi. It is supposed to be presented as the real hardware to the FreeNAS VM. And then all the disks should be visible in FreeNAS but of course not in ESXi.
 

Dave Hob

Contributor
Joined
Jun 15, 2014
Messages
106
Er .. if you passthrough the HBA it is supposed to be invisible for ESXi. It is supposed to be presented as the real hardware to the FreeNAS VM. And then all the disks should be visible in FreeNAS but of course not in ESXi.

OMG, freaking ey

so I didn't have an issue than, just lack of knowledge and experience :)
 

Dave Hob

Contributor
Joined
Jun 15, 2014
Messages
106
The H730 is a RAID controller IIRC. Whether or not it has "HBA mode" is irrelevant... please refer to

https://www.ixsystems.com/community...bas-and-why-cant-i-use-a-raid-controller.139/

per the article provided it says:

If you cannot see the type of device (such as "ST6000DX000-1H217Z" in "camcontrol devlist", you DO NOT HAVE A TRUE HBA

Looks like the camcontrol devlist and smartctl look fine. Does that mean that there's a chance for the card to be passing access well directly to the hdds?

hba.PNG
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
You can't just pick one point out of that article. See points 5) and 6). If your card isn't using the mps or mpt drivers, that's likely to be a problem.
 

Dave Hob

Contributor
Joined
Jun 15, 2014
Messages
106
You can't just pick one point out of that article. See points 5) and 6)

not my intention, wanted to make sure I'm not missing something

If your card isn't using the mps or mpt drivers, that's likely to be a problem.

is this an absolute statement, or I need to check what drivers is my card using?

Thank you for the feedback.
 

Dave Hob

Contributor
Joined
Jun 15, 2014
Messages
106
I also came across this thread discussed in 2016, on the same setup


does this look more promising in using this card? hope I'm not annoying anyone, at least "too much"
 

Dave Hob

Contributor
Joined
Jun 15, 2014
Messages
106
I guess my only option (to my knowledge) is to get the H330 mini monolithic flashed to it mode.

Can someone suggest if that H330 mini will be able to handle all my hard drives? Front 12 caddies, rear 2 caddies + option to add 4 internal.

I can't figure out where can I connect an ssd on this machine that is not going through the Raid Controller card, since on ESXI, once I do passthrough all my drives disappear? I did order a pci e - nvme combo but is not here yet.

thank you
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
I guess my only option (to my knowledge) is to get the H330 mini monolithic flashed to it mode.

Can someone suggest if that H330 mini will be able to handle all my hard drives? Front 12 caddies, rear 2 caddies + option to add 4 internal.

I can't figure out where can I connect an ssd on this machine that is not going through the Raid Controller card, since on ESXI, once I do passthrough all my drives disappear? I did order a pci e - nvme combo but is not here yet.

thank you

There's no guarantee that a server built by one of the Big Boys is going to be optimized for FreeNAS, much less for a hyperconverged system. Most of them are designed with the expectation that you are running Windows on a big RAID5 and have wired up the system with that end goal in mind, so everything is designed and wired to be run by a single RAID card.

I've refurbed a bunch of 12-bay Dell R510's, which have no obvious attachment abilities for a FreeNAS boot device, aside from maybe USB. You can put a second Dell HBA and an M.2-to-SATA converter to power some SSD's (because there's literally nowhere else to tap power without soldering). The damn Dell R510's actually *DISABLE* their internal SATA ports on a 12-bay chassis on the theory "there's nowhere to put other drives."

This is why all the hypervisors I have are specifically designed for the purpose. ESXi runs off a dedicated LSI 9271-8i and a bunch of SSD's installed on an internal sled and/or M.2-to-SATA cards, and the front bays are hooked to the PCH SATA or an LSI HBA. This is part of the power of going Supermicro, they are not imposing a particular design on you.

For example, even though it isn't designed for this, I needed some low power platforms, so I like to put an X10SDV-7TP4F into a SC216BA-R920LPB . The mainboard has a 16 port HBA on it which is great for FreeNAS, so that gets wired to the last 16 bays. The first 8 bays get wired to an LSI 9271CV-8i for ESXi VM storage. The other PCIe slot gets a dual 10G card to supplement the onboard 10GbE, giving the 4x10GbE design that all our hypervisors use here.

You can't *FIND* anything like that from the Big Boys, because they all have their own ideas about how to do things, and most of those ideas do not include two completely separate disk I/O subsystems. It is likely you will have to make some suboptimal tradeoffs, and it is entirely possible that there is no good solution.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
An M.2 2242 to USB case, either SATA or NVME and a small SSD should be good for booting, though?
 

Dave Hob

Contributor
Joined
Jun 15, 2014
Messages
106
First of all thank you all/both of you for taking the time to deal with my newbie questions, and answered them in such great detail

I definitely understand more than when I started this project. The goal here was to learn something new, more of a home lab testing and learning. By accident, came across this unit on craigslist (I should know better not to look for anything similar in the future :) ) and I think I got it from a screaming deal unless I'm missing something ($600 + tax)

Since it's not a production unit and I have it, I am thinking to try to make it work for now, else I would sell it again on craigslist and try to find something more FREENAS friendly.

I am to receive this PCIe NVME M2 adapter tomorrow and I assume that it might work to boot the system from? right?

https://www.amazon.com/gp/product/B074ZV31CF/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1

Do you think that using a usb to SSd adapter + ssd drive will work for booting ESXi from?

Asked the guy from Art of Server about an IT mode controller and he recommended this one:

Dell H330 mini monolithic (=9300-8i) w/ P16 IT mode HBA330 ZFS FreeNAS unRAID

* anyone has any experience on this and how it can handle all the drives? If I use this IT mode card, would the risks of running into issues drop considerably?

The general question of running Freenas under ESXI. Since the virtualized FREENAS is used as backup for the main one that's installed directly on the hardware, how big are the risks? If something goes bad will the Virtualized FREENAS let me know or it's very probable that it will not let me know within those 2 weeks of replication storage?

Thank you all again for all the help and assistance.
 

Dave Hob

Contributor
Joined
Jun 15, 2014
Messages
106
I was able to install ESXI on the ssd through the USB adapter, but it's not letting me access the VMFS partition, so unless I am missing anything looks like that's no a option.

I have another R420 poweredge and bought a DVD ssd caddie replacement and you are correct, it's not letting me access the ssd once there another hba controller there (using the H310 mini flashed to it mode). It's not like they blocked that sata port but it's not showing the ssd when it's plug in. I tried resetting the idrac to see if it helps but no luck.

I have a 710 as well and I think I had the same issue with replacing the dvd caddie with the ssd adapter (not showing the ssd).

You are correct this PowerEdges are very restrictive for sure.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,112
I was able to install ESXI on the ssd through the USB adapter, but it's not letting me access the VMFS partition, so unless I am missing anything looks like that's no a option.

ESXi won't mount or create VMFS on USB devices by default - you can force it to, but it won't like it.

I have another R420 poweredge and bought a DVD ssd caddie replacement and you are correct, it's not letting me access the ssd once there another hba controller there (using the H310 mini flashed to it mode). It's not like they blocked that sata port but it's not showing the ssd when it's plug in. I tried resetting the idrac to see if it helps but no luck.

I have a 710 as well and I think I had the same issue with replacing the dvd caddie with the ssd adapter (not showing the ssd).

This is as @jgreco pointed out - a large number of (if not all) Dell PE systems have their onboard SATA ports disabled entirely in the denser chassis configurations. Don't expect a solution unfortunately.

You are correct this PowerEdges are very restrictive for sure.

The way Dell will have you booting this is either off the IDSDM (Internal Dual SD Module) or the BOSS (Boot Optimized Storage Solution) which is a fancy (and often overpriced) dual-SATA-M.2 to PCIe RAID card. The former is USB, and will have the same VMFS issues; so you might as well stick with the USB-to-SATA adaptor (and I think I've seen some users reporting trouble with getting it to boot FreeBSD) - the latter is often overpriced but if you can find it cheap as a used-pull makes a great single-card solution for a RAID1 ESXi boot in a single slot.

Did your NVMe adaptor card work for booting the system? I don't think the R730s are locked out of generic NVMe EFI boot.

Edit: Oh, and the H330 flashed to HBA330 is a fine solution for drives. Don't use an unflashed H330 though as they have the same poor queue depth that plagued the H310. H730 in pure HBA mode (as in, forced to run there with zero RAID capabilities) should use the mrsas driver and work as well, but the HBA330 is guaranteed to always be in that mode.
 

Dave Hob

Contributor
Joined
Jun 15, 2014
Messages
106
I am very frustrated today, as I was excited to receive the PCI NVME M2 ssd adapter today, I see that I am not able to select that for boot option. I was able to see the drive and installed ESXI on it, but it's not giving me the option to boot from it

I bought this adapter:

Vantec M.2 NVMe SSD PCIe x4 Adapter (UGT-M2PC100)

and this 256gb ssd

Samsung 970 EVO Plus SSD 250GB - M.2 NVMe

If anyone has any experience on getting OS booted from another PCIe adapter, please advice (feeling very discouraged at the moment)

Thank you
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
I'd suggest booting from USB, then. ESXi actually does pretty well at this. Use the USB for boot and the NVMe for datastore.
 

Dave Hob

Contributor
Joined
Jun 15, 2014
Messages
106
I'd suggest booting from USB, then. ESXi actually does pretty well at this. Use the USB for boot and the NVMe for datastore.

After I took a few minutes to calm down from my frustration figured out too that that could be an issue. giving it a try now. it should work. Not what I expected but probably don't matter much.

If someone has any experience with some server grade pcie nvme ssd option to boot on the poweredge I would be very interested
 

Dave Hob

Contributor
Joined
Jun 15, 2014
Messages
106
It works BUT it doesn't like the adapter or M2 ssd drive and turns the fans on at high speed and becomes very noisy.

So I guess in a way is more correct to say that it doesn't work
 

Diver206

Cadet
Joined
Aug 23, 2020
Messages
3
It works BUT it doesn't like the adapter or M2 ssd drive and turns the fans on at high speed and becomes very noisy.

So I guess in a way is more correct to say that it doesn't work
I just purchased a near identical rig, R730xd 16+2 LFF with the H730 mini, so this is all very familiar. I installed the same NVMe drive in a PCIe adapter. The ramped up fans when using an unsupported device is expected behavior. I found a video that walked me through setting the fan speed to a static value manually to quiet them down. I set them to 60%, not great but quieter than 100% for sure. There are also scripts that will take that a step further and allow the speed to increase when a certain temp is reached rather than remain at the static value (percentage). I have not gotten there yet but some googling should point you there.

The Dell Internal dual SS module (IDSDM) can be found on eBay relatively inexpensive as can Dell 16GB SD cards. It’s plug and play as it’s Dell and will be visible in the bios. This is the route I went to boot ESXi, with the cards set to mirrored in the bios. The installation was painless and Dell has a video on the whole process, including the ESXi install.

The reason I installed the Samsung 970 m.2 was to confirm my hope that the drives would show hen the controller was attached to the vm. My intent was (and still is) to create the FreeNAS vm on the SSDs in the flex bay, but because they are attached to the controller that was passed through and not visible to ESXi, I needed somewhere else to install it and I already had that drive handy.

I was able to create a Datastore with the NVMe drive and create the FreeNAS vm on it. I attached the passed through controller (I had already swapped out the H730 mini for an HBA330 mini mono (which I just read here is not ideal, bummer) when I couldn’t see the drives in ESXi after passing it through (because I too was new and confused). After installing FreeNAS, I confirmed all the drives were visible. I did have to open the GUI in a different browser to get the drive info to populate. At first it only showed the serial numbers but no sizes or anything else. After the info populated, it was visible in all other browsers.

Because I ultimately want FreeNAS to run on the 2 rear flex bay SSD drives (mirrored), I purchased another H330 (card, not mini mono). Unfortunately, the description in the listing was wrong and I received the HBA330 instead, and I want the drives presented to ESXi as a mirror so I need the RAID. I’m waiting on the correct one now.

In the meantime, just to make sure everything will eventually work, I installed the HBA card in an open riser slot. I disconnected the cable that connects the rear backplane to the front backplane at the front end and rerouted it to the HBA card. This causes a warning at pre-boot that that the cable is disconnected, two actually - one for each end. It can be bipassed by pressing F2 to enter setup from said warning screen and then just exiting setup. It will continue to boot to ESXi just fine, and the drives are visible and available in ESXi. It will also cause the amber status light to stay on, but from what I have read and noticed, it has no other effect. I did try removing the sensor cable from the rear backplane in an attempt to circumvent the error, but it had no effect. So I’ll live with the error and no blue light even though it bugs me.

I now have no reason to believe my original configuration plan won’t work when the correct card arrives later this week.
This will be an AIO server with other VMs (Plex, lab, etc. in ESXi, not FreeNAS). I think I might keep the NVMe drive in it and use it as the OS drive for the other VMs. I will have to implement the fan control script to make that a viable option.

I know this was lengthy, but I wanted to share the whole process because it was so recent and similar. I know it would have helped me a lot. I realize you may have already found your way in the last few weeks. If not, hopefully some of my experience will be useful to you. Hopefully it will help at least someone.
 
Top