Dell PE R730XD, PCIE NVME m2 recommandation

[Dave Hob]

Hi all

I have this Dell PowerEdge R730XD and looking for run FREENAS on under ESXI.

I am looking for some recommendations to run a compatible PCIE NVME M2 drive (or similar). I would need to figure out which adapter card will work and what ssd, if anyone has any experience

Ideally would love to have a boot option as well, if possible (since from all the google searches, it's not looking promising, but who knows maybe someone had some luck).

At the moment I did purchased and tried the following adapter and drive with no luck (ESXi sees the drive but the Power Edge doesn't like it and turns the fans on at 30% - 50%):

Vantec M.2 NVMe SSD PCIe x4 Adapter (UGT-M2PC100)

Samsung 970 EVO Plus SSD 250GB - M.2 NVMe

I need this to install the FREENAS VM on and after that I think it wil get more straight forward once I pass through the HBA card (which ever it will be)

Thank you in advance.

 

[Dave Hob]

So far I have tried this 2 adapter with no noticeable difference between them:

Rivo PCI-E Riser PCIe M.2 PCIe SSD to PCIe Express 3.0 x4 Adapter Card - Supports M2 NGFF PCI-e 3.0, 2.0 or 1.0, NVMe or AHCI, M-Key, 2280, 2260, 2242, 2230 Solid State Drives

Vantec M.2 NVMe SSD PCIe x4 Adapter (UGT-M2PC100)

The following drives that I tried, were recognized by the OS (ESXI or FREENAS had 2 separate usb tests) but fans turned up high as the machine is not happy with the drives:

Samsung 970 PRO SSD 512GB - M.2 NVMe Interface Internal Solid State Drive with V-NAND Technology
Samsung 970 EVO Plus SSD 250GB - M.2 NVMe Interface Internal Solid State Drive with V-NAND Technology

the following was not recognized by the OS or the machine at all

Samsung 860 EVO SSD 250GB - M.2 SATA Internal Solid State Drive with V-NAND Technology

still hoping that someone might have some experience with this Dell Poweredge R730XD and had some luck with any nvme M2 drives, instead of me continuing to buy and try more models :)

Thank you

 

[subhuman]

the following was not recognized by the OS or the machine at all

Samsung 860 EVO SSD 250GB - M.2 SATA Internal Solid State Drive with V-NAND Technology

Not at all unexpected. There is a difference between a SATA M.2 drive and a PCIe one.

 

[HoneyBadger]

but fans turned up high as the machine is not happy with the drives:

You need to disable the "Third-Party PCIe card Default Cooling response" setting in the firmware. Keep an eye on overall temps when you do this especially if you are packing the system full of drives (which I expect you are, it's an XD after all)

How To Disable The Third-Party PCIe Card Default Cooling Response On PowerEdge 13G Servers. | Dell US

 

[Patrick M. Hausen]

I can recommend running Grafana for that:

 

[Dave Hob]

Not at all unexpected. There is a difference between a SATA M.2 drive and a PCIe one.

didn't even pay attention to that, thank you

You need to disable the "Third-Party PCIe card Default Cooling response" setting in the firmware. Keep an eye on overall temps when you do this especially if you are packing the system full of drives (which I expect you are, it's an XD after all)

The only one I found is for Windows Server. I will attach some screenshots of what I found. Should I assume I am supposed to install Windows server, install the application, and then from the windows command line run those commands? If I delete the Windows install will this setting remain in the system?

Thank you for all the feedback

 

[HoneyBadger]

The ipmitool command in FreeNAS should do it as well, but before that, try just ssh'ing to the iDRAC interface and running racadm set system.thermalsettings.ThirdPartyPCIFanResponse 0 - I think that's exposed as writeable in the 13G servers. 14G servers (and possibly 13G with a new enough iDRAC + firmware combination) require a slot-by-slot modification which is more complicated, see the PDF link below.

ftp://ftp.dell.com/manuals/common/poweredge_pcie_cooling.pdf

Obligatory warning - certain add-in cards are very demanding on the LFM (Linear Feet per Minute) of cooling they want. If you're stacking lots of high-TDP cards in there (10G networking, multiple HBAs, etc) then consider being conservative with your tweaks. Last thing you want is to introduce stability issues or cook a card. Monitoring your drive temperatures is one thing, monitoring card/CPU is another.

 

[Dave Hob]

All you guys been amazing.

Thank you @HoneyBadger for the ipmitool recommendation. I did follow this tutorial and was able to make that work:

You need to disable the "Third-Party PCIe card Default Cooling response" setting in the firmware. Keep an eye on overall temps when you do this especially if you are packing the system full of drives (which I expect you are, it's an XD after all)

How To Disable The Third-Party PCIe Card Default Cooling Response On PowerEdge 13G Servers. | Dell US

@HoneyBadger on a previous post of mine you mentioned that I would be able to boot from this BOSS dell pcie card:

BOSS (Boot Optimized Storage Solution) which is a fancy (and often overpriced) dual-SATA-M.2 to PCIe RAID card

do you think that one of this will work to have the R730xd boot from it?

Dell PCIe Dual M.2 Solid State Drive Adapter Card JV70F + 2x 256GB SSD

@Patrick M. Hausen thank you for recommending Grafana. I googled it a little bit still trying to figure out how to make that work. I am planning to use the free version of ESXI

As an update I was able to boot on ESXI from USB, the NVME M2 Samsung 970 PRO is available there and installed FREENAS. I set H730 mini to passthrough. Card is in HBA mode.

The problem that I am experiencing is that it's not booting when I give it the add pcie device and the H730 mini.

FREENAS is booting if:
- give the VM 1 vcpu (boots with H730 mini designated to it)
- VM 6 vcpu no H730 mini designated to it

Notes:
- FREENAS works on barebone installation on this machine with no problem (tested)
- I also did the following to ensure the mrsas driver is being used

To ensure the mrsas driver is being used edit /boot/device.hints, add the following line and reboot
Code:
hw.mfi.mrsas_enable="1"

attaching a print screen on the boot issue

You guys been amazing, thank you again for all the help.

 

[Patrick M. Hausen]

@Patrick M. Hausen thank you for recommending Grafana. I googled it a little bit still trying to figure out how to make that work. I am planning to use the free version of ESXI

Sorry, I somehow missed that ESXi part. FreeNAS/TrueNAS has got collectd builtin, so you can send all your health data to e.g. InfluxDB and run Grafana on that. It's trivial to setup in a FreeNAS jail. I don't know how/if ESXi can send data, but a quick search looks promising. Telegraf seems to be the tool of choice to pull data from ESXi and send it to InfluxDB.

I can surely help with the Influx and Grafana part.