Are there anyone running TrueNAS successfully here with Dell R730XD?

[Whattteva]

I'm close to pulling the trigger on a Dell PowerEdge R730XD.
I'm going to probably virtualize TrueNAS since this machine kinda' has an overpowered spec for TrueNAS.
Just in case it's relevant, it's 2x E5-2690 V4 with 128 GB RAM and the storage pool is going to be just 4x 3 TB (will be upgraded to 4x 6TB soon) drives striped mirrors.
I've seen some threads but they're all from around 4-5 years ago. I'd like to get some feedbacks from others here who are currently running a similar setup successfully more recently. Also, if you are running it successfully, I have a few more questions:

1. How long have you been running it.
2. Are you using Proxmox or ESXi or even straight-up TrueNAS CORE bhyve or TrueNAS SCALE KVM?
3. Did you have to replace the H730 controller? If not, did you have to do anything extra to get it to passthrough successfully?
4. This server comes with an iDRAC 8 express. Does upgrading to Enterprise require another card or just a license key?

As I understand it, ESXi is more recommended, but I'm pretty sure there are people out there using Proxmox and Proxmox is what I'm already familiar and currently running now on another system. I also may look into trying out TrueNAS SCALE KVM (I've only ever run CORE up to now).
Past threads seem to not recommend H730, but I do see it on the FreeBSD 13.0 HW compatibility list, so again, I'm fishing for more recent experience.

Finally, I do daily drive two workstation machines running vanilla FreeBSD and a Linux Mint respectively so I am very familiar with both nix systems so if all else fails, I could probably just run a straight-up vanilla Debian host, but would rather avoid having to do all that manual management.

Much appreciated for the feedback.

 

[HoneyBadger]

Bear in mind that the virtualization/PCIe passthrough is still considered "experimental" [1] in ProxMox - that's one of the reasons that ESXi is recommended when doing this. If you're familiar with and comfortable in *nix-based systems already, SCALE might be the ideal combination for you.

Regarding your other questions:

3. The H730 should be replaced by an HBA330 or an H330 that's been cross-flashed into one. The H730 uses the mrsas driver which isn't as mature as the mpr used by the latter.
4. Upgrading to Enterprise is just a key - Dell P/N is 385-BBHP

Cheers!

[1] https://pve.proxmox.com/wiki/Pci_passthrough#Introduction

 

[Whattteva]

Hey, thanks for the only lone feedback so far!!!

Bear in mind that the virtualization/PCIe passthrough is still considered "experimental" [1] in ProxMox - that's one of the reasons that ESXi is recommended when doing this. If you're familiar with and comfortable in *nix-based systems already, SCALE might be the ideal combination for you.

I was thinking of doing this, but I have heard rumors about SCALE KVM maybe having some issues. Is this true? Which issues might I expect to run into?

3. The H730 should be replaced by an HBA330 or an H330 that's been cross-flashed into one. The H730 uses the mrsas driver which isn't as mature as the mpr used by the latter.

Bummer, I was hoping I wouldn't have to do this as it's an extra cost. Significant one too at $140 on average at eBay. If you know of a way to get these cheaper, I'm all ears!

4. Upgrading to Enterprise is just a key - Dell P/N is 385-BBHP

That's good to hear.

 

[anaxagorasbc]

so i am in the process of upgrading my ancient supermicro x9 based system to an r730xd. I was using vmware with the supermicro box and pcie passthrough for storage. I started off virtualizing solaris with napp-itt and pcie passthrough for probably 7 years, and then truenas for the last 2 years or so. I've had zero problems in all that time.

With the r730xd i decided to migrate to proxmox since broadcom brought vmware, and broadcom doesn't give me warm fuzzies, also i wanted to try something else. I'm only a couple weeks into the adventure. I've moved all of my vm's over to proxmox except for truenas so far. I have a new truenas vm under proxmox that i'm testing with.

the integrated array controller: I changed that from an h730p to an h330, and cross flashed it to IT mode. I read you don't want to pass the h730 through for various reasons, if you search here you'll find them. The 12 drive front backplane is plugged into the integrated h330, and the 4 drive midplane is daisy chained off of the backplane. The h330 is being pcie passthroughed to my truenas VM. I'm in the load testing phase with some old drives i had hanging around. So far so good.

I added a pcie lsi 9300-8i hba and plugged the rear flex back plane into it. i have the option to pass it through to another vm or use it for my proxmox storage as an additional pool or backup destination. I may virtualize the proxmox backup server and pass it through to that for backups. adding 2nd hba card with the same chipset as the integrated adapter caused some issues with passthrough that i had to solve with a vfio bind mount.

I am booting the system off of a 1tb sata ssd that i plugged into one of the onboard sata ports and got a slimline power adapter for. It's just shoved in the chassis wherever i can find room for it. it's jank, but since no moving parts should be no issue.

my vm storage pools are on a bifurcated 2 port pcie to nvme adapter. i'm being dangerous and running a zfs raid0 here. But with enough backups it's worth the risk to me.

my system came with idrac enterprise, but i think it's a liscense key, i think you can buy them on ebay for like $30.

 

[jgreco]

If you want to run ESXi and then virtualize TrueNAS under it, you can use your H730 to provide storage for the ESXi hypervisor environment. This works fine. You then use some other HBA with PCIe passthru for the ZFS storage.

However, the H730 SHOULD NOT BE USED FOR ZFS STORAGE.

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

 

[Whattteva]

If you want to run ESXi and then virtualize TrueNAS under it, you can use your H730 to provide storage for the ESXi hypervisor environment. This works fine. You then use some other HBA with PCIe passthru for the ZFS storage.

However, the H730 SHOULD NOT BE USED FOR ZFS STORAGE.

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

I see, I guess I will look into purchasing another HBA card. Would this conflict with the current backplane or can you somehow divvy up the backplane among the two HBA cards? Realistically, I only need max of 2 bays for the H730 for the boot pool and I want the rest to go to passthrough.

 

[jgreco]

I see, I guess I will look into purchasing another HBA card. Would this conflict with the current backplane or can you somehow divvy up the backplane among the two HBA cards? Realistically, I only need max of 2 bays for the H730 for the boot pool and I want the rest to go to passthrough.

Urg, I'm not really a Dell guy but I can tell you that Dell and HP both typically expect you to make a big RAID5 array out of their 12-drive/2U systems. I expect the front drives are all on a single expander backplane. That's not really shareable.

Doesn't the R730xd have two 2.5" "Flex Bays" on the rear?

If not, what you might be able to do is to burn a PCIe slot using one of these adapters referenced in this article:

Cool hardware: Addonics AD3M2SPX4 M.2 to PCIe converter

I was a bit bothered by the large number of available SATA ports and one or two free PCIe slots in the VM storage server here. With the boost from 64GB to 128GB of RAM, the 256GB M.2 Samsung XP941 with its AHCI interface for L2ARC was feeling a little dated anyways. M.2 offers several...

www.truenas.com

This can supply power and a mounting spot to two SATA M.2 SSD's, which you then connect to your boot pool controller.

 

[HoneyBadger]

Urg, I'm not really a Dell guy but I can tell you that Dell and HP both typically expect you to make a big RAID5 array out of their 12-drive/2U systems. I expect the front drives are all on a single expander backplane. That's not really shareable.

"Dell guy" here, he's right - the R730XD uses a single expander backplane. A regular R730 non-XD with 8 bays does I believe use a non-expanding one though, split as 4+4.

Doesn't the R730xd have two 2.5" "Flex Bays" on the rear?

It's an optional add-on, yes. More costly than the alternative mentioned below.

If not, what you might be able to do is to burn a PCIe slot using one of these adapters referenced in this article:

Cool hardware: Addonics AD3M2SPX4 M.2 to PCIe converter

I was a bit bothered by the large number of available SATA ports and one or two free PCIe slots in the VM storage server here. With the boost from 64GB to 128GB of RAM, the 256GB M.2 Samsung XP941 with its AHCI interface for L2ARC was feeling a little dated anyways. M.2 offers several...

www.truenas.com

This can supply power and a mounting spot to two SATA M.2 SSD's, which you then connect to your boot pool controller.

This would be a good call. The BOSS card is another option, but note that it seems to have a firmware-level disagreement with FreeBSD drivers now and doesn't like to work in non-RAID mode.

 

[jgreco]

XD

So "XD" stands for... Xtra Drives?

 

[Volts]

So "XD" stands for... Xtra Drives?

No they're totally different. The XD has one fewer USB port.