Storage hardware for my boot pool on Dell r730xd

[CheeryFlame]

* TLDR at the bottom in quote block

Hello, I would like to get some insight about my boot pool hardware.

I was first using a SATADOM which got corrupted after a cold shutdown thus corrupting my TrueNAS install. I won't consider this solution anymore or USB drives at all.

I temporarily switched to a SAS SSD which after I pressed iDRAC Java Console > Power > Restart System (warm boot), my install got corrupted. For example now I can't update TrueNAS, I'm getting this error:

Or when I restart my server it's taking between 5-15 minutes more than usual at this step:

I've stumbled across more annoying issues like NFS service taking lot of time of start or wouldn't start at all.

Now is that I'd like to get back my 800gb SAS SSD and put it back in my vdev.

The issue here is that I don't have any more drive slots available in the server since it was built using a SATADOM for the boot-pool.

TLDR; I would like to know if I could use this and an NVME drive in a PCI slot of my server (see specs in my signature) as a reliable boot pool once and for all.

Thank you.

 

[jgreco]

I don't think the R730XD supports NVMe boot, so, briefly, no.

You might wish to try what's outlined in the Resources section.

Highly Available Boot Pool Strategy

I've pounded a few versions of this out over the years, but I hate explaining over and over. TrueNAS allows a ZFS boot mirror pool to be created to increase the reliability of the NAS. This sounds great in theory, but there's a flaw. Due to...

www.truenas.com

You can combine an IR-mode HBA card with something like an Addonics card which gives you two SATA M.2 SSD's in RAID1 for booting, along with an NVMe M.2 SSD for ZFS redundancy, taking up two PCIe slots but no drive bays.

 

[CheeryFlame]

I don't think the R730XD supports NVMe boot, so, briefly, no.

You might wish to try what's outlined in the Resources section.

Highly Available Boot Pool Strategy

I've pounded a few versions of this out over the years, but I hate explaining over and over. TrueNAS allows a ZFS boot mirror pool to be created to increase the reliability of the NAS. This sounds great in theory, but there's a flaw. Due to...

www.truenas.com

You can combine an IR-mode HBA card with something like an Addonics card which gives you two SATA M.2 SSD's in RAID1 for booting, along with an NVMe M.2 SSD for ZFS redundancy, taking up two PCIe slots but no drive bays.

I would prefer not to change my current HBA setup since when something is working I like to keep it as-is.

Would this NVME to USB be a reliable option since I can't boot from PCIe?

https://www.amazon.ca/ELUTENG-M-2-NVME-NGFF-Converter/dp/B0BJCYR1P7

Thank you for helping!

 

[CheeryFlame]

something like an Addonics card

Just re-read your post and noticed the link you sent that I haven't seen the first time. So the card in my original post would not work since it would not see the NVME drive from the PCIe lane right?

So this card https://www.amazon.ca/StarTech-com-M-2-Adapter-Port-Express/dp/B01IR05DLK that have been recommended in your post would power the NVME from PCIe lane BUT get the data from the SATA port I would connect to my server motherboard?

Is this correct?

EDIT: I'd also buy one of these which is the cheaper I could find from a reputable brand.

 

[jgreco]

I would prefer not to change my current HBA setup since when something is working I like to keep it as-is.

This isn't a change, it's an addition. I've used this on about a dozen R510's; the units get a PERC H310 in IT mode to replace the H700 in the internal controller slot, and then a PERC H200 in IR mode along with the Addonics card, wired together with a SFF-8087 breakout cable. You are allowed to have more than one HBA in a server.

Would this NVME to USB be a reliable option since I can't boot from PCIe?

It's not as reliable as what I've suggested. USB-to-HDD adapters can be rather quirky because they are often programmed with the same device serial number and/or other weird stuff.

So the card in my original post would not work since it would not see the NVME drive from the PCIe lane right?

It will see it just fine, it's just that it is unlikely to be picked up as a boot option since there's no boot ROM and also no support in the host BIOS to boot from NVMe.

So this card https://www.amazon.ca/StarTech-com-M-2-Adapter-Port-Express/dp/B01IR05DLK that have been recommended in your post would power the NVME from PCIe lane BUT get the data from the SATA port I would connect to my server motherboard?

The Addonics/StarTech card powers three SSD's, two SATA and one NVMe. It connects the NVMe to the PCIe bus, but you need a controller of some sort to handle the SATA. That can be your mainboard if you like, but that is a less reliable option than the IR-mode HBA technique I discuss in the resource. The IR-mode HBA is a highly reliable device that can fail over from one SATA SSD to the other, but does not preclude the possibility of a ZFS block checksum. The addition of a third (potentially unbootable) SSD provides someplace for ZFS to obtain redundancy from. It depends how much you want the reliability as to whether you go to this level.

 

[CheeryFlame]

The Addonics/StarTech card powers three SSD's, two SATA and one NVMe. It connects the NVMe to the PCIe bus, but you need a controller of some sort to handle the SATA. That can be your mainboard if you like, but that is a less reliable option than the IR-mode HBA technique I discuss in the resource. The IR-mode HBA is a highly reliable device that can fail over from one SATA SSD to the other, but does not preclude the possibility of a ZFS block checksum. The addition of a third (potentially unbootable) SSD provides someplace for ZFS to obtain redundancy from. It depends how much you want the reliability as to whether you go to this level.

I see this as the optimal method although I'm really tight on the budget at the moment. Since I have a daily TrueNAS configuration file backup with multi_report, could I buy the StarTech card with 2x of these M.2 drives and wire them to the server's motherboard available sata ports?

In your resource it states to use 3x drives but I don't understand why it wouldn't work with only 2x plugged in the server's motherboard. Since it's uncommon territory for me I'll re-use your wording and modify it with a 2x drive scenario and let me know if something doesn't make sense;

Use the first as the bootable device in the host BIOS and leave the second as a passthrough device. Now, when you boot up the TrueNAS installer, use the first disk and the second SSD as your mirrored boot device. Now if something goes wrong with the data on the first device, ZFS has a source for redundancy and can correct the error as it finds it, from the data on the second SSD.

Thank you for helping me!

 

[jgreco]

why it wouldn't work with only 2x plugged in the server's motherboard.

Because most mainboards, even server ones, will not be able to recover from a partial failure of the first boot device. It takes some intelligence to decide to take some of the boot image from the first device and some from the second device; an IR mode HBA will do that. Your mainboard, probably not.

 

[NickF]

I would be more inclined to recommend one of these:

https://www.amazon.ca/StarTech-com-SATA-Drive-Adapter-Cable/dp/B00HJZJI84/ref=sr_1_14?crid=22S19ZDFIP87I&keywords=usb+to+sata&qid=1683587922&s=electronics&sprefix=usb+to+sata%2Celectronics%2C65&sr=1-14

Coupled with 120GB Sata SSD from your dealer of choice.

I've had a couple dozen of these deployed for years with good success, I haven't had one die yet. Only issue I had was one of the SSDs came unplugged...whoops

SATADOMs are no better than USB thumb drives..

 

[CheeryFlame]

Because most mainboards, even server ones, will not be able to recover from a partial failure of the first boot device. It takes some intelligence to decide to take some of the boot image from the first device and some from the second device; an IR mode HBA will do that. Your mainboard, probably not.

I found this one that states it can do IR mode, would it be safe to order expecting it to do what discussed in resources? It's the cheapest I could find.

 

[jgreco]

In theory it should be fine. I've never had to work with one of the Fuji's though, so I don't know if there are special gotchas like there are with the Dells.

 

[CheeryFlame]

In theory it should be fine. I've never had to work with one of the Fuji's though, so I don't know if there are special gotchas like there are with the Dells.

What about those special gotchas? Maybe I can try and dig more about it.

 

[jgreco]

For the Dell cards, you have to downgrade them with the Dell firmware to P7, then take them over it IT mode, and then upgrade them. There are a variety of steps that need to be followed.

 

[CheeryFlame]

For the Dell cards, you have to downgrade them with the Dell firmware to P7, then take them over it IT mode, and then upgrade them. There are a variety of steps that need to be followed.

Good to know, I'll try not to buy a Dell one.

I've found this one https://www.ebay.ca/itm/195577453895

Which as more sold, more reviews and have the breakout cables included which isn't the case with the other listing.

Although it's in IT mode which I understand is software managed which isn't what we want in this case. IR mode if I understood your resource link would be accessible from the boot sequence so I could setup the raid-1 for disks 1+2 and set disk 3 as passthrough.

Now how can I flash the card to IR mode instead of IT mode? I already sent a message to the seller if he could do it, but in case he wont, I wonder how and where I can find information about such a process.

Thank you for your reply!

 

[Ericloewe]

In theory it should be fine. I've never had to work with one of the Fuji's though, so I don't know if there are special gotchas like there are with the Dells.

Apparently there are: https://www.truenas.com/community/t...ant-i-use-a-raid-controller.81931/post-759599

 

[Ericloewe]

Now how can I flash the card to IR mode instead of IT mode?

Very easy, just flash the IR firmware instead of the IT firmware. They're both supported in the same configuration, without the more elaborate crossflash procedure.

 

[CheeryFlame]

Very easy, just flash the IR firmware instead of the IT firmware. They're both supported in the same configuration, without the more elaborate crossflash procedure.

I'm just wondering if it's worth savin 10$ on Amazon for the breakout cable. Is the flash this quick and easy? I doubt so hehe, if I buy it already in IR mode I guess it'll be plug and play no?

 

[jgreco]

Very easy, just flash the IR firmware instead of the IT firmware. They're both supported in the same configuration, without the more elaborate crossflash procedure.

Not true, at least for the H200's I've done, I had to downgrade them to P7-IT and then take them over to P7-IR, then up to P16-IR (which was current at the time). This was unfortunate because I was repurposing H200's that had already been previously flashed to IT mode. It's possible there is a shortcut but I only had a dozen cards to do.

 

[Ericloewe]

Hmm, yeah, the Dells may have some extra weirdness going on. I'm your experience, did sas2flash just refuse to work or did the cards not work after flashing IR without jumping through hoops?

 

[CheeryFlame]

I'll take a chance and ask again just in case:

If I buy it already in IR mode I guess it'll be plug and play? Or it needs some kind of special IR firmware?

 

[victort]

What about those special gotchas? Maybe I can try and dig more about it.

SOLVED - Fujitsu D2607 IT mode not seeing Serial Numbers

TL:DR = It wasn’t in IT mode. I have a (above mentioned) LSI megaRAID card that has been flashed to IT mode, and works fine with unRAID, but not TrueNAS. It sees the disks and everything, but not the serial numbers. It also names the disks (mfisyspd0, mfisyspd1, mfisyspd2, mfisyspd3, mfisyspd4...

www.truenas.com

The D2607 card took me about 15 minutes to flash to IT mode following the guide linked to in this thread.

Here is the link to the guide itself. He has all the tools in one zip file.

Cross-Flashing a Fujitsu D2607 RAID Controller

I recently picked up a few Fujitsu D2607 RAID Controllers (cheap off eBay!) to replace the Rocket RAID controller in my NAS, which I have come to distrust as it has at times been a little flaky (plus, they have never provided a single firmware update, and support if asking anything is...

davewinfield.au