Dell R740xd2 Truenas Questions

[davidano2]

Hello TrueNas people,

I am building my first TrueNas server and need some input. I did a good amount of research before purchasing any hardware and thought I was ready, but now that I actually have the server I realize I was never ready. I have a fairly good understanding of the ZFS pools / vdevs, but some of the hardware compatibility with ZFS is what's scarring me.


Use Case: Offsite Veeam Backup Target - We have 3 office locations in the US, we will be placing this colo facility to act as an offsite backup target for our 3 offices. Estimated 250TB of space needed.


Hardware Purchased:

• Dell R740xd2 (2U server with 24 x 3.5" hard drive bays)
• 2 x Intel Silver 4116 12C 2.1Ghz 16.5M DDR4-2400 85W CPUs
• 4 x Dell 16GB DDR4 ECC RDIMM 3200Mhz RAM
• Dell PERC H330 Mini Mezzanine Controller
• On-board- Broadcom 5720 Dual Port Gigabit Ethernet
• Dell Broadcom 57416 Dual Port 10GbE SFP+ LOM
• Dell Dual SD Internal Module
• 2 x Dell 1100W 80 Plus Platinum Efficiency PSUs

HBA Adapter Questions

I understand the basics of ZFS well enough to know the OS needs direct access to the drives and to avoid RAID at all cost, got that part.

I have read that the PERC H330 can be flashed to a dumbed down HBA (IT mode), but every time I see that mentioned they reference this article from Serve The Home that seems to have been removed - https://forums.servethehome.com/ind...o-hba330-12gbps-hba-it-firmware.25498/page-17

Currently I have the RAID card set to HBA mode (not flashed, just changed the mode in the GUI) and all caching is disabled by the RAID card. When I run the Smartctl or camcontrol commands in TrueNas I can see all of the drive information like it is hooked up through a standard HBA. Through my research the main issue most people seem to have with running the H330 card like this is that TrueNas uses the wrong driver and it can impact performance.

Since the article has been removed I'm wondering if the driver issue has been fixed and a re-flash is no longer required? Regardless, this will be receiving data over a WAN connection, so I don’t imagine the drive performance will be a real bottleneck. Are there any data integrity concerns of running the PERC H330 with IR firmware set to HBA mode on the latest version of TrueNas?

Another issue I have heard of is that the PERC card may not get enough PCI lanes. This guy was running all NVME drives though, and I will be using spinning disks. I don’t think the PCI lanes will be an issue in my case, but I do plan to use all 24 drive bays, any thoughts?

If your plan is to eventually use all 24 bays, make sure you get two of the 16X PCI Express expander cards from Dell or get two of the 9400 "tri-mode" internal Broadcom HBAs. The maximum number of PCI Express lanes that you can get in this chassis from Dell is 96. From the factory, Dell configures this back plane with four cables connected to a pair of dual port 16X PCI Express expander cards which give you a total of 32 lanes max.
From <https://www.truenas.com/community/t...icro-superserver-a-server-2113s-wn24rt.88867/>


After reading about the PERC firmware issue AND the PCI lane issue, I thought "to hell with this" I will just replace the card with a true LSI HBA card, but after opening this server up the connection to the PERC card appears to be proprietary. On top of that because of the form factor of the R740XD2 there isn't a whole lot of room to add expansion cards. I'm sure it could be done, but I would like to stick with the PERC card as long as I won't be committing a huge TrueNas Sin.



Drives?

I am planning to use 16-20tb drives to achieve the pool size I need and leave some room for future expansion. I have seen several posts warning against SMR drives, got it, don’t use SMR.

Does anyone have any recommendations for large capacity spinning drives that will work well with ZFS / TrueNas? I was leaning towards the Seagate Exos x20, Exos x18 series drives. Looking at the specs these appear to be CMR drives. Does anyone have an opinion on these drives?



Proper Shutdown Procedure

Ok this is a super noob question. Is there any procedure that needs to be completed before shut down (like a VMware equivalent of maintenance mode to prevent data loss). Do I need to dis-mount the pool, or do anything to prevent active writes before shutting down? Or is it really just as simple as clicking the power icon in the Gui?



Backups / Catastrophic Server Failure Planning

What is the best way to backup my TrueNas server, or is that even required? When I am referring to backups, I don’t mean backing up the actual data that resides on the storage pools, I mean a backup of the OS configuration (boot drives). Let's say in some hypothetical scenario my boot drives (dual SD cards) fail and I need to re-build the server. How should I prepare for that?

Another hypothetical, let's say this physical server gets damaged unexpectedly; but all of the drives stay in tact. How would I go about moving this pool to a new server? Is it as simple as moving all of the drives to a new server and TrueNas will automatically recognize/import the pool?




Thank you all in advance!

 

[Ericloewe]

• 2 x Intel Silver 4116 12C 2.1Ghz 16.5M DDR4-2400 85W CPUs
• 4 x Dell 16GB DDR4 ECC RDIMM 3200Mhz RAM

I sort of expect that to result in weird bottlenecks. Running only two memory channels per CPU, with two CPUs, at the low speed supported by a Xeon Silver 41xx does not sound like fun. If you see weird performance issues, consider upgrading with another four DIMMs.

I have read that the PERC H330 can be flashed to a dumbed down HBA (IT mode), but every time I see that mentioned they reference this article from Serve The Home that seems to have been removed - https://forums.servethehome.com/ind...o-hba330-12gbps-hba-it-firmware.25498/page-17

You should've bought the HBA330 instead. Still an option if you don't feel like crossflashing it yourself. You're not crazy, I remember that thread and it does seem to be gone. Sort of looks like the author was banned, which isn't crazy given some of his more colorful comments. Don't know where else to find the instructions...

Another issue I have heard of is that the PERC card may not get enough PCI lanes. This guy was running all NVME drives though, and I will be using spinning disks. I don’t think the PCI lanes will be an issue in my case, but I do plan to use all 24 drive bays, any thoughts?

Not a concern in the slightest with SAS/SATA.

After reading about the PERC firmware issue AND the PCI lane issue, I thought "to hell with this" I will just replace the card with a true LSI HBA card, but after opening this server up the connection to the PERC card appears to be proprietary. On top of that because of the form factor of the R740XD2 there isn't a whole lot of room to add expansion cards. I'm sure it could be done, but I would like to stick with the PERC card as long as I won't be committing a huge TrueNas Sin.

The HBA-side connector is proprietary, but the other end is standard SFF-8643 (at least it is on the Gen 13s, it may be a newer connector in the Gen 14s). Dell also sells cables in the right length, most likely, but generic cables would work. That said, the HBA330 is fine since it uses basically LSI firmware with the header changed to satisfy the Dell firmware's checks.

Does anyone have any recommendations for large capacity spinning drives that will work well with ZFS / TrueNas? I was leaning towards the Seagate Exos x20, Exos x18 series drives. Looking at the specs these appear to be CMR drives. Does anyone have an opinion on these drives?

All HDDs suck, but they're a sad necessity. Get whatever's CMR and meets your requirements (careful with the power disable nonsense pseudo-feature) and confirm with the datasheet.

Ok this is a super noob question. Is there any procedure that needs to be completed before shut down (like a VMware equivalent of maintenance mode to prevent data loss). Do I need to dis-mount the pool, or do anything to prevent active writes before shutting down? Or is it really just as simple as clicking the power icon in the Gui?

It is mostly that simple, unless clients are actively doing something that wouldn't take well to being turned off, or VMs or some such thing.

What is the best way to backup my TrueNas server, or is that even required? When I am referring to backups, I don’t mean backing up the actual data that resides on the storage pools, I mean a backup of the OS configuration (boot drives). Let's say in some hypothetical scenario my boot drives (dual SD cards) fail and I need to re-build the server. How should I prepare for that?

Just export a config backup and store it well (outside your server, don't let that bite you).

Another hypothetical, let's say this physical server gets damaged unexpectedly; but all of the drives stay in tact. How would I go about moving this pool to a new server? Is it as simple as moving all of the drives to a new server and TrueNas will automatically recognize/import the pool?

Yes, it is that simple if you include the OS cards. If not, you'll want to import your config to get back up and running. If you just need to desperately save your data, you don't even need that, as long as you're not using encryption. Things get trickier with encryption.

 

[Arwen]

...
Currently I have the RAID card set to HBA mode (not flashed, just changed the mode in the GUI) and all caching is disabled by the RAID card. When I run the Smartctl or camcontrol commands in TrueNas I can see all of the drive information like it is hooked up through a standard HBA. Through my research the main issue most people seem to have with running the H330 card like this is that TrueNas uses the wrong driver and it can impact performance.

Since the article has been removed I'm wondering if the driver issue has been fixed and a re-flash is no longer required? Regardless, this will be receiving data over a WAN connection, so I don’t imagine the drive performance will be a real bottleneck. Are there any data integrity concerns of running the PERC H330 with IR firmware set to HBA mode on the latest version of TrueNas?
...

The problem with RAID firmware, is that it could perform elevator seeks and writes. This prevents ZFS from writing in-order for it's designed copy on write.

People say, "But my ZFS NAS worked for months and months, even years with RAID firmware in JBOD mode!"

And that is the exact problem. People want it to work and it APPEARS to work perfectly. Except on rare occasions, especially during power loss when writing, out of order writes will corrupt a ZFS pool. Or maybe just some data.


ZFS was purposefully designed to survive hundreds, even thousands of un-expected power cycles, Even unlimited un-expected power cycles. WITHOUT data loss. Of course any data being written at the time of the power loss, would be gone. But, that applies to any file system. The difference is that ZFS won't corrupt any existing data.

With software RAID or disk controllers, (using hardware RAID), those in-order writes are no longer a guarantee. Throwing out one of ZFS' best write consistency methods.


That said, will it ever bite you?
No telling.

But, if you care about your data, don't use IR firmware...

 

[davidano2]

@Ericloewe Thank you very much for your response!

I sort of expect that to result in weird bottlenecks. Running only two memory channels per CPU, with two CPUs, at the low speed supported by a Xeon Silver 41xx does not sound like fun. If you see weird performance issues, consider upgrading with another four DIMMs.

This system will be used for backup storage only, and will only be receiving data over the WAN. Even when backing up over the LAN the bottleneck is 99% at the source. Fingers crossed performance shouldn't be an issue, but I will keep the RAM upgrade in the bank of my mind.

You should've bought the HBA330 instead. Still an option if you don't feel like crossflashing it yourself. You're not crazy, I remember that thread and it does seem to be gone. Sort of looks like the author was banned, which isn't crazy given some of his more colorful comments. Don't know where else to find the instructions...

Unfortunately I was buying the server refurbished and this was the only configuration option. I will see if anyone else has a copy of the instructions, maybe i can find it on the wayback machine.

The HBA-side connector is proprietary, but the other end is standard SFF-8643 (at least it is on the Gen 13s, it may be a newer connector in the Gen 14s). Dell also sells cables in the right length, most likely, but generic cables would work. That said, the HBA330 is fine since it uses basically LSI firmware with the header changed to satisfy the Dell firmware's checks.

Thanks, I will take another look, might end up swapping to a true HBA card instead. From what I read flashing the H330 card over seemed pretty complicated.

 

[davidano2]

The problem with RAID firmware, is that it could perform elevator seeks and writes. This prevents ZFS from writing in-order for it's designed copy on write.

People say, "But my ZFS NAS worked for months and months, even years with RAID firmware in JBOD mode!"

And that is the exact problem. People want it to work and it APPEARS to work perfectly. Except on rare occasions, especially during power loss when writing, out of order writes will corrupt a ZFS pool. Or maybe just some data.


ZFS was purposefully designed to survive hundreds, even thousands of un-expected power cycles, Even unlimited un-expected power cycles. WITHOUT data loss. Of course any data being written at the time of the power loss, would be gone. But, that applies to any file system. The difference is that ZFS won't corrupt any existing data.

With software RAID or disk controllers, (using hardware RAID), those in-order writes are no longer a guarantee. Throwing out one of ZFS' best write consistency methods.


That said, will it ever bite you?
No telling.

But, if you care about your data, don't use IR firmware...

Thank you for the response, this is exactly why I decided to ask.

Looks like a new HBA card or a firmware flash it is!

 

[beagle]

I haven’t tried myself but there is a YT video explaining how to perform the crossflash:

How to crossflash Dell H330 to IT mode firmware

In this video, I'm going to show you how to flash a Dell H330 MegaRAID controller with HBA IT mode firmware from the HBA330 controller. This method is brough...

www.youtube.com

He even sells the cross flashed cards on eBay.

There seems to be a version also uploaded on GDrive:

Flash Dell H330 Full Write Up with Pics v3.0.pdf

drive.google.com

 

[Davvo]

The wayback machine saves the day!

Flash/Crossflash DELL H330 RAID Card to HBA330/12Gbps HBA IT Firmware

Please Help!! The card seems to be stuck at this level. I don't know where to lead from here.

web.archive.org

 

[davidano2]

I haven’t tried myself but there is a YT video explaining how to perform the crossflash:

How to crossflash Dell H330 to IT mode firmware

In this video, I'm going to show you how to flash a Dell H330 MegaRAID controller with HBA IT mode firmware from the HBA330 controller. This method is brough...

www.youtube.com

He even sells the cross flashed cards on eBay.

The wayback machine saves the day!

Flash/Crossflash DELL H330 RAID Card to HBA330/12Gbps HBA IT Firmware

Please Help!! The card seems to be stuck at this level. I don't know where to lead from here.

web.archive.org

Thanks both, I guess I will take the plunge and attempt the crossflash!