Hello TrueNas people,
I am building my first TrueNas server and need some input. I did a good amount of research before purchasing any hardware and thought I was ready, but now that I actually have the server I realize I was never ready. I have a fairly good understanding of the ZFS pools / vdevs, but some of the hardware compatibility with ZFS is what's scarring me.
Use Case: Offsite Veeam Backup Target - We have 3 office locations in the US, we will be placing this colo facility to act as an offsite backup target for our 3 offices. Estimated 250TB of space needed.
Hardware Purchased:
HBA Adapter Questions
I understand the basics of ZFS well enough to know the OS needs direct access to the drives and to avoid RAID at all cost, got that part.
I have read that the PERC H330 can be flashed to a dumbed down HBA (IT mode), but every time I see that mentioned they reference this article from Serve The Home that seems to have been removed - https://forums.servethehome.com/ind...o-hba330-12gbps-hba-it-firmware.25498/page-17
Currently I have the RAID card set to HBA mode (not flashed, just changed the mode in the GUI) and all caching is disabled by the RAID card. When I run the Smartctl or camcontrol commands in TrueNas I can see all of the drive information like it is hooked up through a standard HBA. Through my research the main issue most people seem to have with running the H330 card like this is that TrueNas uses the wrong driver and it can impact performance.
Since the article has been removed I'm wondering if the driver issue has been fixed and a re-flash is no longer required? Regardless, this will be receiving data over a WAN connection, so I don’t imagine the drive performance will be a real bottleneck. Are there any data integrity concerns of running the PERC H330 with IR firmware set to HBA mode on the latest version of TrueNas?
Another issue I have heard of is that the PERC card may not get enough PCI lanes. This guy was running all NVME drives though, and I will be using spinning disks. I don’t think the PCI lanes will be an issue in my case, but I do plan to use all 24 drive bays, any thoughts?
If your plan is to eventually use all 24 bays, make sure you get two of the 16X PCI Express expander cards from Dell or get two of the 9400 "tri-mode" internal Broadcom HBAs. The maximum number of PCI Express lanes that you can get in this chassis from Dell is 96. From the factory, Dell configures this back plane with four cables connected to a pair of dual port 16X PCI Express expander cards which give you a total of 32 lanes max.
From <https://www.truenas.com/community/t...icro-superserver-a-server-2113s-wn24rt.88867/>
After reading about the PERC firmware issue AND the PCI lane issue, I thought "to hell with this" I will just replace the card with a true LSI HBA card, but after opening this server up the connection to the PERC card appears to be proprietary. On top of that because of the form factor of the R740XD2 there isn't a whole lot of room to add expansion cards. I'm sure it could be done, but I would like to stick with the PERC card as long as I won't be committing a huge TrueNas Sin.
Drives?
I am planning to use 16-20tb drives to achieve the pool size I need and leave some room for future expansion. I have seen several posts warning against SMR drives, got it, don’t use SMR.
Does anyone have any recommendations for large capacity spinning drives that will work well with ZFS / TrueNas? I was leaning towards the Seagate Exos x20, Exos x18 series drives. Looking at the specs these appear to be CMR drives. Does anyone have an opinion on these drives?
Proper Shutdown Procedure
Ok this is a super noob question. Is there any procedure that needs to be completed before shut down (like a VMware equivalent of maintenance mode to prevent data loss). Do I need to dis-mount the pool, or do anything to prevent active writes before shutting down? Or is it really just as simple as clicking the power icon in the Gui?
Backups / Catastrophic Server Failure Planning
What is the best way to backup my TrueNas server, or is that even required? When I am referring to backups, I don’t mean backing up the actual data that resides on the storage pools, I mean a backup of the OS configuration (boot drives). Let's say in some hypothetical scenario my boot drives (dual SD cards) fail and I need to re-build the server. How should I prepare for that?
Another hypothetical, let's say this physical server gets damaged unexpectedly; but all of the drives stay in tact. How would I go about moving this pool to a new server? Is it as simple as moving all of the drives to a new server and TrueNas will automatically recognize/import the pool?
Thank you all in advance!
I am building my first TrueNas server and need some input. I did a good amount of research before purchasing any hardware and thought I was ready, but now that I actually have the server I realize I was never ready. I have a fairly good understanding of the ZFS pools / vdevs, but some of the hardware compatibility with ZFS is what's scarring me.
Use Case: Offsite Veeam Backup Target - We have 3 office locations in the US, we will be placing this colo facility to act as an offsite backup target for our 3 offices. Estimated 250TB of space needed.
Hardware Purchased:
- Dell R740xd2 (2U server with 24 x 3.5" hard drive bays)
- 2 x Intel Silver 4116 12C 2.1Ghz 16.5M DDR4-2400 85W CPUs
- 4 x Dell 16GB DDR4 ECC RDIMM 3200Mhz RAM
- Dell PERC H330 Mini Mezzanine Controller
- On-board- Broadcom 5720 Dual Port Gigabit Ethernet
- Dell Broadcom 57416 Dual Port 10GbE SFP+ LOM
- Dell Dual SD Internal Module
- 2 x Dell 1100W 80 Plus Platinum Efficiency PSUs
HBA Adapter Questions
I understand the basics of ZFS well enough to know the OS needs direct access to the drives and to avoid RAID at all cost, got that part.
I have read that the PERC H330 can be flashed to a dumbed down HBA (IT mode), but every time I see that mentioned they reference this article from Serve The Home that seems to have been removed - https://forums.servethehome.com/ind...o-hba330-12gbps-hba-it-firmware.25498/page-17
Currently I have the RAID card set to HBA mode (not flashed, just changed the mode in the GUI) and all caching is disabled by the RAID card. When I run the Smartctl or camcontrol commands in TrueNas I can see all of the drive information like it is hooked up through a standard HBA. Through my research the main issue most people seem to have with running the H330 card like this is that TrueNas uses the wrong driver and it can impact performance.
Since the article has been removed I'm wondering if the driver issue has been fixed and a re-flash is no longer required? Regardless, this will be receiving data over a WAN connection, so I don’t imagine the drive performance will be a real bottleneck. Are there any data integrity concerns of running the PERC H330 with IR firmware set to HBA mode on the latest version of TrueNas?
Another issue I have heard of is that the PERC card may not get enough PCI lanes. This guy was running all NVME drives though, and I will be using spinning disks. I don’t think the PCI lanes will be an issue in my case, but I do plan to use all 24 drive bays, any thoughts?
If your plan is to eventually use all 24 bays, make sure you get two of the 16X PCI Express expander cards from Dell or get two of the 9400 "tri-mode" internal Broadcom HBAs. The maximum number of PCI Express lanes that you can get in this chassis from Dell is 96. From the factory, Dell configures this back plane with four cables connected to a pair of dual port 16X PCI Express expander cards which give you a total of 32 lanes max.
From <https://www.truenas.com/community/t...icro-superserver-a-server-2113s-wn24rt.88867/>
After reading about the PERC firmware issue AND the PCI lane issue, I thought "to hell with this" I will just replace the card with a true LSI HBA card, but after opening this server up the connection to the PERC card appears to be proprietary. On top of that because of the form factor of the R740XD2 there isn't a whole lot of room to add expansion cards. I'm sure it could be done, but I would like to stick with the PERC card as long as I won't be committing a huge TrueNas Sin.
Drives?
I am planning to use 16-20tb drives to achieve the pool size I need and leave some room for future expansion. I have seen several posts warning against SMR drives, got it, don’t use SMR.
Does anyone have any recommendations for large capacity spinning drives that will work well with ZFS / TrueNas? I was leaning towards the Seagate Exos x20, Exos x18 series drives. Looking at the specs these appear to be CMR drives. Does anyone have an opinion on these drives?
Proper Shutdown Procedure
Ok this is a super noob question. Is there any procedure that needs to be completed before shut down (like a VMware equivalent of maintenance mode to prevent data loss). Do I need to dis-mount the pool, or do anything to prevent active writes before shutting down? Or is it really just as simple as clicking the power icon in the Gui?
Backups / Catastrophic Server Failure Planning
What is the best way to backup my TrueNas server, or is that even required? When I am referring to backups, I don’t mean backing up the actual data that resides on the storage pools, I mean a backup of the OS configuration (boot drives). Let's say in some hypothetical scenario my boot drives (dual SD cards) fail and I need to re-build the server. How should I prepare for that?
Another hypothetical, let's say this physical server gets damaged unexpectedly; but all of the drives stay in tact. How would I go about moving this pool to a new server? Is it as simple as moving all of the drives to a new server and TrueNas will automatically recognize/import the pool?
Thank you all in advance!