I appreciate your insight. I am new to this and naturally, I presumed that dedicated hardware is better than software.
Right, that's why we're guiding you away from it. I'm the king of pricey dedicated hardware. But that also includes an understanding of whyitfails.
I have two Ascend/NetStar Gigarouters in inventory here, purchased nearly twenty years ago, for over $100,000. They once provided about half the Internet connectivity for a major American metro area. One of the very first wirespeed routers, they were out of service within about five years because the world had just passed it by. Today, a Ubiquiti EdgeRouter is able to perform very high speed gigabit routing in hardware-assisted software ... and only costs a hundred bucks, less than one one-thousandth of the Gigarouters.
Part of the deal with specialized hardware is that you have to be willing to let go rather quickly. That's one of the reasons I tend to prefer general purpose gear, you can often retask it to some new and different task. The P410 wasn't an outstanding controller even in 2009 when it was introduced. We've got some in production on hypervisors. They are poor performers. Their batteries die and need replacing. All the usual crap.
Honestly, if I could enable HBA mode on my controller, I would do it.
As for features on the p410, one in particular that I appreciate having, that ZFS does NOT, is the ability to expand pools.
First, ZFS does have the ability to expand its pool. You do, however, have to do it the ZFS way.
But let us look for a moment at what you're saying. Indeed, you can allocate more disk space to a P410 LUN. That's a nice feature. But if you do so on a ZFS based system, ZFS has no way to grow into that added space. It doesn't recognize that the underlying device has fundamentally changed, because ZFS is designed to be directly accessing raw hard disks, and raw hard disks don't just suddenly grow in size without also changing other characteristics. I deem the P410's space expansion a worthless feature if you're using the controller on a FreeNAS box.
Other desirable features include near zero processor load,
What else is your NAS supposed to be doing? You want to pay good money for a CPU just to leave it idle?
ZFS calls that scrubbing
many different raid options, etc.
*ZFS* has many different RAID options. Your P410 does not have triple parity RAID, or the ability to aggregate a few hundred disks into a single pool.
If the 2.2TB issue you refer to is a drive size limit, that has been fixed. I am running 4TB drives at the moment.
The 2.2TB thing is an unfixable hardware issue that relates to the use of 32 bit integers instead of 64. It plagues the LSI1068. I *think* the P410 is PMC Sierra though.
What I really don't like about it is the fact that HP does not supply BSD utilities for it.
Well, there's that. I think there's a native hpacucli but it was for FreeBSD 7, and that means a lot of puttering around. More troubling would be that the ciss driver seems very focused on obscuring problems with the physical storage from the host, which leads to unintended consequences.
For example, I cannot even access SMART information except through a jail. Perhaps that is the real beauty of software raid; less junk to get in your way.
Well, that's more or less it. You're supposed to see ZFS itself as being your RAID controller. It was designed that way and offers massive resources to the well-designed NAS appliance. Where are you going to find a RAID controller with ~~50GB of read cache and half a dozen GB of write cache (on a 64GB FreeNAS system)? The potential traffic ZFS can push at the underlying storage system is staggering, and can cause significant problems with a crappy several-gen-old controller like the P410 because it just totally swamps the poor thing.
