Refreshing my storage server -- considering FreeNAS, but I want hardware RAID

[tycoonbob]

So yeah. I have a pretty beefy storage server I built 2 years ago.
CURRENT SPECS:
Case: Norco RPC-4224
Mobo: ASUS P8B-E/4L ATX
CPU: Xeon E3-1220v2
RAM: 2x4GB Kingston ECC
RAID Controller: LSI MegaRAID 9261-8i
Expander: HP SAS Expander
OS Drive: Mushkin 60GB SSD
Data Drives: 7x2TB (RAID6), 2x3TB (RAID1)

Currently I'm running Windows Server 2012, but I am doing some hardware refresh with this server.

• replace HP SAS Expander with Chenbro CK23601

• Add ASMB5-iKVM (iKVM for ASUS motherboard)

• Add second Mushkin 60GB SSD (hardware RAID, for OS)

• 6x6TB drives (RAID10) for data volume (expand as needed in the future) (HGST Deskstar NAS or WD Red)

So yeah, I have a hardware RAID controller (LSI MegaRAID 9261-8i) and will be replacing my HP SAS Expander with a Chenbro. I want to use hardware RAID, and I'm not really interested in using ZFS at this time. With this refresh, I am considering switching OS, and have pretty well come to terms that it will either be a vanilla CentOS 6.6/7 install, or FreeNAS 9.3. I like FreeNAS for the web UI, easy out of the box config, quick ROI, and a few of the Plugins (CrashPlan, mainly). I have a Windows 8 VM running with the LSI MegaRAID Storage Manager that would be able to connect to the controller remotely to manage it, so that's not a problem. Also, this is for a home environment. I have 3 Dell R610's running Proxmox, but use Ceph on those nodes as primary storage (with Crucial MX100 512GB SSD's), but do also use NFS storage from this server for a few lower priority VM's.

So yeah...I know with 9.3 they did away with UFS, and only have ZFS. I would configure the RAID controller to only have 2 array's...a RAID 1 with the 2 60GB SSDs, and a RAID10 with the 6TB drives. FreeNAS would only see those two volumes, and I would want FreeNAS installed on the RAID1. So essentially (I would think) FreeNAS would think there was only 1 drive in the server.

Would this work? Would this cause any problems?

Again, I'm not interested in using ZFS, but I am very interested in all the other features that FreeNAS brings to the table.

Thanks!

 

[danb35]

If you aren't interested in ZFS, what do you think FreeNAS brings to the party that another solution wouldn't?

 

[tycoonbob]

If you aren't interested in ZFS, what do you think FreeNAS brings to the party that another solution wouldn't?

[...] I like FreeNAS for the web UI, easy out of the box config, quick ROI, and a few of the Plugins (CrashPlan, mainly).[...]

I answered that above.
Being able to install FreeNAS, load a web UI, configure CIFS/NFS/AFP shares and iSCSI targets, run CrashPlan from the box, Web UI for a quick overview of everything, etc. I travel a lot for work (I'm a technical consultant), so my time at home is a premium to me and I don't want to have to install all the packages manually, and manage another Linux server. I have a SSH tunnel from my work computer to my home network every time I connect, so I can easily access a Web UI to see what's going on with my storage with FreeNAS. Jails are also nice, that I can use for various things.
FreeNAS supports DLNA, and I have a SONOS setup at home, so that would be nice. I can also run Plex from there eliminating the need of a separate VM. Same goes for my current uTorrent VM.
S.M.A.R.T. monitoring, email alerts, and more that I'm probably not thinking about right now.

In other words, I like everything FreeNAS has to offer, but I'm not interested in using ZFS. If I did want to use ZFS, this would be a no brainer. I'm just not aware of any comparable OS that has all of this out of the box, in a system that has a large community following. If OpenFiler had a stronger community, didn't have a paid iSCSI plugin, and was based on something more mainstream (like CentOS, since they talked about moving to that), I would go with that.

Is there something better to fit my needs than FreeNAS?

 

[pirateghost]

What about hardware raid do you think is more beneficial than zfs?

 

[tycoonbob]

What about hardware raid do you think is more beneficial than zfs?

I'm not saying hardware RAID is superior to ZFS, nor am I saying the opposite. I am not here to argue whether or not to use ZFS, but that I already have a RAID controller with BBU, I am familiar with said RAID controller, and I plan to continue using said RAID controller.

 

[depasseg]

Would this work? Would this cause any problems?

Yes, it will work. I can't see there being any more problems with any other config. You might want to test a disk failure to see if it's possible to get notifications. Since the HW controller is presenting only the 2 disk to the OS, monitoring and visibility might be limited. My concern would be what happens in the case of a failure.

 

[tycoonbob]

Yes, it will work. I can't see there being any more problems with any other config. You might want to test a disk failure to see if it's possible to get notifications. Since the HW controller is presenting only the 2 disk to the OS, monitoring and visibility might be limited. My concern would be what happens in the case of a failure.

Thank you for staying on topic and answering my question.

LSI makes drivers for FreeBSD, which I would assume would work on FreeNAS. Recovering an array (removing failed drive, adding/expanding array(s) shouldn't be a problem since I can use the management software from an external server. You are correct that getting notifications might be difficult, but that's something I can look into. I know the controller logs events, but I'm not sure how/where those are logged, and if I can get those syslog'd, I can send them to my Splunk server and use Splunk to alert me if a [WARN] or [ERR] event happens.

But based on your feedback, I think it's worth testing FreeNAS on this hardware once I am ready to make the switch, which is really what I was hoping to accomplish with this post.

 

[cyberjock]

I'm going to say this just once, because I'm tired of saying it.

It's in the manual, it's in my noobie guide, and there's dozens of "examples" of how badly hardware RAID and ZFS go. Either go hardware RAID and find another OS or go ZFS and ditch the hardware RAID. PERIOD.

Good luck with whatever option you choose.

 

[danb35]

...and apparently you missed the part where he said he didn't want to use ZFS.

 

[tycoonbob]

I'm going to say this just once, because I'm tired of saying it.

It's in the manual, it's in my noobie guide, and there's dozens of "examples" of how badly hardware RAID and ZFS go. Either go hardware RAID and find another OS or go ZFS and ditch the hardware RAID. PERIOD.

Good luck with whatever option you choose.

Scanning through your manual, I didn't find anything about why not to use a RAID controller with FreeNAS.

I also am not finding anything on the forum specific to my situation. Why would a hardware RAID 10 array presented to FreeNAS be a problem? Wouldn't FreeNAS just see it as a single drive? Since UFS has been removed from 9.3, ZFS is the only available file system, so wouldn't it be the exact same as running ZFS on a single drive?

 

[Mlovelace]

[...] I like FreeNAS for the web UI, easy out of the box config, quick ROI, and a few of the Plugins (CrashPlan, mainly).[...]

I answered that above.
Being able to install FreeNAS, load a web UI, configure CIFS/NFS/AFP shares and iSCSI targets, run CrashPlan from the box, Web UI for a quick overview of everything, etc. I travel a lot for work (I'm a technical consultant), so my time at home is a premium to me and I don't want to have to install all the packages manually, and manage another Linux server. I have a SSH tunnel from my work computer to my home network every time I connect, so I can easily access a Web UI to see what's going on with my storage with FreeNAS. Jails are also nice, that I can use for various things.
FreeNAS supports DLNA, and I have a SONOS setup at home, so that would be nice. I can also run Plex from there eliminating the need of a separate VM. Same goes for my current uTorrent VM.
S.M.A.R.T. monitoring, email alerts, and more that I'm probably not thinking about right now.

In other words, I like everything FreeNAS has to offer, but I'm not interested in using ZFS. If I did want to use ZFS, this would be a no brainer. I'm just not aware of any comparable OS that has all of this out of the box, in a system that has a large community following. If OpenFiler had a stronger community, didn't have a paid iSCSI plugin, and was based on something more mainstream (like CentOS, since they talked about moving to that), I would go with that.

Is there something better to fit my needs than FreeNAS?

Have a look at OpenMediaVault, its based on Debian and has most of what you're asking for.

 

[jgreco]

The fundamental problem here is that he doesn't want to use ZFS and FreeNAS only supports ZFS.

So we're talking about someone who doesn't want to buy diesel fuel but is drooling at a car with a diesel engine.

We're so sorry, it won't work out well for you. Please move on and good luck.

 

[depasseg]

He's not adverse to using ZFS, he just wants to use HW RAID under ZFS. And I've found Zero warnings in the manual or the forums. In fact the Oracle Solaris ZFS Admin guide doesn't even say "Not supported. Go Away."
It says https://docs.oracle.com/cd/E23823_01/html/819-5461/zfspools-4.html

• Hardware RAID
  • Consider using JBOD-mode for storage arrays rather than hardware RAID so that ZFS can manage the storage and the redundancy.
  • Use hardware RAID or ZFS redundancy or both
  • Using ZFS redundancy has many benefits – For production environments, configure ZFS so that it can repair data inconsistencies. Use ZFS redundancy, such as RAIDZ, RAIDZ-2, RAIDZ-3, mirror, regardless of the RAID level implemented on the underlying storage device. With such redundancy, faults in the underlying storage device or its connections to the host can be discovered and repaired by ZFS.

The only FreeNAS warning that I can find is about using JBOD mode to pass through individual drives to FreeNAS, which I agree with. But that isn't what the OP is asking about.
https://forums.freenas.org/index.php?threads/confused-about-that-lsi-card-join-the-crowd.11901/

 

[depasseg]

Don't get me wrong. I think it's asking for trouble, and would rather prefer to see the drives presented to FreeNAS.

 

[cyberjock]

He's not adverse to using ZFS, he just wants to use HW RAID under ZFS. And I've found Zero warnings in the manual or the forums. In fact the Oracle Solaris ZFS Admin guide doesn't even say "Not supported. Go Away."

Two things, and I'm locking this thread because there are plenty of these kinds of discussions that turn to fire and brimstone and I'm not about that tonight.

1. Read the hardware requirements. It makes it perfectly clear that you shouldn't use hardware RAID with ZFS.
2. Oracle's guide is N/A because Oracle sells you a finished product. They don't have to talk about hard requirements in heavy detail because if you called them and said you wanted hardware RAID and ZFS they'd laugh at you hand hang up the phone.

Thanks to all that participated. I'm going to lock this thread.

 

[jgreco]

I'm going to follow up despite Cyberjock's uncharacteristic mellow because there's one question that was asked that should actually be answered here.

ZFS on a single drive is a bad idea. ZFS on top of an array that has redundancy but is being presented as a single drive is just idiotic.

When you have a single drive, ZFS has no redundancy on which to draw if it finds a corrupted block. This undercuts one of the main ideas in ZFS, which is that errors can be identified and corrected. And bitrot does happen, occasionally, on single drives.

On a RAID array, however, this is worse! There's actually redundancy available, but you've hidden it away from ZFS, so ZFS can't go and retrieve the data from a redundant block when it finds a corruption. So now you develop a problem with your pool (hopefully a data block, and not a metadata block) and there's really nothing ZFS can do about it. This could potentially be catastrophic to the pool.

The other bad aspects of a RAID controller are that it basically gets in the way; ZFS will constantly be pummeling a RAID controller's cache with requests an order of magnitude (or worse!) larger than what it was designed for, and that error detection and failure prediction get masked from the end user.

The only FreeNAS warning that I can find is about using JBOD mode to pass through individual drives to FreeNAS, which I agree with.

Despite your not being able to find it, this is a discussed-to-death topic. You'll want to look for discussions of SMART, HBA, and RAID.