Anybody with a working PERC + FreeNAS configuration?

[Chris Hoefler]

Ok, so hardware, and especially SATA/SAS controllers has been discussed at length in various forum posts. For example, these recent posts about HBA vs. RAID cards,
http://forums.freenas.org/threads/confused-about-that-lsi-card-join-the-crowd.11901/
http://forums.freenas.org/threads/so-you-want-some-hardware-suggestions.12276/

Usually, when the Dell Perc controller is brought up, the recommendation is to throw it away and get some other controller. There are posts elsewhere that suggest that a Perc is at least usable (perhaps not ideal) with ZFS as long as you create RAID-0 volumes for each disk.

Rather than focus on recommended SAS/SATA cards, I would like to pose the simple question: Has anybody achieved a stable and reliable FreeNAS configuration with a Dell Perc card? Either Perc 5 or 6 with any version of FreeNAS, but especially 9.2.0.

The reason I ask is because I spent some time tracking down a very hard to diagnose problem that ended up being the Perc card. There was no obvious error message and no core dump. The card was recognized and supported by the mfi driver. The firmwares were up-to-date. The drives were accessible and working. The problem was that the system would spontaneously reboot. It seemed to happen when under a write load, primarily, but it also didn't seem to be a problem under certain benchmarks (ex: iozone). So, I still don't know what the problem was, whether it was the card or the driver or some magic aether dust, but I stopped getting spontaneous reboots as soon as I moved to a different controller.

An obvious question, why didn't I heed the advice to get an HBA card? Well, a few reasons,

1. The concerns around the Perc seemed to center on the need to create RAID-0 problems, but not on reliability or stability issues. I had my main ZFS volume on a different card in an enclosure. I just needed the Perc for the drives that FreeNAS was going to be installed on, the ZIL, and the L2ARC, so I wasn't concerned about needing to create a handful of RAID-0 volumes.
2. I was already buying a SAS HBA to drive the enclosure, did I really need to buy two?
3. On my box the Perc is integrated, so I really would be throwing away otherwise perfectly usable silicon (by not being able to put it into a different box not running FreeNAS).

The Perc is, perhaps unfortunately, the card that ships with most default Dell server configurations. So if you are buying a new Dell server, or a used, it will probably have a Perc in it unless you explicitly choose something different. If many people have tried, and nobody has succeeded in getting Perc cards to work with FreeNAS, maybe it would be helpful to create a new section in the Docs titled something like "Hardware to Avoid" and add Perc cards to that list.

This, in my opinion, is a bit different from hardware incompatibility. FreeBSD recognizes the card. It is supported. The drivers have been around for ages. This is a more subtle issue that may be specific to ZFS, or using two controllers at once. Whatever the actual issue, I would like to know if there was any reason to think the Perc was hopeless from the beginning. Because if that is case, I think it would be good to document somewhere.

 

[cyberjock]

I helped someone 2 weeks ago that had a Perc card. It didn't work quite right and the server would randomly reboot. Here's the flaws I see in your discussion:

1. RAID cards are pretty much bad joo joo with FreeNAS. If you can't disable their write cache, can't run in jbod, and/or can't do SMART, you are in deep waters without a boat. For him, we couldn't do any of those 3.
2. Perc may or may not perform to the expected needs of the user. Depending on the model it may not even support 6Gb SATA and may not even support 2TB drives. ZFS was designed to go enormously big. So limiting yourself with either of those is illogical. If your system is that small, FreeNAS may not even be the best option for you. FreeNAS is meant for really big systems. You wouldn't buy a mac truck to drive to work, but for some people that is what they are trying to do.
3. There is a BIG gaping hole between supported and well supported. The bottom line is that regardless of how good or bad Perc is with performance and regardless of the fact that it's a RAID card you are still dealing with hardware that isn't "well supported".
4. Time = money. This guy has literally put almost 80 hours into trying to get this Perc card to work. I asked him if he got paid $1.50 an hour for his job. Of course he responded with a "No, I make alot more than that". He responded that he makes about $20/hour. So I asked him where the financial sense in trying to make a card work that has cost his business about $1600 and still doesn't work when they could have bought an M1015 for about $120 and in less than an hour had a perfectly capable card reflashed to IT mode. He then asked for a link to the card on ebay and bought it immediately.

The bottom line is that there's a point where its just not logical to continue to push against the waves. The ocean is rolling, and instead of accepting the fact that the waves are coming you are pushing against it. If you are a hardcore FreeBSD developer guy and you are hellbent on figuring out the problems you may be able to figure it out in a few hundred hours of development time. For the rest of us mere mortals, we'll just buy crap that "just works" and smile that we may have spent a little more money we might have had to, but we also didn't spend a solid 2 weeks trying to get a server up that should have been up and running in 2 hours. You're clearly not a developer, and there's a reason why the FreeBSD developers haven't tried to fix it. Not sure what the reason is, but a noob such as yourself and myself aren't likely to get a fix on our own.

Last server I built had FreeNAS up and running with data being copied to the pool in less than 2 hours. That was total assembly, installation, and basic setup. So the question I have for you is, "How much time have you spent on trying to get your server to work, and how much more are you going to spend before you figure out that the Perc card isn't worth trying to reuse?" The answer you are likely to provide is exactly why people like me see a post with the word "perc" and immediately close our web browser. We're going to listen to the same stuff we hear from every user... its cheap(or free).. its there.. I think it should work.. I don't care that much about performance.. etc etc etc. And our answer is "we don't care".

As for adding a section that says hardware to avoid, that's just not a good idea. Exclusion lists are a bad idea for this purpose. You'll see that some obscure hardware you might find online is not on that "bad list" and assume it must be good. We do not want to foster that behavior. We provide a link to the FreeBSD compatibility matrix, and for those that want to do their homework there is plenty of anecdotal evidence of what hardware works very well and what doesn't. Not to mention, if someone didn't want to be inconvenienced with finding this anecdotal evidence themselves, providing the "bad list" isn't any more useful than providing the "good list" that people will either not find or ignore. In your case you appear to have justified to yourself(inappropriately in my opinion) reasons to use the card. To me it appears that if we had a "bad list" you'd still have used those justifications to attempt to use the card, so what value was there in having the list if it would be ignored anyway? I don't want to spend time keeping track of lists. I've dumped tons of hours into my presentation already, and I bet at least 30% of posts here in the forum and an even higher percentage in IRC are people doing precisely what I tell them not to do in my presentation. So remind me and the other regular posters why we'd want to create yet another document to be ignored?

 

[Chris Hoefler]

Cyberjock, thanks for your response and comments. I just want to say that I have seen your and jgreco's many contributions to these forums and offers to help new users. I think it is really great of you guys to volunteer your time in this way. The FreeNAS community is greatly served by your efforts, so thanks for making newcomers, like myself, feel welcome.

With regard to my post, I do not intend to try to make the Perc work. I just want to say that up front. I agree with you that it is a complete waste of time for ~$200 in savings. Here's the thing, though, I wasn't able to find a clear guide when I was at the hardware purchasing stage that made the definitive statement, "Perc cards don't work, period!" I would gladly heed such advice, because I don't like wasting my time. The reason why I didn't find that guide, I think you will agree, is because it doesn't exist. There is, however, a lot of discussion about Perc cards in the forums. But forums don't make a very good knowledge base for things like hardware compatibility for at least two reasons. One is the general problem of searching and finding relevant discussions. A search for something like "Perc" uncovers almost a hundred posts. Some of them are general discussions about the pros and cons of various cards. Some are troubleshooting questions about specific models. And some are solicitations for hardware purchasing advice. While some of these discussions are useful and interesting, none of them really provide a clear answer to the question of hardware compatibility.

The second reason is that with many topics, hardware in particular, there are two types of information one may be interested in: does it work, and what are some general "best practices". The former is definitive. The latter is more of an opinion. Forum discussions sometimes encapsulate the former, but usually consist of the latter. The fact of the matter is, here and elsewhere, there are tons of "best practices" and "advice" guides for servers, NAS, personal clouds, you name it. And they don't all agree on what those best practices are. Self-proclaimed noobs and experts. Amateurs setting up home media servers and professionals setting up 100s of TBs SANs. Some will say don't use a RAID card at all. Some will say only use a RAID card with JBOD, some will say RAID cards are fine even if you are using ZFS. Some will say RAID cards are better than using ZFS. Some will say you must use ECC memory. Some will say it's not so important. Some will say you must use enterprise drives. Some will say regular retail drives are fine. Some will say you are wasting your time if you don't buy SAS drives. Others will say SATA offers a better price/performance. And so on and so on. It's all good. There are a lot of opinions out there, no doubt backed up by a lot of grunt, sweat, and experience, but nonetheless not definitive. And, of course, different situations/requirements will necessarily lead to different, but adequate, solutions to different needs.

With my question, though, I want to answer the former: does it work? Now, that question can easily be answered if there is a well-kept hardware compatibility list. But it is hard to capture all possible hardware models and chipset configurations. For example, on the guide for FreeBSD 9.2, both the Perc 5 and 6 are there, marked as supported by the mfi driver. However, two LSI cards I have used that I also know are supported, the 1068E by the mpt driver and the 9200E by the mps driver, are not on that list. The second problem, is that even if a driver supports a card, it may have bugs, or it may crap out in certain situations. And that was my problem. This is just not easy information to find anywhere.

So I just want to try to capture that information somewhere. Maybe even in this thread if not in an official guide. Based on my experience, I would say Perc 6 cards do not work, even though they are supported, with FreeNAS 9.2. But that is only one data point and may be configuration specific. If there are others, I would like to hear about them, and maybe we will save the next person some time and headaches. ;)

 

[TheSmoker]

I am using Perc 5i in more than 1 FreeBSD server it it has performed flawlessly ( freebsd 8.x, 9.x and now 10). Btw, it's not a dell server. The card was flashed to latest lsi firmware ( not dell). Having a mix of raid 10 and raid 5.
I am not using perc 5i in my 2 freenas servers for obvious reasons one of them being that i have more than enough sata ports on mobo the other one being zfs. :)

 

[Chris Hoefler]

Thanks TheSmoker. I did, of course, expect the Perc to work just fine with FreeBSD because there is an officially supported driver. So maybe the problem is not that, but something specifically to do with my stock firmware, or a bad interaction with ZFS. Either way, it's good to know.

 

[Lordadmiral Drake]

Well, what about a list of Cards that says
Good
Itchy
Bad
And last but not least:
Cards that are not listed have NOT been testen, try at you OWN fucking risk

 

[BobJ]

When using a RAID card with freenas or nas4free you are not using any of the raids features its just there to find the drives. The OS then does a software raid so the controller should work fine in this setting.

I'm building a 4th nas and I have 2 brand new SAS drives that I never used and a few old Dell Perc5s and a HP raid controller. I just need the card to recognize the drives on boot. Which they do. IN the next week I'll test them out to see if they work ok for a NAS. I have so many parts I could make 8 nas boxes. So for me its no big deal if there are issues. I will let you know how it goes.

 

[danb35]

When using a RAID card with freenas or nas4free you are not using any of the raids features its just there to find the drives. The OS then does a software raid so the controller should work fine in this setting.

Why did you necro a thread that's been dead for a year and a half, only to add incorrect (or at least incomplete) information? The truth is that it depends very much on the RAID card. If the card will allow you to completely disable or bypass its RAID features, and simply act as an HBA, then it will be as you describe, and things will be fine. This is the case, for example, with the popular IBM M1015, when cross-flashed to IT mode firmware. However, not all RAID cards will do this. With some, your only options are to create a multi-disk RAID volume using the controller, and present that to FreeNAS as one disk (evil); or set up each disk as its own RAID0 device, and present each of those to FreeNAS as one disk (only slightly less evil).