Need RAID Controller Recommendations

[kesh]

Hi, I'm trying to put together a FreeNAS server for my research lab. My plan is simply to put two 2TB drives in RAID0. Reliability is the keyword here as it will be storing research experiment data (i.e., irreplaceable). (Yes, I'm also planning on running a backup NAS server on a remote site.)

I've chosen motherboard & HDDs:

Motherboard - Asrock H61M/U3S3 (paired with i3-2100)
HDDs - WD2002FAEX (Caviar Black) x2

And I've been contemplating about RAID controller. The options are
1. Software RAID
2. Get H67 motherboard and use Intel RST (formerly Matrix RAID)
3. Separate RAID controller (preferably in $100-$200 range or should I splurge for more?)

Any suggestions on which one? Again, reliability is the foremost importance. If go with 3, which RAID card do you recommend? Oh, I'm also interested in "anti-recommendation" as well :)

While I'm at it, what do you think of the HDD choice? My choice was simply based on most affordable 2TB drive with 5 year warranty (hoping for implied better quality???)

Thanks for your inputs!

 

[survive]

hi kesh,

Honestly, if reliability is the keyword don't do RAID0. Just don't....remote backups or no don't do it.

Instead, take the cash you were going to spend on a controller and get a board with more SATA ports and 2 more drives and do a 4 disk ZFS mirror or RAIDZ2 for your storage...that gets you much better redundancy so a single disk failure won't wipe out all your data.

Personally I would take a look at the AMD E-350 boards, I particularly like the ASUS E35M1-M because it has the PCI-e 16x slot though I have never used one. Take a look here:

http://www.newegg.com/Product/Produ...00446 600110169&IsNodeId=1&name=AMD Hudson M1

More info:

http://forums.freenas.org/showthread.php?27-AMD-E-350-Thread-(now-in-new-forum-)

More thoughts:

I can't put it more bluntly than this, but figure the data you are going to store on here is the most valuable thing in your lab, so don't go the cheap route! If you are a PI this is probably going to hold your life's work, if you are a grad student doing the leg work for a professor this thing will be holding the key to completing your thesis.

The drive choice is nice...the 5 year warranty is certainly worth something, but that just means you get a refurbed drive if one fails in the next 5 years instead of three. I would look at the Western Digital "RE" drives or the Seagate Constellation ES drives if the budget allows. You might even want to go so far as getting some of each just to avoid some lot or manufacturer-specific defect causing a mass failure. While you are at it get a spare drive you can park (try it before you trust it!) close by.

Be sure you set up the e-mail feature so you get reports & notifications. Also include the price of a good UPS. Don't skimp on the case, get one with good airflow over the drives....a cool drive is a happy drive and buy quality fans (I like Panaflo's) for it. Be sure to get a good power supply, pay for quality over watts (Antec, Corsair & SeaSonic are excellent choices, look for an 80Plus gold or silver from one of those guys) and be sure to put the device somewhere out of the way that is climate controlled (a cool server is a happy server). You might want to consider getting a couple of external drives you can sync to for backups as well. Replication is nice, but it will also replicate errors, accidental deletions & corruption just as well as it will replicating good data.

I actually work in a university environment doing IT and 3 or 4 times a year we get a call from a professor\researcher who is asking for help with some solution they put together that has failed...the conversation usually starts with "I can't see my storage" and ends with an (expensive) shipment to Ontrack. I would actually ask your IT guys, if you have some, for their input because they are going to be the ones getting the call when you have a problem; consider it a common courtesy if nothing else....they might actually offer a service hosted on their central infrastructure that you can use freeing you from the task and worries of doing the IT work and letting you get to doing the research. Thinking more globally, if the IT staff doesn't have something like this for you I would also encourage you to speak to your dean and tell them that this is a service you need and they should think about funding such a service....your IT guys will love you for it!

-Will

 

[kesh]

Will - Your thoughts and great suggestions are much much appreciated!! You got me re-think over this from scratch :) I've built numerous PCs in the past but this would be the first attempt to build a server, and I want to get it right.

First, your suggestion about running it through our IT department is a good one, especially my employment is not a permanent one (and I, an EE, is an odd ball in the department that is very much tech-non-savvy) and definitely do not want to leave my boss hanging with a crippled system after I leave.

OK, so no RAID0. I'm a little confused about ZFS disk pool solutions though. (sorry, I'm completely new to ZFS system and slowly learning about it)

Instead, take the cash you were going to spend on a controller and get a board with more SATA ports and 2 more drives...

Are you saying not to use on-board SATA controllers? Do they have reliability issues?
I think I will strongly pursue 4 drive, RAID-Z system as you suggested.

> ASUS E35M1-M

That is a good board, indeed. I just put together a music jukebox PC, so I was keeping close eyes on the AMD Zacate Mini ITX boards. (Although an Atom board won out in the end as the Zacate's video capability is not necessary for a pure music player.)

Anyway, I actually thought about using this very MB, but decided to go with the Sandy Bridge, thinking about future addition of a light database (MySQL?) task to the server (to associate experiment parameters to each data file). Do you think E-350 would provide sufficient CPU power to serve all our needs (ZFS/CIFS/MySQL/PHP/etc.)? I foresee that the server will be accessed by less than 10 users at a time.

Cases, UPS, etc... Let me go ahead and spill the rest of my config so you can critique/suggest alternatives:

Case: COOLER MASTER Centurion 5
PSU: Antec BP550 Plus 550W
USB Flash Drive : ADATA 8GB S102 USB 3.0 Flash Drive (hosts FreeNAS OS)
Memory: G.SKILL 2 x 4GB DDR3 1600 (too much?)
NIC: Intel Gigabit CT NIC (also use the MB NIC; needs 2 connections)
UPS: APC BE650G 650 VA 390 Watts UPS

The case selection probably is the biggest concern. I thought that while it's a fairly cheap case but it looks spacious with large fan in the back. It, however, lacks a fan by the drive bays, which you are suggesting is a critical case feature for a NAS server.

Actually, I've been drooling over the Fractal Design Define R3 ATX Mid-Tower case, but I told myself to hold back in my initial list. Maybe it isn't a bad choice after all. Whaddaya think?

 

[survive]

Hi kesh,

Ok, first thing, since I gather you are a grad student doing this for a professor and will be handing this off to "the next guy" at some point I would urge you to document, document, document what you are doing. Start with defining the problem you are trying to solve, propose the solution, get the PI to sign off on it and then detail your build...it's good practice for the real world and "the next guy" will thank you for it. After the build is done & in use write up how it worked out and what you would do differently.

Next, read this to get familiar with the RAID types:

http://en.wikipedia.org/wiki/RAID

And read this for more info on ZFS:

http://en.wikipedia.org/wiki/Zfs

Briefly, you have mirrors & nested mirrors, raidz which is like RAID5 and raidz2 which is like RAID6. Raidz can loose 1 disk, raidz2 can loose 2 and still preserve your data.

The on-board SATA controllers are fine and there's not really any reason not to use them or to look at an add-on card till you need more than what's on-board. Both Intel & AMD make fine SATA controllers. People run into trouble with the cheap controllers or when they add a PCI controller that saturates their PCI bus, you should not have that problem with a box the size you are looking at.

Read through this thread:

http://forums.freenas.org/showthread.php?27-AMD-E-350-Thread-(now-in-new-forum-)

For some other FreeNAS users experiences with the E-350 boards. Best I can tell these boards provide enough horsepower to saturate a gig-e interface, but keep in mind I have never actually used one.

Personally I use an AMD 790GX board in my NAS, intially with a 4850e proc and currently with a PhenomII 940 left over from a system update, which is so much more proc than my NAS could ever use. I like the GX boards because they have on-board graphics with dedicated memory so I'm not using system memory to drive the BSD prompt.

As for using the box for other services....you are going to run into a couple of problems. First, FreeNAS8 doesn't really allow you to add software right now. There will be a package manager that does so coming soon but it is not available yet. That's kind of one of the big sticking points between FreeNAS 0.7 and 8...if you have been using 0.7 you have had the ability to do all sorts of additional tasks with your storage box, but 8 is based off a special version of FreeBSD called nanobsd that is geared toward embedded installs that makes supporting additional software more complex.

Personally, to be blunt, this is a storage box...use it for storage and nothing else. You want this thing to hum along serving files. If you want to add services later just get a second box and give it a NFS share from the FreeNAS box

As for case recommendations, I haven't really been keeping up with the market as I have a *ton* of Antec Solo cases left over from lab machine refreshes. I adore these cases:

http://www.newegg.com/Product/Product.aspx?Item=N82E16811129018

Quiet, very well finished, 120mm fan in back & 2 92mm fans in fron.

PSU: I'd shy away from anything with basic in the name, that's just me. For something close to that price (if you can deal with rebates) I'd look at these:

http://www.newegg.com/Product/Product.aspx?Item=N82E16817703026
http://www.newegg.com/Product/Product.aspx?Item=N82E16817139020
http://www.newegg.com/Product/Product.aspx?Item=N82E16817151096

You can probably go with something smaller with the E-350 board & 4 drives, I'm guessing you would do well under 200 watts even running full bore, but I'm a big believer in overbuilding & paying for quality PSU's.

Flash drive:

It's tough to screw up a flash drive these days that should be fine. You only need a 1Gb drive for FreeNAS right now, but this is also the least expensive componet anyway.

Memory:

Gskill is a good manufacturer, I have no problems with them. Personally I would spec some ECC memory if the board you select supports it. Check the compatibility list if there is one. There is no such thing as to much memory, especially in a ZFS box, whatever you select, get 8Gb minimum...memory is so cheap these days there's no reason not to.

NIC:

Nothing wrong with the Intel you have selected. Personally I have a big bias against Realtek, always have so if you plan on doing 2 links you might want to look at one of these:

http://www.newegg.com/Product/Product.aspx?Item=N82E16833106015

Please note, that is my own strong personal bias showing through here, ther's no reason to think the on-board shouldn't be perfectly fine, but you asked!

As for your case concerns, I'm a big fan of paying for a quality case, mostly because they tend to stick around...I've been using my Solo since the AMD 3800X2 came out. I don't want to say the front fans are critical as much as keeping the drives cool is critical but they are an excellent way of doing so. Semantics I know. As for that R3....well I wish you hadn't turned me on to that....I'm going to have to look into that. Turns out Newegg actually sell them and at ~$110 it looks positively drool-worthy.

Of course after all that, you could also take a look at one of these:

http://h10010.www1.hp.com/wwpc/us/en/sm/WF05a/15351-15351-4237916-4237918-4237917-4248009.html

http://forums.freenas.org/showthread.php?32-Freenas-8-on-Hp-Microsvr-Nl361

http://hardforum.com/showthread.php?t=1555868

-Will

 

[kesh]

Will,

Again, I appreciate your inputs! I'll be reading these RAID/ZFS links closely to familiarize myself with the technology.

I gather you are a grad student doing this for a professor...

Close but no, I'm a postdoc :). But yeah, I need to put conscious efforts to document this. Although Boss will not have any idea of what will go on, asking her to sign it off will certainly make me more structured. And as for "post-me" if something goes wrong, it'll be in the hands of IT guys to fix it. Maybe I can keep reminding her of that to make her keep me permanently :D

Personally, to be blunt, this is a storage box...use it for storage and nothing else.

Yeah, I guess I've gotta agree with you on that. I was thinking overly about minimizing the running cost, but in the end a couple hundred dollars a month for extra server probably is nothing to the department.

I have a *ton* of Antec Solo cases left over from lab machine refreshes.

It's a pretty descent looking case. If you have a ton of them, I can certainly give you my shipping address :p

PSU: I just bought that Antec PSU for another machine, and I'm pretty happy with it. So, I might stick with that. A ton of newegg customers bought'em and it's one of the highest rated PSUs there so I figure it can't go wrong. However,

I'm a big believer in overbuilding & paying for quality PSU's.

I strongly agree here. Have you read the recent article from Tom's, reviewing dirt cheap PSUs? Quite amazing how some manufacturers can get by with their cheapo low-quality products.

NIC: I saw that dually NIC, but I cannot justify $160 for it. I'll give the on-board a try and see how it goes. As long as I use a MicroATX MB, adding another NIC card isn't that big of a deal.
Case: If onboard SATA controllers suffice, I may turn to a Mini ITX solution.

As for that R3....well I wish you hadn't turned me on to that....I'm going to have to look into that. Turns out Newegg actually sell them and at ~$110 it looks positively drool-worthy.

LOL, I'm glad that I helped you keeping your mouth wet:p Actually, R3 would be my 2nd choice among the Fractal Design cases. If you're a Mini ITX proponent and want to really drool, look for their Array R2 Mini ITX NAS Case:

http://www.fractal-design.com/?view=product&category=2&prod=43

* Mini DTX support (2 expansion slots)
* 6 HDD bays
* 14" front fan

It's too bad 'Egg doesn't carry it (yet). I read a review on R2 some time ago, but back then none of Fractal Design cases were available in the U.S. It's just a matter of a last few month their case became more available in the U.S., so 'Egg may stock it up soon.

Meanwhile, I'll check out the HP ProLiant server.

Oh, 1 more thing. Do you think my choice of the UPS sufficient for the server? Shutting NAS down immediately in case of outage, I thought a low capacity APC Back-UPS would suffice. But, I may speced it a little too low.

Kesh

 

[survive]

Hi kesh,

The UPS only really needs to power the box for long enough for it to shutdown....figure that if you are on batteries all the clients are already off, so you don't really need more runtime than a couple of minutes.

Here is the output graph for the UPS:

http://www.apc.com/resource/include/techspec_index.cfm?base_sku=BE650G

Honestly it seems a little close than I like, but that's because i don't really know what you box will end up drawing. Be sure to try it (set it up & yank the plug) so you know for sure.

You might want to check the microserver thread and see if anyone a put one on a Kill-a-watt, or check out any reviews on the hardware sites...somebody must have already come up with this information.

I know the NIC is to much....it's silly to pay more for the NIC than the whole board, but my NIC suggestions are loaded with bias. I just don't like the crab:

http://rvxtm.com/wp-content/uploads/2010/03/1231601932_realtek_logo.png

The more I think about it the more I think the Microserver is the way to go....besides, if you get your drives & ram it's pretty easy to simply give it a shot & if it doesn't perform well, send it back...you will use the other components on the next thing you try . You might also want to check with you campus IT people & see if you have an actual HP rep and put the squeeze on them...see if they can't get you one.

-Will

 

[kesh]

Will,

I do like the form factor of the HP MicroServer a lot (especially as my home server) but it would be a more expensive solution than a Zacate build. The only benefit (that I can think of) would be that it uses unbuffered ECC memory (although it puts $40+ premium over regular DDR3 memory).

Realistically, what are the odds of memory messing up the data files? As I commented earlier, the traffic on this server will not be very heavy (max ~5 users at a time). Is the memory-intensive nature of ZFS increases the chance of data corruption?

Kesh

P.S., I just realized that in my OP, I meant RAID1 (mirroring) not RAID0 :o When I first read your first reply, I was a little puzzled by your (decisive) comment to never use RAID0...

 

[survive]

Hi Kesh,

It might be more expensive, it might not...tough to say. The nice thing about buying a complete solution is that it's a complete solution. no putting together parts, no monkeying with a case..just buy it and go. It also makes it a pretty easy purchase to justify when you get down to it. You might even be able to get 3 years worth or warranty on it if you get it through your HP rep (if your school has one) not that it matters really, but a 3 year warranty fits with into a system refresh program and gives the PI plenty of time to budget a replacement. There all all sorts of intangibles that go along with something off the shelf....

One thing to keep in mind is that the Microserver is actually a low power Athlon chip, which is significantly more powerful than the Zacate board.

Re: ECC memory:

Who knows what the odds are, it's up to you to decide if it's worth the risk or not, but there's a reason that servers use ECC memory. Is it worth $40 bucks? Will it cost them more than $40 for you to restore a corrupt file? What if that corrupt file throws off 100 hours of analysis? Is it worth it then? Cheap insurance is cheap. ZFS is indeed memory intensive, and the filesystem check-summing can conrtol some of that risk, but if the data is corrupted on the way to the disk it won't help. Tough to say really.

I apologize if I came off abrupt in my initial response, but I have these sort of conversations way too often. Usually though it's more someone looking for validation of their plan than actually seeking advice. Even though we (the IT guys) will wind up having to support it no matter what, it's always good to get the concerns out in the open. Honestly, if I may speak for IT guys everywhere, I want you to put together a system that's going to cause you the fewest headaches, because that causes me the fewest headaches.

-Will

 

[kesh]

Will,

You're not coming off abrupt at all, and your comments from IT perspective are extremely helpful. I'm trying to shift my ways of thinking towards IT guys', away from PC no-gaming enthusiast's; however, the cost is a huge constraint in my system. So, I'm seeking for a good cost-reliability tradeoff point.

Warranty. So true... didn't even think about it. Funny how what often is my reason not to buy Dell, HP, etc. for my home machine is now a good thing to have.

ECC memory. +$40 wouldn't a too big of a deal, I was just curious how significant of odds we have for memory I/O errors to occur (more from my engineering self).

ECC feature + HP warranty/support is a good combo to justify the HP.

Let me ask you a couple basic questions.

1. If the budget only allows 2-3 HDDs, what are my best options? 2HDD=ZFS RAID1, 3HDD=ZFS RAID-Z?
2. How easily can I upgrade RAID1/RAID-Z to RAID-Z2 later when more funds are available for more HDDs?
3. Returning to starting point, why not use a cheaper off-the-shelf NAS box? e.g., Synology DS411J, Thecus N4100PRO, or QNAP TS-410-US to name a few (off newegg). How do they stick up with FreeNAS? Silly me, but I realized that my initial reasoning for choosing FreeNAS was to reuse a retired PC (thus keeping the cost low), but that plan somehow :o snowballed into building a new system with 4 HDDs.

 

[Bohs Hansen]

not to butt in, but in all benchmarks i've seen, the zacate scores better then the atom d525, also noteworthy in case you want to use ZFS, the atom structure only allows for 4gb ram vs the 8gb max of the zacate :)
That aside, I wish you the best of luck finding the solution perfect (or as much as possible) for you.

 

[survive]

Hi Kesh,

1 & 2) You could start with a 2 drive mirror and add a second mirror but that's about it. That's one of the funny things about ZFS, for as much as it can do you can't expand an existing "virtual device"....when you make a ZFS filesystem ZFS first creates a virtual device from your disks and then puts a volume on top of them. You can add a second virtual device and the volume will grow to include it, so in your case you would end up with something analogous to a RAID 10 system underneath. I know it sounds weird, but this is what allows you to create the monster filesystems people build with ZFS...nobody in their right mind would make a 72 disk raidz2 volume, but they might put a dozen 6 disk raidz devices into a volume. I would say your best bet (with a microserver) is to build it all at once with 4 disks in the volume configuration you want and be done with it.

3) Have you priced out any of those boxes? The Synology (on newegg) is the cheapest at $300 and you go up from there. I can't say that they are bad, I actually have a professor who built himself an 8 disk QNAP that I'm on the hook for (see...see what happened there?) and it hasn't blown up yet...it seems nice... If you want to investigate them check out each makers forums, maybe cold-call the support guys and see what they are like.

If you have the parts for a proof-of-concept build it. See if it works for what you want...performance might not be that good but does it do what you need?

You might also want to see what one of these would run....it should be pretty turn-key:

http://www.ixsystems.com/ix/storage/titan-freenas/titan-freenas-mini

-Will

 

[kesh]

@Bohs - Thanks for your tip on Atom vs. Zacate. I just glanced over your blog, and the performance difference between your FreeNAS & QNAP boxes are huge! With each of our data files typically is in 500 MB - 1 GB range, I may need to be more attentive to the transfer speed specs.

@Will - Call me silly (again). $300+ off-the-shelf NAS solutions (w/o disks) would be as expensive as Microserver + FreeNAS, and if they'll be slower than FreeNAS, well, what's the point of considering them?

In any case, guys, thank you very much for your help/insights, I'll give it a few more days of thoughts and present a few alternatives to boss to decide on which route to take.

Cheers,
Kesh

 

[survive]

Hi kesh,

Yes, that's exactly my point in answering #3 above. I wouldn't get an off the shelf solution, but that's not to say there isn't any reason to get one. I think the most obvious reason would be support. If you buy a QNAP\Thecus\Synology device and use disks from their approved list you darn well best be able to get some help with it if it breaks.

That's the reason I included the link to the ix systems box. You buy something from them and I think it's reasonable to expect you will get a good fast trouble-free storage box. For folks who do stuff other than IT a turn-key solution can be pretty compelling!

-Will

 

[kesh]

Will,

I put together both HP microserver and complete DIY solutions, and they actually came out to be pretty even (right around $1400).

Now, I'm trying to come up with a sub-$1000 option, and really the only place where I can significantly reduce the cost are the HDDs. Meanwhile, you suggested earlier that

1 & 2) [snip] I would say your best bet (with a microserver) is to build it all at once with 4 disks in the volume configuration you want and be done with it.

and that going from RAID1 to RAID-Z2 requires complete rebuilding of the RAID array (while data being backed up elsewhere). If you can only afford $1000 now but will have more funds within the next 2 years, which one do you recommend?

A) Buy 2 enterprise-grade HDDs and set up RAID1 (and decide how to deal with the upgrade later) or
B) Buy 4 consumer-grade HDDs and replace them all to 4 enterprise-grade HDDs later. These 4 freed HDDs could then be used as back-up drives.