FreeNAS with EMC arrays

[panivino]

Our business does datacenter hardware remarketing, and we often come across high end storage stuff that is mostly scrapped. I’m looking into giving part of these components a “second life”, and I’ve thought FreeNAS might be an option. The issue with high end storage solutions is that the controllers are mostly useless without licenses, but the storage arrays and drives are in perfect working order. Proprietary cabling is sometimes necessary, but this is also easy to come by for us. The following is an example of what I’m trying to do, and I’d like to hear your opinion.

I’d use an HP DL360 G5 or similar with the following specs:
1 x XEON QC at 3GHz
32 GB of RAM (64 is possible but really expensive)
1 x P400i RAID controller with BBU.
3 x 36 GB hotswap 2,5” 15 K SAS Hard drives
1 x 8Gbit HBA (Dell G425C) This is necessary because of the proprietary cables used by the EMC devices described below.
To the HBA I will connect, Daisy chained, between 1 and 8 EMC arrays. These are “dumb” fibre channel devices that come from a DMX4-950. This would give me a total of between 30 and 120 drives. Right I'd test with 400 GB 10K RPM fibre drives. (Total 48 TB max)
The devices are recognised by the HBA, and I have been able to use DBAN (Linux based) to wipe the drives. I’m seeing about 400 MB/sec as the arrays are only 4Gbit. I could of course use more HBA’s in a different model server to increase bandwidth.

I will install FreeNas on a hardware RAID 1 + 1 hotspare, on the P400i Controller, using 36GB 15K SAS drives. I know I won’t be able to use the space on those drives but I (obviously) don’t care.

The 30 to 120 drives on the storage arrays should be seen by FreeNAS as JBOD, the HBA does no hardware raid, of course.

So my questions:
1. I know it's recommended to put the OS on a USB stick but … because wasted space on the small SAS drives is not an issue, and because of reliability I think the HDD option might be better. Would you agree with me or should I still use your recommended USB method?
2. Would I benefit from a second CPU on the DL360 or would it make no difference?
3. Do you know if the rest of the hardware will be compatible with FreeNAS?
4. Performance isn’t that important, but I would like the whole thing to be reliable. Should I be OK?
5. If using the max number of drives, would I need more RAM?
6. Does using FreeNas for this even make sense … or is it too much for FreeNas to handle? (no offence intended!). I know it’s a bit of an open question but anyway …

Thanks in advance.

 

[L]

I actually would never run a commercial system on a single usb stick. But I would if I could mirror them as I can in 9.3.

Also the fc support is questionable. I would check the freebsd hardware compat list.

 

[cyberjock]

AFAIK there is no official fiberchannel support in FreeNAS at all without hacking. Even then, it's very limited in "what works". If you must go FC then I'd definitely look elsewhere unless you want to spend some time hacking away at the OS to get FC to work in whatever state it will work for your hardware.

Most of your questions depend on your use-case and such. So i can't comment on them. Hardware may or may not work (I don't have that hardware so I can't test it), but the RAID controller is not even close to a good idea. Repurposing is great, but this is going to take some money to make work, particularly for the RAID controller.

 

[sfcredfox]

I'll skip the questions answered or that I can't help with, and I'm not an expect or even that awesome in the world of FreeNAS/ZFS, but I'll try to help anyway.

2. Would I benefit from a second CPU on the DL360 or would it make no difference?

Some functions are single threaded such as CIFS (I believe still the case), so it's better to have a faster processor than more cores for that. I believe that the iSCSI target CTL is multi-threaded now, so it can benefit from more processing power. If you have a need to do silly/complicated stuff like encryption or depud, you'll obviously need the processing power, and way more memory than the G5 platform can handle, so I'd suggest 'No' on those extra features.

3. Do you know if the rest of the hardware will be compatible with FreeNAS?

I can't speak to EMC stuff, sorry, but I use an HP MSA70 (2U, SAS enclosure with 2.5" single port SAS, connected via 8088 and LSI HBA) FreeBSD doesn't care about the enclosure, probably just the controller. Unless you find someone who's actually doing it now, and has been doing it for a long time with no issues, you're on your own. Maybe it's on the FreeBSD list, but people round here are still skeptical until it's been used by lots of people for a long time. You might just have to test it for a while and see.

4. Performance isn’t that important, but I would like the whole thing to be reliable. Should I be OK?

The OS is known to be extremely reliable, if you haven't seen the comercial version, it's truenas. If you are extremely concerned about reliability, I think the general attitude would be 'you need to use the hardware that is recommended and known to excellent' rather than some stuff that might work. I have learned that just because you have some enterprise quality stuff, doesn't mean FreeNAS will love it. FreeNAS loves what it's been designed and developed for from what I gather.

5. If using the max number of drives, would I need more RAM?

Everyone will tell you more RAM is better. Some people get ridiculous and say you need a metric crap ton. From what I have learned, and what the people who really know ZFS, it loves crap tons of RAM. Read all the product documentation and the newbie guide if you're serious about getting into this. I'm not saying 32GM won't work, it just depends on what you plan to do with your system and how much work you think it's going to do.

6. Does using FreeNas for this even make sense … or is it too much for FreeNas to handle? (no offence intended!). I know it’s a bit of an open question but anyway …

I think you can go see TrueNAS and their customer list for this, but of course those systems are designed very specifically, with the right hardware, the right config, etc for the work load. I think it would be difficult to answer that question for any one person without knowing everything about what they want to do and expect the system to handle.

If you notice, my lab system is very similar to what you're talking about, but it's just that, a lab. It does what I need, and it does it well, but I don't run corporate/important data on it since I'm not really an example of a 'perfect' freenas system. I don't have to worry about it crashing because I didn't follow the best practices for hardware selection like you're talking about. If you're worried about a resume generating event, you probably want to think about following the guides/manual/advice/newbie guide much closer to get the real solid reliability out of this OS. Not doing so introduces some risk that you have to except and hope for the best. Maybe a chance to test your backup scheme :)

Hope that helps some.

*edit* after thoughts, you'll want to loose all the RAID controllers and use HBAs only if you want to play with this. Look for articles on IBM M1015 SAS controller, it's the favorite and cheap.

 

[panivino]

I'll skip the questions answered or that I can't help with, and I'm not an expect or even that awesome in the world of FreeNAS/ZFS, but I'll try to help anyway.

Some functions are single threaded such as CIFS (I believe still the case), so it's better to have a faster processor than more cores for that. I believe that the iSCSI target CTL is multi-threaded now, so it can benefit from more processing power. If you have a need to do silly/complicated stuff like encryption or depud, you'll obviously need the processing power, and way more memory than the G5 platform can handle, so I'd suggest 'No' on those extra features.

I can't speak to EMC stuff, sorry, but I use an HP MSA70 (2U, SAS enclosure with 2.5" single port SAS, connected via 8088 and LSI HBA) FreeBSD doesn't care about the enclosure, probably just the controller. Unless you find someone who's actually doing it now, and has been doing it for a long time with no issues, you're on your own. Maybe it's on the FreeBSD list, but people round here are still skeptical until it's been used by lots of people for a long time. You might just have to test it for a while and see.

The OS is known to be extremely reliable, if you haven't seen the comercial version, it's truenas. If you are extremely concerned about reliability, I think the general attitude would be 'you need to use the hardware that is recommended and known to excellent' rather than some stuff that might work. I have learned that just because you have some enterprise quality stuff, doesn't mean FreeNAS will love it. FreeNAS loves what it's been designed and developed for from what I gather.

Everyone will tell you more RAM is better. Some people get ridiculous and say you need a metric crap ton. From what I have learned, and what the people who really know ZFS, it loves crap tons of RAM. Read all the product documentation and the newbie guide if you're serious about getting into this. I'm not saying 32GM won't work, it just depends on what you plan to do with your system and how much work you think it's going to do.

I think you can go see TrueNAS and their customer list for this, but of course those systems are designed very specifically, with the right hardware, the right config, etc for the work load. I think it would be difficult to answer that question for any one person without knowing everything about what they want to do and expect the system to handle.

If you notice, my lab system is very similar to what you're talking about, but it's just that, a lab. It does what I need, and it does it well, but I don't run corporate/important data on it since I'm not really an example of a 'perfect' freenas system. I don't have to worry about it crashing because I didn't follow the best practices for hardware selection like you're talking about. If you're worried about a resume generating event, you probably want to think about following the guides/manual/advice/newbie guide much closer to get the real solid reliability out of this OS. Not doing so introduces some risk that you have to except and hope for the best. Maybe a chance to test your backup scheme :)

Hope that helps some.

*edit* after thoughts, you'll want to loose all the RAID controllers and use HBAs only if you want to play with this. Look for articles on IBM M1015 SAS controller, it's the favorite and cheap.

Thanks a lot for for your feedback. I've gone the "you might just have to test it for a while" way, and ... guess what ... everything worked (almost) out of the box. The single volume on the P400i was recognised just fine and FreeNAS installed on it OK. As for the HBA initially it wasn't recognised but enabling the "Host Adapter Bios" option in the card's configuration solved that. It's just a Qlogic 2560 BTW, I come across these all the time and would've been surprised if at least FreeBSD didn't support them.

I'm now playing around with it ... and it seems to be working fine. I'll put it through its paces over the next few weeks and post any results or gotchas in this thread.

One last question ... your setup is indeed similar to mine. From what I know a lot of the functionality on an array (including my EMC's or your MSA) ir handled by the proprietary controller. On your system, do the LEDs actually work properly on the HDD's? I don't know if you've ever had a drive fail, but if so ... does the right LED indicate a failed drive? Figuring out which drive is bad when you have over 15 of them is almost impossible ...

 

[anodos]

One last question ... your setup is indeed similar to mine. From what I know a lot of the functionality on an array (including my EMC's or your MSA) ir handled by the proprietary controller. On your system, do the LEDs actually work properly on the HDD's? I don't know if you've ever had a drive fail, but if so ... does the right LED indicate a failed drive? Figuring out which drive is bad when you have over 15 of them is almost impossible ...

I personally use the super high-tech method called a label-maker and spreadsheet. If things are configured correctly, freenas will email details of failed/failing hard drive which then can be looked up on the spreadsheet.

Let's face it, if you're not using a label-maker for everything then you're doing it wrong. Label all the things! :)

 

[panivino]

I personally use the super high-tech method called a label-maker and spreadsheet. If things are configured correctly, freenas will email details of failed/failing hard drive which then can be looked up on the spreadsheet. :)

Let's face it, if you're not using a label-maker for everything then you're doing it wrong.

You nerd you ... I'll use my chisel and a rock, not some flimsy spreadsheet. ;-)

 

[sfcredfox]

One last question ... your setup is indeed similar to mine. From what I know a lot of the functionality on an array (including my EMC's or your MSA) ir handled by the proprietary controller. On your system, do the LEDs actually work properly on the HDD's? I don't know if you've ever had a drive fail, but if so ... does the right LED indicate a failed drive? Figuring out which drive is bad when you have over 15 of them is almost impossible ...

Crap, I can't remember. I have already hot-swapped out a bad drive. I want to say no, so in case it does, you'll be pleasantly surprised rather than disappointed.

I want to say the activity light (green) was off when I pulled the drive. Maybe it goes out after you follow the procedure for changing a bad drive and the system stops using the drive. I just cant remember.

After I did a backup, I had to do the same method (kinda) as anodos. All my drives are in vDevs of 5 drives, so I counted them to double check I was pulling the right one. I double checked a few times :)

Side note, I think you must have an HP RAID controller to flash it's firmware. I use an LSI HBA to connect my MSA using the SFF-8088, but I already brought its firmware to current.

 

[sfcredfox]

I personally use the super high-tech method called a label-maker and spreadsheet. If things are configured correctly, freenas will email details of failed/failing hard drive which then can be looked up on the spreadsheet.

Are you also using an MSA or some other JBOD, or were you talking about internals?

You guys should check out my post about using externals, I'm interested in feedback:
https://forums.freenas.org/index.php?threads/external-jbod-enclosures-what-are-you-using.25206/

 

[anodos]

You nerd you ... I'll use my chisel and a rock, not some flimsy spreadsheet. ;-)

Or to keep things simple you could always write a script that causes the server to beep the number of the failed drive in Morse code.

 

[sfcredfox]

One last question ... your setup is indeed similar to mine. From what I know a lot of the functionality on an array (including my EMC's or your MSA) ir handled by the proprietary controller. On your system, do the LEDs actually work properly on the HDD's? I don't know if you've ever had a drive fail, but if so ... does the right LED indicate a failed drive? Figuring out which drive is bad when you have over 15 of them is almost impossible ...

One last thing I just remembered: When I wanted to confirm that I had the right drive (the bad one), I ran a SMART test against it, and then found the drive that had a solid activity light.

 

[panivino]

Are you also using an MSA or some other JBOD, or were you talking about internals?

You guys should check out my post about using externals, I'm interested in feedback:
https://forums.freenas.org/index.php?threads/external-jbod-enclosures-what-are-you-using.25206/

Just to clarify, what I'm using is the following:

HBA: Dell G425C. This is a QLE2560, an 8GB Single Channel PCI-E Fibre Channel thingy.
I had to use this HBA because the FC arrays mentioned in my first post use HSSDC.
The HBA is different to the usual FC ones in that it does not have an integrated mini-GBIC. In an EMC DMX4-950 the connections between the controllers and array are electric, not optical. I'm pretty sure there's a cheaper way, as the G425C is quite expensive (about 250 US). Because of the array speeds a 4Gbit controller would be just fine. But I haven't been able to find a 4Gbit HBA without an integrated mini-gbic.

The external array cabinets are KTN-STL4. These are 15 drive "dumb" fibre arrays. It was pulled from an DMX4-950. This is a (in its day very expensive very high end) EMC storage solution.

The controller in the back is an EMC P/N 100-562-126. Each array has two of these, for redundancy. Each controller has two ports. I'm only using one controller for now, but I'm sure redundancy can be implemented in some way by using both.

The way I've connected is: server <----> array 1 <-----> array 2 <------> array 3 etc. etc.
There is a small switch on each controller that lets you give the array an ID from 1 to 8. Hence maximum drives = 8 arrays x 15 drives = 120 drives.

Cable (computer - array 1): EMC P/N 038-003-546. This is a DAE to HSSDC cable.
Cable (array 1 to 2, 2 to 3 etc.): EMC P/N 038-003-508. HSSDC to HSSDC cable.

As I've mentioned before, FreeNAS recognises all drives just fine as individual disks, after enabling "Host Adapter BIOS" in the controller's "Adapter Settings" menu.

Now, I'm a real newbie at FreeNAS ... and I thought deciding partition sizes for linux installs was complicated .... ;-)
So by no means am I saying this is reliable. But I'll play around with it and let y'all know if there are any problems, but initial impressions are good.

 

[panivino]

Crap, I can't remember. I have already hot-swapped out a bad drive. I want to say no, so in case it does, you'll be pleasantly surprised rather than disappointed.

I want to say the activity light (green) was off when I pulled the drive. Maybe it goes out after you follow the procedure for changing a bad drive and the system stops using the drive. I just cant remember.

After I did a backup, I had to do the same method (kinda) as anodos. All my drives are in vDevs of 5 drives, so I counted them to double check I was pulling the right one. I double checked a few times :)

Side note, I think you must have an HP RAID controller to flash it's firmware. I use an LSI HBA to connect my MSA using the SFF-8088, but I already brought its firmware to current.

But as you'll probably know the drives on an MSA or the drives on the P400i actually flash orange when they're bad, at least when using HP's RAID functionality.
I also know the EMC's I'm using do have orange LEDS to indicate failure ... these actually work with some commercial erasure software, but not with DBAN, for example.
It might sound silly but having a little orange LED flashing to indicate a failed drive is just so much more reliable IMHO than relying on a spreadsheet or a green light that's off. I'll try using a drive that I know is bad and see what happens ... pulling a drive doesn't work on the EMC's because the LEDS are integrated in the bracket instead of on the backplane.

 

[Ericloewe]

I'm under a self-imposed ban on label making machines. I am only allowed to own A4 sheets of labels which are a pain to print.

Otherwise, I'd end up labeling the cat.

 

[anodos]

I'm under a self-imposed ban on label making machines. I am only allowed to own A4 sheets of labels which are a pain to print.

Otherwise, I'd end up labeling the cat.

I personally like A4 labels. If you have access to a paper cutter then you can create your own custom label sizes. That being said, A4 is actually a good size for a cat label.

 

[panivino]

I personally like A4 labels. If you have access to a paper cutter then you can create your own custom label sizes. That being said, A4 is actually a good size for a cat label.

PLEASE will you just STOP spoiling my thread with your silly comments. It's obvious that A3 labels are much better, I can label my cat-eating dog with those!

 

[fifa]

Hi panivino, I want to know if you are still running this setup? If so how reliable is it? Are you getting a 4Gb FC link between the server and EMC?? Do the drive LED's working accordingly? Thanks

 

[RegularJoe]

PLEASE will you just STOP spoiling my thread with your silly comments. It's obvious that A3 labels are much better, I can label my cat-eating dog with those!

I think the experimental part of FreeNAS FC support is when it is a target.

You should be able to use a FC HBA as long as it is supported by the OS, as you have seen you may have to enable the BIOS on the FC HBA. Block devices are block devices. The only thing I would be worred about is if the smart reporting works right as there mayu be some slight differences in how a FC drive reports bad blocks and sectors. In SATA/ATA land that was Smart v1, v2, v3... and drives older than 2004 I would not use per the SMART spec

"Depending on the type of interface being used, some S.M.A.R.T.-enabled motherboards and related software may not communicate with certain S.M.A.R.T.-capable drives. For example, few external drives connected via USB and Firewire correctly send S.M.A.R.T. data over those interfaces. With so many ways to connect a hard drive (SCSI, Fibre Channel, ATA, SATA, SAS, SSA, and so on), it is difficult to predict whether S.M.A.R.T. reports will function correctly in a given system."

http://en.wikipedia.org/wiki/S.M.A.R.T.

 

[RegularJoe]

you must do scrubs....

"A field study at Google covering over 100,000 drives during a nine-month period found correlations between certain SMART information and actual failure rates. In the 60 days following the first uncorrectable error on a drive (SMART attribute 0xC6 or 198) detected as a result of an offline scan, the drive was, on average, 39 times more likely to fail than a similar drive for which no such error occurred. First errors in reallocations, offline reallocations (SMART attributes 0xC4 and 0x05 or 196 and 5) and probational counts (SMART attribute 0xC5 or 197) were also strongly correlated to higher probabilities of failure. Conversely, little correlation was found for increased temperature and no correlation for usage level. However, the research showed that a large proportion (56%) of the failed drives failed without recording any count in the "four strong S.M.A.R.T. warnings" identified as scan errors, reallocation count, offline reallocation and probational count. Further, 36% of drives failed without recording any S.M.A.R.T. error at all, except the temperature, meaning that S.M.A.R.T. data alone was of limited usefulness in anticipating failures.[3]"