Hosting a 16 drive DAS (MD1000) over.... 1 cable?

[jaypub]

A little confused over what hardware to use for cabling to achieve to best performance out of this surplusy system....

I have:

- MD1000 Dell Powervault
- (15) 2TB SAS Drives populated in powervault
- (2) IBM / LSI cards (mostly M5015, pictured) - but can get whatever card is required here.

As I understand it, the MD1000 is 3GBPS. We would like to have all of the drives as part of a single pool.

I think the connections on the MD1000 are SFF-8470 (see picture).

If the card has 2 ports with "4 channels" each (8 drives?) how I would I get it to support the 15 drives --- and what to do with all those ports on the cards to get them converted to a single (?) SFF-8470 cable?

Could I just notch the bracket and use a SFF8087-to-SFF-8470 SAS cable on a single channel/port from the LSI card to access the DAS?

Could I use the other free port for internal drives on the host server?

If the card is made for 8 drives, wouldn't this all affect performance somehow?
-------

@jgreco seems to be a guru on this topic... hopefully he can chime in!

 

[jgreco]

So.

Unclear if you've seen the SAS primer. See https://www.ixsystems.com/community/resources/dont-be-afraid-to-be-sas-sy.48/ if not!! But some of this is just a little outside the sphere of that.

Bearing in mind that these things can come in a variety of configs, I'm going to suggest that you will need to do a little research. Generally, there's a pair of SAS expanders involved, and these units are typically designed so that two servers can get redundant access to the unit. You would connect each server to both controllers to get full redundancy, which is not really necessary for FreeNAS. It may be necessary to log in to the shelf to make appropriate settings, but I'm not seeing an obvious console port, so that suggests that it doesn't have any onboard RAID capabilities and is probably just a dumb pair of SAS expanders exposing the drives directly, which is great if so.

Unfortunately, SAS 3Gbps expanders can be a little twitchy and I've seen cases where drives and expanders get screwed up and do strange things especially when a drive is failing. Make sure the expander firmware is up to date and the drive firmware updates are applied, if any are available.

Your pictured cable is an internal (SFF8087) cable. Ideally you want an SFF8088 to SFF8470 SAS cable. If you want to use a card with internal connectivity, get an adapter plate. Search eBay "sff-8087 sff-8088". Most of them are dual. That's fine even if you don't use the second port. Really a bad idea to try a janky approach like finding some hack to thread an internal cable out the back of your server.

SAS cards have "channels" or "lanes", which some people confuse as the number of "drives" a card supports. In a basic configuration, a SAS lane can directly connect to a drive. An SFF8087 has four lanes and can directly attach to four drives through a breakout cable. A typical HBA has two SFF8087's so can directly attach eight drives. But this is a trite config.

SAS is a storage "network". You can take a connection and do interesting things with it. For example, if you had four external disk shelves, you can take a single HBA and a single SFF8087 connector on it, connect it to the first disk shelf's primary controller, and then take another port on the primary controller and daisy-chain that down to the next shelf, and repeat that two more times. Suddenly you have 60 disks attached to the server via a single SFF8087.

HBA/RAID cards generally have some configured limit as to how many drives they will handle. This, and the amount of bandwidth available on the SFF8087 (24Gbps for 6Gbps SAS, 12Gbps for your older 3Gbps stuff) are the big limiting factors.

You can get redundant ("multipath") connections to your disk shelves by using two controllers and two cables, one to the shelf's primary controller, one to the secondary. This is only available if you're using SAS drives.

You can usually do an "x8 wideport" with two cables from a single controller to the shelf's primary controller if more bandwidth is necessary, but I've found this doesn't always seem to work as designed and I prefer to avoid it.

Finally, your RAID controllers there need to be replaced with true HBA's. This is not optional. It is a requirement. If you're an IBM shop, IBM M1015, still one of my favorites. Dell, PERC H200 or PERC H310. True LSI, LSI 9211-8i. All need to be crossflashed to IT firmware with P20.00.07.00 firmware.

 

[2nd-in-charge]

Finally, your RAID controllers there need to be replaced with true HBA's. This is not optional. It is a requirement. If you're an IBM shop, IBM M1015, still one of my favorites. Dell, PERC H200 or PERC H310. True LSI, LSI 9211-8i. All need to be crossflashed to IT firmware with P20.00.07.00 firmware.

I'm happy to be wrong, but I don't think there is M1015e with external SAS connectors. Dell H200e certainly exists.

Could I use the other free port for internal drives on the host server?

A 9207-4i4e card will do what you want in a way @jgreco recommends

 

[jgreco]

Well actually I recommended using an internal-to-external SFF8087-to-8088 plate. This is a much better choice for situations where you want to use some internal and a bunch of external drives, which the OP seemed to want the option to do. The -8e HBA's have no good way to allow you to attach internal drives without having an external cable attached and looped back.

The -4i4e offers similar flexibility but is typically a lot more expensive and harder to find. Dell seems to like tossing semi-superfluous RAID controllers into their systems, which means that the H200/H310 (internal ver) are often very cheaply available on eBay. But you can absolutely go the -4i4e route if you wish. SAS is *incredibly* flexible and it is hard to cover all bases.

 

[jaypub]

@jgreco Thank you for the detailed response and links, very helpful! And of course there are just expanders in the MD1000.... the big commercially packaged box threw me off and had me thinking using it as a JBOD/DAS without Dells intended primary unit would take some extra hacking. I guess essentially it is just like the DIY DAS systems out there.... HBA expanders, power, backplane, chassis, done. Thanks for demystifying that bit.

@2nd-in-charge - I went ahead with the card you suggested. Seems reasonable for the card-collecting, flashing, and testing the person selling them has already done. The $20 premium or whatever they are charging over ebay cards in the wild seems worth it since it'll save me the time of doing that. And genuine LSI, FWIW.

So now I just need a SFF8088 to SFF8470 cable and I can give it a go.

Thanks both for the help & tips.