IBM SERVERAID M1015 Cabling to support 16 devices

[Spud]

Hello again,

I now have one of these IBM Serveraid M1015 cards and after reading this

http://www.redbooks.ibm.com/abstracts/tips0740.html

I see that the card is capable of supporting 16 devices (Although I thought I read somewhere else it was 32) but I can live with 16.

So my question is what cables would I need to purchase to be able to support 16 devices and is there any preferred brand of cable to use over another?

Also, I take it WD reds work without any problems with this card?

Thanks

 

[jgreco]

The M1015 should support up to at least 32, but there's a caveat: it only has eight lanes. It's essentially a 9240-8i or wotevr. It can DIRECTLY attach to eight devices. You can add a SAS expander to it which loses you four lanes but then gets you the additional devices, but usually the cost of the expander exceeds the cost of the M1015 (at least on eBay).

So this isn't just a cabling issue. The IBM redbook may be assuming that it is integrated into an IBM chassis that has an expander, but I'll also say it isn't exactly clear why they're saying 16.

 

[jgreco]

p.s. https://forums.freenas.org/index.php?threads/confused-about-that-lsi-card-join-the-crowd.11901/

 

[Spud]

Thanks!

 

[jgreco]

Not a problem. Getting your head around unfamiliar technology is always fun.

 

[Spud]

Not a problem. Getting your head around unfamiliar technology is always fun.

Yes thats very true and although I google all the questions I ask here sometimes you just can't seem to get a straight answer to the question at hand.

So I'm going to ask a few more here if I can. :)

Going on what you said about price for a SAS expanded like the Intel raid controller res2sv240 would I be better off just buying a second M1015 and not worrying about the SAS expander altogether. Thats going to give me a total of 16 devices which is plenty.

Now the only problem I can see with using the Intel RES2SV240 is that its going to slow things down to some of the drives by only connecting to 6 of the WD reds at 3GB instead of 6GB which the other 4 reds would get from plugging them straight into the other port on the M1015.

So by connecting some of the drives at 3GB that would slow down scrub times I'd think wouldn't it?

So in your opinion would I be better off with 2xM1015 or 1xM1015 and a RES2SV240 expander?

And I haven't found anything on connection/compatibility problems using WD reds on a M1015 or RES2SV240 so far.

thanks for the help

 

[marbus90]

if 1 card supports 16 devices total, connect two sff-8087 cables to the expander. that is 8x 6Gbps. Means you get 300MB/s troughput per SATA port on the expander. Means you need SSDs to notice a slight decrease in performance.

 

[jgreco]

if 1 card supports 16 devices total, connect two sff-8087 cables to the expander. that is 8x 6Gbps. Means you get 300MB/s troughput per SATA port on the expander. Means you need SSDs to notice a slight decrease in performance.

Well, saying 16 adds confusion. Say 32 so it's more-clear that you need the expander for that.

In any case, the basic point is correct. Connecting some hard disks at 3Gbps rather than 6Gbps makes almost no difference at all. The newest generation of WD Red 6TB drives is capable of pushing just a little more than 200 MB/sec peak, which is very approximately 1.6Gbps (200 * 8 but actually probably closer to 2Gbps once overhead is factored in). So handwringing about "3Gbps" vs "6Gbps" is sort of meaningless because your drive can't go that fast. The 6Gbps link for a hard drive merely reduces latency a little bit.

Are you actually planning to support more than 16 devices? Because you can use a combination of the M1015 and the mainboard ports if that is sufficient. You can of course also get a second M1015 and there's nothing wrong with that, used on eBay for about $100-$125 these days is probably cheaper than the expander. You seem to be talking about 10 Red's, so if that's what you're looking to service, as long as your mainboard ports aren't some crap like Marvell, maybe use them. Like the Intel PCH-supplied SATA ports are known-good. The second M1015 would be burning another 10 watts for no return on investment if that were the case.

 

[Spud]

Well, saying 16 adds confusion. Say 32 so it's more-clear that you need the expander for that.

Are you actually planning to support more than 16 devices? Because you can use a combination of the M1015 and the mainboard ports if that is sufficient. You can of course also get a second M1015 and there's nothing wrong with that, used on eBay for about $100-$125 these days is probably cheaper than the expander. You seem to be talking about 10 Red's, so if that's what you're looking to service, as long as your mainboard ports aren't some crap like Marvell, maybe use them. Like the Intel PCH-supplied SATA ports are known-good. The second M1015 would be burning another 10 watts for no return on investment if that were the case.

I'd love to be able to use the onboard sata/sas ports on the x10sf7-f but no matter what I try the WD reds or greens just won't show up and aren't on the "compatibility list" which is one thing I over looked when buying the MB. Never even thought to check a compatibility list to be honest I just assumed a sata HD would work on a sata port, how wrong I was.

And googling that sata/hd incompatibility has come back with sweet bugger all as well.

Oh and yes down the track I probably will expand the system with more HD's but this time they will be something I know works on the SAS ports.

 

[Ericloewe]

I'd love to be able to use the onboard sata/sas ports on the x10sf7-f but no matter what I try the WD reds or greens just won't show up and aren't on the "compatibility list" which is one thing I over looked when buying the MB. Never even thought to check a compatibility list to be honest I just assumed a sata HD would work on a sata port, how wrong I was.

And googling that sata/hd incompatibility has come back with sweet bugger all as well.

Oh and yes down the track I probably will expand the system with more HD's but this time they will be something I know works on the SAS ports.

Hold it there.

Every last SATA/SAS port on the X10SL7-F is known good for pretty much any drive - WD Greens and Reds included, as several people can attest to.

SATA from the PCH should just work (as long as it's set to AHCI in BIOS).
SATA/SAS from the LSI SAS 2308 controller requires that it be flashed to P16 IT mode (like the M1015 - in fact, they're nearly the same thing).

If that doesn't work, you have a very, very defective motherboard.

 

[Spud]

Hold it there.

Every last SATA/SAS port on the X10SL7-F is known good for pretty much any drive - WD Greens and Reds included, as several people can attest to.

SATA from the PCH should just work (as long as it's set to AHCI in BIOS).
SATA/SAS from the LSI SAS 2308 controller requires that it be flashed to P16 IT mode (like the M1015 - in fact, they're nearly the same thing).

If that doesn't work, you have a very, very defective motherboard.

I have tried 2T Reds, 3T Reds, 2T Greens, 880.8G Greens and an old 74G Raptor on all ports and the only thing that will show up in the Bios is a DVD ROM that I installed freenas with to a USB. Nothing shows up in on any SAS ports.

Seeing the DVD showed up in the BIOS I kind of ruled out a fault with the MB and then checked the supermicro website and found none of the drives I have were compatible with the x10sl7-f thats what I thought the problem was.

I have flashed the SAS controller to IT mode but its version P19 that shouldn't be stopping the controller from at least seeing the drives and them showing up in the config utility. I mean down grading to version P16 is only needed to keep freenas happy isn't it?

 

[Z300M]

I have tried 2T Reds, 3T Reds, 2T Greens, 880.8G Greens and an old 74G Raptor on all ports and the only thing that will show up in the Bios is a DVD ROM that I installed freenas with to a USB. Nothing shows up in on any SAS ports.

Seeing the DVD showed up in the BIOS I kind of ruled out a fault with the MB and then checked the supermicro website and found none of the drives I have were compatible with the x10sl7-f thats what I thought the problem was.

I have flashed the SAS controller to IT mode but its version P19 that shouldn't be stopping the controller from at least seeing the drives and them showing up in the config utility. I mean down grading to version P16 is only needed to keep freenas happy isn't it?

None of the Seagate drives I'm using (see my sig. -- and I've used a couple of other kinds temporarily too) is on the Supermicro compatible list, but they all work just fine.

I have an M1015 in addition to the on-board SAS ports, but only the drives attached to the M1015 show up in the boot screen -- yet FreeNAS accesses them all.

 

[mjws00]

Version doesn't matter for just connectivity. I've certainly got a mix of drives on all my ports. Did you mess with bios settings? Maybe reset and see what's up. There are no compatibility issues that I've ever seen or heard of.

 

[Ericloewe]

Bad power could also be causing the issues.

 

[marbus90]

I have tried 2T Reds, 3T Reds, 2T Greens, 880.8G Greens and an old 74G Raptor on all ports and the only thing that will show up in the Bios is a DVD ROM that I installed freenas with to a USB. Nothing shows up in on any SAS ports.

Seeing the DVD showed up in the BIOS I kind of ruled out a fault with the MB and then checked the supermicro website and found none of the drives I have were compatible with the x10sl7-f thats what I thought the problem was.

I have flashed the SAS controller to IT mode but its version P19 that shouldn't be stopping the controller from at least seeing the drives and them showing up in the config utility. I mean down grading to version P16 is only needed to keep freenas happy isn't it?

ODDs are different from HDDs. Also, check out the Virtual Console and Virtual Media trough IPMI.

In case the HDDs won't work in FreeNAS even when flashed to P16 IT mode, then the controller is (partly) broken and that board needs to go to RMA.

 

[Spud]

Ok I have now reset the BIOS and installed P16 in IT mode.

I have installed a brand new Corsair RM550 80+ Gold power supply with this build so I don't think its a power issue

Booted the system and still nothing showing up in the BIOS or SAS controller.

Let the system start Freenas 9.3 click on storage then volume manager and no discs there either.

I don't understand how the MB could be faulty if its showing the DVD ROM in the BIOS

Why is it that if I plug in the DVD ROM its showing up in the BIOS but no HD's ever show up, could this still mean the MB is faulty even though the DVD is showing up?

thanks

 

[mjws00]

Is this the same on the pch ports as well? Seems pretty odd that the controller would flash nicely then misbehave. But anything can happen. I'm always pretty slow to blame hardware, especially quality gear. Not much you can do. Swap psu, cables, drives, wipe/reset the board. I'd want to try a a single known good drive no usb or dvd boot device. After that get a new one. This is supposed to be the easy part. :)

 

[Spud]

Is this the same on the pch ports as well? Seems pretty odd that the controller would flash nicely then misbehave. But anything can happen. I'm always pretty slow to blame hardware, especially quality gear. Not much you can do. Swap psu, cables, drives, wipe/reset the board. I'd want to try a a single known good drive no usb or dvd boot device. After that get a new one. This is supposed to be the easy part. :)

Yep its the same for all the ports, every single one of them. The only device that will show up is a DVD Rom on the sata ports.

I have already tried all the cables that came with the MB and a few I had laying around here. I've also tried powering the HD's from a different power supply, no go...

I'll pull the USB stick and see if that makes any difference.

Thanks for the input!

 

[mjws00]

Interesting problem. Sata dvd works but no drives is definitely a new one. Unfortunately it doesn't matter how good we are if the board has a physical problem. Try not to let it taint your view too badly. Your choice was excellent, you just got unlucky.

 

[Ericloewe]

It definitely looks like a bad board. Still, quite a specific problem to have, over two separate controllers.