Multipath Old Data

[cyberjock]

Oh, and for the record, there's no reason for you to break multipath on purpose. Its added redundancy for your system in case a disk loses a particular data path it has a backup. So unless you have a good reason to break multipath (and to be honest, I can't think of any kind of weird scenario where you wouldn't want multipath since you've already got the hardware for it) I would leave it alone.

One thing you should note. If you ever do a disk replacement without powering off the system for a multipath-capable system, after you insert the new disk, but before you actually put it in your zpool, you should run the command "service ix-multipath start" as this will create the signature for your new disk. This info should be in the freenas docs at some point, but I don't believe it is at the moment.

 

[bestboy]

It would be interesting to know if iXsystems is actually using "dual linking" (1 HBA ==2 links==> 1 expander) for this FreeNAS certified gear. If this is true, then we can assume dual linking is working contrary to some reports [1][2][3]. This would be good news, because it might keep some people from jumping to SAS3 due to bandwidth concerns.

Not true.. because....

This config wouldn't help bandwidth as one connection is passive. This is only to protect against a cable failure. Your thinking of wide porting.

I have to admit, I'm a bit confused about the options for connecting HBAs to an expander backplane. I blame Supermicro for their vague and lacking documentation of their expanders :/

So you are saying that when I got an 8 port HBA with 2 SFF-8087 connectors and connect both of them to 2 of the 3 SAS2 expander ports, then the second link is just providing 1+1 redundancy as it is passive?

Are you sure that's really the case? Because there are various sources on the interwebz (e.g. here) that state the second connection is active and can be used to double the bandwidth between HBA and expander (i.e. dual linking).

 

[depasseg]

2 of the 3 SAS2 expander ports

The number depends on the backplane. Since you said 3 ports, I assume you are talking about a single path 846e16. One of those ports comes from the HBA and carries 4 channels, or lanes (think of it as 4 SAS or SAS ports all rolled into one connector and cable). The other 2 ports are to allow additional backplanes to be cascaded and branched off as depicted in the following diagrams from the SC846 manual. The redundancy (multipath) comes from the second set of 3 ports (only available on the 846e26).

double the bandwidth between HBA and expander (i.e. dual linking).

I didn't see anywhere on that page that mentioned "doubling". I saw "spreading". But I might have missed it.


http://www.supermicro.com/manuals/other/BPN-SAS2-826EL_1.0.pdf
https://www.supermicro.com/manuals/chassis/tower/SC846.pdf

 

[bestboy]

Hi Greg,
yes I'm referring to the E16 backplanes for the 24 bay chassis 846. The SAS1 and SAS2 variants have 3 ports and the new SAS3 backplanes have 4 ports. But I have read that those ports are not really dedicated for input or output. In fact they are bidirectional and can be used either for connecting to an HBA or to another expander. The manual just gives examples of a few topologies and only for the backplane version with 2 expanders that allow for multipathing, cascading and failover (i.e. the one OP has).
But the single expander version does not have a single topology description in the manual (The connection diagrams you posted are for the E26 backplane). So from the manual it remains totally unclear if you can connect 2 ports of a single expander to

a) 2 ports of a single HBA or
b) 1 port of 2 HBAs​

All you can infer from the manual is that connecting 1 port of the HBA to 1 port of the expander will give you working connection that you can cascade.
But internet gossip seems to suggest that there are possible connections that the Supermicro manual does not mention.
Maybe we'd have more luck looking for the documentation of the LSI expander chips instead of the backplane itself...

 

[depasseg]

The connection diagrams you posted are for the E26 backplane)

True, but the diagram legend points out the ports that aren't available on the E16 variant.

Maybe we'd have more luck looking for the documentation of the LSI expander chips instead of the backplane itself...

This is probably helpful, but we would also need to understand what SM is doing in the backplane. It looks like @jgreco tried hooking up 2 ports last year with less than stellar results.
https://forums.freenas.org/index.ph...bas-to-backplane-questions.38238/#post-232345

 

[jgreco]

True, but the diagram legend points out the ports that aren't available on the E16 variant.

This is probably helpful, but we would also need to understand what SM is doing in the backplane. It looks like @jgreco tried hooking up 2 ports last year with less than stellar results.
https://forums.freenas.org/index.ph...bas-to-backplane-questions.38238/#post-232345

Actually that was years ago, I just commented on it last year. In theory you're supposed to be able to aggregate arbitrary numbers of SAS lanes, so going two 6Gbps/24Gbps x4's wide ought to give you 48Gbps, but I'm guessing there may be some lower level configuration thing you need to do.

Almost any SAS topology that does not introduce a loop is "valid", but whether it is actually useful or supported in software is another question. You may connect your controller to any of the ports, etc.

Hi Greg,
yes I'm referring to the E16 backplanes for the 24 bay chassis 846. The SAS1 and SAS2 variants have 3 ports and the new SAS3 backplanes have 4 ports. But I have read that those ports are not really dedicated for input or output. In fact they are bidirectional and can be used either for connecting to an HBA or to another expander. The manual just gives examples of a few topologies and only for the backplane version with 2 expanders that allow for multipathing, cascading and failover (i.e. the one OP has).
But the single expander version does not have a single topology description in the manual (The connection diagrams you posted are for the E26 backplane). So from the manual it remains totally unclear if you can connect 2 ports of a single expander to

a) 2 ports of a single HBA or
b) 1 port of 2 HBAs​

All you can infer from the manual is that connecting 1 port of the HBA to 1 port of the expander will give you working connection that you can cascade.
But internet gossip seems to suggest that there are possible connections that the Supermicro manual does not mention.
Maybe we'd have more luck looking for the documentation of the LSI expander chips instead of the backplane itself...

The documentation of the backplane says basically zero useful things, because it assumes you understand SAS. It gives some basic examples in case you don't.

The backplane with one expander and two expanders are exactly identical except for the lack of the second expander which attaches exclusively to the secondary SAS channel on SAS hard drives. Therefore the BE26/BE2C are basically pointless if you have SATA drives, though you could maybe use the secondary to provide additional external SFF8088's or something like that.

 

[bestboy]

Thanks guys for clearing things up. I really appreciate it.

I just wish we had a working example of SAS2 dual linking. I really feel like this might gain some weight in the near future with the transition to flash-based data disks.
At the moment cheap 1 TB SSDs are at about 230 EUR/USD and the first all-flash builds are popping up in the hardware section. With the prices plummeting SSD data disks will be a thing in a year from now.
And if you are in the market for a not-exactly-cheap Supermicro chassis today and want it to be at least a little future proof and not-stuck-on-spinning-rust, then the bandwidth topic is something to think about.
Oh and yes, I do understand that the array bandwidth will be masked by the network bandwidth. But we home users are usually not just having a dedicated file server for which it's usefulness is determined by how much other systems can get out of it. We home users typically use those boxes as home servers with Emby or Plex servers, backup applications, VMs, Owncloulds and whatever goodies the plugin stores and bhyves can come up with on top of the usual filer tasks. And with all that stuff running directly off the array, I think home users might actually be able to put the array bandwidth to good use...

So with the uncertainties around dual linked SAS2, I guess the safest way is to go with SAS3: A SAS3 backplane connected with a trusty SAS2 HBA for now to drive the spinning rust. And when it's time for data SSDs, switch it up to a SAS3 HBA for 48 Gb/s goodness. I'm sure at that point in time we are going to have battle-tested SAS3 drivers in FreeNAS to make the SAS3 HBA a safe bet, too.

PS: The LSI expander chip documentation did not loose a single word about possible SAS topologies. But I learned that there is a 200 resp. 300 MHz ARM CPU on the expander chips. I'm now counting the days until someone is porting Quake to run it on the backplane... :D

 

[jgreco]

Those aren't the first all-flash builds

But anyways, you're probably best advised not to get all worried about this. If you really feel you need the additional throughput to the pool, just move to 12Gbps SAS and be done with it. A 48Gbps interconnect is incredibly fast. The LSI 12Gbps controllers have been working without issue for a number of people, and even if they don't have a billion hours on them under FreeNAS, there's a point at which it pays to stop worrying and embrace the new technology. We're probably there now.

 

[bestboy]

Well, yes. That was not meant as an absolute statement. I just wanted to point out that there are already all-flash FreeNAS build reviews of early adopters in the hardware section of this forum.