SAS Multipath support on TrueNAS Scale 22.12.X

[morganL]

So I rebuilt one of the systems with the JBOD chassis cabled with only one SAS cable from a single HBA port to a single port on a single controller, as you say that would be the supported configuration. I completely blew away the old Truenas VM and wiped the the physical drives, redid the passthroughs, and everything was done from scratch. There are a combination of SAS and SATA drives in the array that was rebuilt.

After rebuild, when I went to create a new pool. In the pool creation wizard, the SATA drives showed up as regular devices (i.e. da24). The SAS devices showed up as multipath devices (i.e. multipath/disk5) though in the wizard. It was very odd because after initial Truenas install and before creating the first pool there was a not a multipath tab showing. When I created the first pool, then the Multipath tab started appearing under Storage as well. When I had installed Truenas the first time, with both JBOD controllers hooked up during install, the multipath tab was there from the beginning.

When I looked in the drives tab all the drives were displayed as individual devices regardless of if they were SATA or SAS. The only places I saw the references to multipath were when I went to create a pool and on the multipath tab (after pools are created). When cabled to only one JBOD controller with a single cable, I then started getting alerts like "Multipath multipath/disk2 connection is not optimal. Please check disk cables." for each SAS drive in the pool after building a pool using SAS drives. I could dismiss the alerts but that just grayed them out. They didn't actually go away. Each JBOD controller has 2 ports on it. So next, I tried cabling both ports on the HBA to the same JBOD controller (wide-porting as you referred to it) to see if that might help but it didn't resolve the multipath alerts. I then went and cabled the second HBA port to the second JBOD controller and all the alerts went away. It seems that even in its current version, Core sees dual-ported SAS drives as multipath regardless of if both controllers in the JBOD are cabled up or not and it expects both ports to be cabled to both controllers.

Given this behavior, are you sure about your statements in regards to how multipath behaves in Core currently? I'm not positive, but I think I did a test Scale install and with Scale and I didn't see anything showing up multipath in the interface regardless of how things were cabled and I didn't get any alerts about it in Scale it seems to corroborate what you have said, but not so with Core...

As indicated above.. Multipath is in CORE 13.0 and was not removed.
It is just no longer tested, supported or recommended.

It might be the system is remembering the previous setup with multipath... one of the challenges.

 

[Johnny Fartpants]

When using gmultipath label on-disk metadata will be stored so even after removing a path / cable the device will still remember it was a multipath member. gmultipath destroy devicename will remove that metadata. gmultipath list will show you all multipath devices. Reboot may be required after deleting the multipath devices before the UI looks right.

 

[pricemc1]

When using gmultipath label on-disk metadata will be stored so even after removing a path / cable the device will still remember it was a multipath member. gmultipath destroy devicename will remove that metadata. gmultipath list will show you all multipath devices. Reboot may be required after deleting the multipath devices before the UI looks right.

Johnny,
Thanks for that information and the commands. I used them to remove the multipathing metadata. Apparently when I "wiped" the disks it didnt really get the metadata so that seems to be why I was still seeing them even on the new builds. Everything seems to be running as defined by morganL now with both HBA ports connected to two ports on a single controller on the JBOD.

Just curious, do you know if there is a command that can be used to confirm that the HBA and JBOD controller are now operating in a wide-path configuration?

Regards,
pricemc1

 

[Johnny Fartpants]

Just curious, do you know if there is a command that can be used to confirm that the HBA and JBOD controller are now operating in a wide-path configuration?

I don’t. Not sure if you can actually see visibility of this.

 

[Johnny Fartpants]

Thought this may be helpful to add:

FNC is Changing to Wide Porting

Notice of change to wide porting for FreeNAS Certified systems.

www.truenas.com

 

[Johnny Fartpants]

To my knowledge when using one SAS cable the server will send DATA and SAS command sets via the same cable however when wide porting DATA and SAS commands are separated allowing the system to work a slightly faster.

 

[morganL]

To my knowledge when using one SAS cable the server will send DATA and SAS command sets via the same cable however when wide porting DATA and SAS commands are separated allowing the system to work a slightly faster.

With wideporting, data can go on either of the cables. It effectively makes a SAS connection change from 4 lanes to 8 lanes.

 

[Arwen]

Even a single cable with 4 lanes is a wide port. sort of. When a single SAS controller is wired up to a SAS Expander, with more than 1 SAS lane, the SAS controller always performs primitive load balancing across all the SAS lanes it can. Whether that is 4 lanes from a single cable, or 8 lanes from 2 cables.

Sometimes I wish the SAS standard did better load balancing when using multiple lanes. But, I also prefer reliability over complexity. (Well, I don't MIND complexity IF it is reliable .