SAS Multipathing or not ?

[VioletDragon]

Hi folks,

I’m upgrading my storage and recently bought a JBOD chassis to expand my Virtualisation Storage Data Store, I’ve plugged the HBA into both controllers on the disk shelve, however what is the best way to enable Multipathing ? Or is it better to just use one controller on the JBOD disk shelf? In my eyes it’s better to have both right ? Incase of failure with cable, controller or port on the HBA.

Regards

Jack

 

[LarsR]

See: https://www.truenas.com/community/t...t-on-truenas-scale-22-12-x.110847/post-766745

Multipathing is no longer supported in both core and scale.

 

[VioletDragon]

See: https://www.truenas.com/community/t...t-on-truenas-scale-22-12-x.110847/post-766745

Multipathing is no longer supported in both core and scale.

That blows. No redundancy then, bit disappointing.

I obviously need to find another solution then, I was thinking at looking at clustered storage as I lost the machine due to a Kernel panic.

I guess TrueNAS doesn’t have what I need anymore.

 

[LarsR]

If you only want to hook up both cables it should still be supportet, as said later in the other discussion thread

morganL said: SAS Wideporting - multiple cables from same HBA to same controller on JBOD - is supported. The drives do not appear as multiple drives in the systems. If one cable is unplugged the system keeps on operating.

 

[VioletDragon]

If you only want to hook up both cables it should still be supportet, as said later in the other discussion thread

Yes drives show up with both paths but it does not work properly,

If you unplug one of the SAS cables the array goes offline, so in event of a failure you are not protected. That is what Im trying to achieve if that is the case redundant Controllers has no support at all which I think is a bit strange because if you loose a Controller or bad cable on either JBOD you loose your array and depending how you have this configured it will destroy all your DATA.

Without Multipathing fully implemented you loose a controller it is game over

 

[Patrick M. Hausen]

You could buy a TrueNAS Enterprise system ...

 

[HoneyBadger]

Yes drives show up with both paths but it does not work properly,

I think the suggestion from @LarsR is to hook up both SAS cables to the same controller (if it has dual SAS input capabilities) - this is the wideporting option, and would survive a cable failure.

I was thinking at looking at clustered storage as I lost the machine due to a Kernel panic.

A panic on the main TrueNAS system wouldn't be prevented by multipath to a JBOD though; you'd need a shared-storage backplane with HA failover, which is available in TrueNAS Enterprise.

 

[VioletDragon]

I think the suggestion from @LarsR is to hook up both SAS cables to the same controller (if it has dual SAS input capabilities) - this is the wideporting option, and would survive a cable failure.


A panic on the main TrueNAS system wouldn't be prevented by multipath to a JBOD though; you'd need a shared-storage backplane with HA failover, which is available in TrueNAS Enterprise.

Wide porting does not work when a cable is just disconnected, all drives have two Paths, but when disconnecting the SAS cable from Controller 1 on the JBOD it causes TrueNAS to become unresponsive meaning have to force a reboot. Disconnecting the cable from Controller 2 makes no difference and system still works as normal.

This is on a test system currently as I don’t want to put something into production that isn’t going to work properly, 100+ VMs.

One thing I’ve noticed, after a fresh install you insert two SAS disks it automatically add them as a Multipath even though the drives are wiped.

When drives are automatically added to a multipath I am not able to wipe the disks or run badblocks I have to manually use FreeBSD Multipath tool to delete the Multipath. Now not sure if this is an issue with TrueNAS 13 or not. I will test in an older version before Multipathing has been deprecated.

How the JBOD and HBA is connected,

Port 1 on H200E connected to Port on Controller 1

Port 2 on H200E connected to Port on Controller 2. I have changed these around Port 1 on HBA connected to Controller 2 and Port 2 on HBA connected to Controller 1 I.e in Crisscross makes no difference at all.

All drives are SAS HGST 2TB Enterprise Drive will be upgraded to 24 8TB once I’ve migrated over.

Regards

Jack

 

[HoneyBadger]

How the JBOD and HBA is connected,

Port 1 on H200E connected to Port on Controller 1

Port 2 on H200E connected to Port on Controller 2. I have changed these around Port 1 on HBA connected to Controller 2 and Port 2 on HBA connected to Controller 1 I.e in Crisscross makes no difference at all.

This is still a "multipath" topology. If possible, connect both ports on the H200E to two ports on Controller 1 (or a single port if you plan to daisy-chain) and your drives will be recognized without being placed into multipath devices.

 

[VioletDragon]

This is still a "multipath" topology. If possible, connect both ports on the H200E to two ports on Controller 1 (or a single port if you plan to daisy-chain) and your drives will be recognized without being placed into multipath devices.

No daisychaining, will add another H200E which is better to do I think. Why when you unplug one of the cables the drives are no longer seen ?

 

[VioletDragon]

Hi folks just a quick update.

I have been reverse engineering how gmultipath works, For Multipathing to work in TrueNAS-13.0-U5.3. You can create them in CLI, so after creating Paths for the disks it now works, pulling the cable for Controller 1 now has a Path to controller two and Pulling the cable from Controller 2 now has a path for Controller 1.

Now just to note, having Active/Active instead of Active/Passive is okay but the system does not detect that the Controller or cable has failed so make sure you use Active/Passive.

More information can be found on the BSDs manpage site. Link https://man.freebsd.org/cgi/man.cgi?query=gmultipath&sektion=8&format=html

Some people do this by Mirrors but i decided to this by Disk as in Da0 / Da2 instead as i feel doing it by the disk and not mirrors in the Zpool is better.

But please beware, If this helps people wanting Multipathing I am not held for any Data Loss.

 

[Johnny Fartpants]

Having used Multipath in FreeNAS and TrueNAS for the last 8 years and deciding to stop using it early this year may I suggest you go through the identification and replacement procedure first before you decide to deploy this in production. Functionality has changed in TN regarding multipath over the last few years and this has made it harder to maintain.

 

[VioletDragon]

Having used Multipath in FreeNAS and TrueNAS for the last 8 years and deciding to stop using it early this year may I suggest you go through the identification and replacement procedure first before you decide to deploy this in production. Functionality has changed in TN regarding multipath over the last few years and this has made it harder to maintain.

That is what i am testing on a test machine, I want to see if it's something I can manage.

Now question for you, now that Multipathing has been preferred to use SAS Wide Port, How exactly do you implement that?

 

[Johnny Fartpants]

Now question for you, now that Multipathing has been preferred to use SAS Wide Port, How exactly do you implement that?

Two SAS cables from one controller to one HBA.

 

[VioletDragon]

Two SAS cables from one controller to one HBA.

That is how i tried SAS Wide Port however when pulling a cable out of Controller all disks in the SAS Enclosure go offline.

 

[Johnny Fartpants]

Interesting. Can we see a pic and/or details of your JBOD?

 

[VioletDragon]

Ieresting. Can we see a pic and/or details of your JBOD?

Yeah sure, Note I am using a old motherboard to test with, I have tested with other boards to.

Dell SC200 which is re-branded MD1200
Dell Perc H200E Flash to IT mode with 9200E Firmware.

 

[Johnny Fartpants]

Doesn't appear to be a supported config according to Dell https://dl.dell.com/content/manual4...x-enclosure-support-matrix.pdf?language=en-us

 

[VioletDragon]

Doesn't appear to be a supported config according to Dell https://dl.dell.com/content/manual4...x-enclosure-support-matrix.pdf?language=en-us

Ok if that is the case, I shall stick with Multipathing as I don't want to run this with just one Controller.

 

[Johnny Fartpants]

In my experience controllers and HBAs rarely fail and even if they do you are unlikely to permanently lose your whole pool as a result. I would suggest you are more likely to come to harm by trying to make multipath work on an appliance that the creators themselves have given up on.