How do I install and configure an LSI SAS 9300-4i4e HBA card in FreeNAS?

[cyberjock]

I would like to think that when we get to FreeNAS 10 things might be better for that card, but there's no promises of course. There's also no ETA on FreeNAS 10, but it will be here someday. I wouldn't bet money on 2015 though. But time will tell.

You do have me worried with your phrase 'I see that reboots of the NAS are going to be a big part of my life moving forward'. You really shouldn't be having to reboot except for disk replacements and updates to FreeNAS. Updates are optional and if you are losing disks *that* often you shouldn't be using them. I've only been "forced" to do reboots maybe 3 times in the last 2+ years that werent from updates.

 

[ashori]

Just a note for anyone reading this thread, seems the JBOD and LSI HBA do support hot swap after all. I may just be doing it all wrong or once in a pool the drives dont like to be offlined or to fail, in which case they then go on to require a reboot? More experimentation is required here but right now my brain is spinning trying to figure out freenas 9.3 and the correct way to replace a failed drive in a mirror.

What I did immediately after deleting my test mirror pool and rebooting the server is pull the two disks, label them and plug them back into the JBOD enclosure. I noted the following message in console and the disks appear ready to be imported.

Code:

Feb  4 20:48:01 Medusa da1 at mpr0 bus 0 scbus0 target 19 lun 0
Feb  4 20:48:01 Medusa da1: <ATA WDC WD10EZEX-00R 0A80> s/n      WD-WMC1S3315776 detached
Feb  4 20:48:01 Medusa (da1:mpr0:0:19:0): Periph destroyed
Feb  4 20:48:09 Medusa da0 at mpr0 bus 0 scbus0 target 18 lun 0
Feb  4 20:48:09 Medusa da0: <ATA WDC WD10EZEX-00R 0A80> s/n      WD-WMC1S3234537 detached
Feb  4 20:48:09 Medusa (da0:mpr0:0:18:0): Periph destroyed
Feb  4 20:54:30 Medusa ses0: pass0,da0: Element descriptor: 'Slot04'
Feb  4 20:54:30 Medusa ses0: pass0,da0: SAS Device Slot Element: 1 Phys at Slot 4
Feb  4 20:54:30 Medusa ses0:  phy 0: SATA device
Feb  4 20:54:30 Medusa ses0:  phy 0: parent 50030480017a383f addr 50030480017a3804
Feb  4 20:54:30 Medusa da0 at mpr0 bus 0 scbus0 target 12 lun 0
Feb  4 20:54:30 Medusa da0: <ATA WDC WD10EZEX-00R 0A80> Fixed Direct Access SCSI-6 device
Feb  4 20:54:30 Medusa da0: Serial Number      WD-WMC1S3315776
Feb  4 20:54:30 Medusa da0: 1200.000MB/s transfers
Feb  4 20:54:30 Medusa da0: Command Queueing enabled
Feb  4 20:54:30 Medusa da0: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
Feb  4 20:54:38 Medusa ses0: pass1,da1: Element descriptor: 'Slot05'
Feb  4 20:54:38 Medusa ses0: pass1,da1: SAS Device Slot Element: 1 Phys at Slot 5
Feb  4 20:54:38 Medusa ses0:  phy 0: SATA device
Feb  4 20:54:38 Medusa ses0:  phy 0: parent 50030480017a383f addr 50030480017a3805
Feb  4 20:54:38 Medusa da1 at mpr0 bus 0 scbus0 target 13 lun 0
Feb  4 20:54:38 Medusa da1: <ATA WDC WD10EZEX-00R 0A80> Fixed Direct Access SCSI-6 device
Feb  4 20:54:38 Medusa da1: Serial Number      WD-WMC1S3234537
Feb  4 20:54:38 Medusa da1: 1200.000MB/s transfers
Feb  4 20:54:38 Medusa da1: Command Queueing enabled
Feb  4 20:54:38 Medusa da1: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)

I would like to think that when we get to FreeNAS 10 things might be better for that card, but there's no promises of course. There's also no ETA on FreeNAS 10, but it will be here someday. I wouldn't bet money on 2015 though. But time will tell.

You do have me worried with your phrase 'I see that reboots of the NAS are going to be a big part of my life moving forward'. You really shouldn't be having to reboot except for disk replacements and updates to FreeNAS. Updates are optional and if you are losing disks *that* often you shouldn't be using them. I've only been "forced" to do reboots maybe 3 times in the last 2+ years that werent from updates.

 

[ashori]

Hi Cyberjock,

What do you think of my backup plans? Any recommendation on Raid60 vs Raid6 with replication? I know both sound very similar.

1. RAID6 5 x 2tb on my desktop data drive
2. Nightly backup to RAID 60 10 x 2tb on the JBOD -or- RAID 6 5 x 2tb with replication to a 2nd RAID6 5x 2tb in the same JBOD (what would you recommend?)
3. Crash Central backup

 

[mikesoultanian]

You do have me worried with your phrase 'I see that reboots of the NAS are going to be a big part of my life moving forward'. You really shouldn't be having to reboot except for disk replacements and updates to FreeNAS. Updates are optional and if you are losing disks *that* often you shouldn't be using them. I've only been "forced" to do reboots maybe 3 times in the last 2+ years that werent from updates.

This is a somewhat old thread, but I'm still curious about how to recover when losing an entire JBOD enclosure. I have 3 JBOD enclosures connected to one server - two of the JBODs have SSDs and host one pool, and the other JBOD hosts an HDD pool - the system dataset is hosted on the boot drives. Both pools are configured RAID10.

Let's say I accidentally pull a cable or kill power or the expander fails or any number of real-world possibilities and I lose the HDD JBOD that hosts an entire pool - is there any way to bring it back online without rebooting the server? I have production storage on the other two JBODs so I'd rather not have to reboot and bring everything down.

Then let's say I lose the JBOD that hosts half of the SSD pool - can I resurrect that pool without a reboot?

I think being able to bring the system back up without a reboot is a realistic expectation for a production SAN. Also, the notion that there's a "right" way to induce a failure by means of just using a bad drive is kinda like putting your head in a hole and pretending the rest of the nasty things that do happen won't happen - I think a better way of wording it (as you alluded to with your mention of having a quality backup) is doing everything you can to be prepared for a crash, and also having a quality storage system that is able to gracefully (and as easily a possible) get back up and running as quickly as possible with the least amount of headaches. Having to reboot is a significant headache, and it isn't graceful.

In regards to this:

Well, a JBOD dying (power or data cable yanked) is a real possibility. Sounds like we need a warning somewhere that the .system dataset should stay as close to the server as possible (meaning inside it).

I don't really think that is necessary. If you've lost enough drives that you have an unmounted pool, who cares if the server is even on? It can't serve data since it has no pool. It's like being upset because the power is off on the server. Can't argue that a powered-off server isn't providing data fast enough, can you? ;)

I also don't think it's a fair assumption to say that most people only run one pool on one FreeNAS instance, therefore losing your JBOD and necessitating a reboot isn't that big of a deal - I would venture to guess that many people run many pools on multiple JBODs for very legitimate reasons.

So, what I'd like to know is if there's a graceful way to return to an operating state after a JBOD crashes (assuming the system dataset is stored elsewhere) without necessitating a reboot.

Thanks!

 

[Ericloewe]

is there any way to bring it back online without rebooting the server?

In principle, it's just a matter of reconnecting the drives and importing the pool. However, you should test this scenario yourself, because it's undefined behavior (or less-than-documented, if you will).

Then let's say I lose the JBOD that hosts half of the SSD pool

If the pool became UNAVAIL, same thing. If it was just DEGRADED, just plug everything back in and ZFS will resilver whatever was missed.

I also don't think it's a fair assumption to say that most people only run one pool on one FreeNAS instance, therefore losing your JBOD and necessitating a reboot isn't that big of a deal - I would venture to guess that many people run many pools on multiple JBODs for very legitimate reasons.

Most people have one data pool. A significant number of people do have multiple data pools (I hesitate to say "many pools").


That said, I must point out that most scenarios that result in failure of the JBOD either destroy the hard drives in the process or require significant work to restore from, making a reboot negligible in the sea of downtime. The rest is the "oops, I unplugged the cable" scenario, which is rather easy to avoid.

 

[mikesoultanian]

In principle, it's just a matter of reconnecting the drives and importing the pool. However, you should test this scenario yourself, because it's undefined behavior (or less-than-documented, if you will).

If the pool became UNAVAIL, same thing. If it was just DEGRADED, just plug everything back in and ZFS will resilver whatever was missed.

It seems kinda odd to me that a disconnected drive disaster recovery would be considered undefined behavior for a storage appliance - that's the whole point of having a software RAID so that you're able to gracefully recover from a disaster such a something as simple as a cable failure.

That being said, I will definitely test this - from previous replies it sounded like it would require a reboot, which apparently it does not. I would think if my HBA supports hot-swap and it's able to detect my JBOD when it's unplugged and plugged back in (which it does), then FreeNAS *should* be able to as well. I do think it's a realistic expectation that there would be data corruption, but that's outside of FreeNAS' control and I respect that.

That said, I must point out that most scenarios that result in failure of the JBOD either destroy the hard drives in the process or require significant work to restore from, making a reboot negligible in the sea of downtime.

And with that I'm guessing that our needs fall outside of the abilities of FreeNAS. This has all been extremely helpful as I'll still need to keep this system running for the next year or so, but ultimately I'm finding that FreeNAS can't (and really isn't meant to) provide the continuous up-time that we require - it's looking like a clustered storage is what we need.

 

[Ericloewe]

Well, if you need High Availability, iX sells TrueNAS, whose main feature (besides the support) is High Availability with all the bells and whistles.

Pricing is said to be very competitive with other storage vendors.

 

[mikesoultanian]

Well, if you need High Availability, iX sells TrueNAS, whose main feature (besides the support) is High Availability with all the bells and whistles.

Pricing is said to be very competitive with other storage vendors.

I'm assuming their TrueNAS solutions are still the same overall FreeNAS system, just with customer support. If so, I'd imagine I'd have all the same issues that have been discussed here - I'm not knocking FreeNAS as it still has a lot of great features, but correct me if I'm wrong, it doesn't support clustering. If I am forced to reboot FreeNAS for some reason or another, that means I need to bring all my VMs offline. For us that is not a preferred scenario.

 

[SweetAndLow]

I'm assuming their TrueNAS solutions are still the same overall FreeNAS system, just with customer support. If so, I'd imagine I'd have all the same issues that have been discussed here - I'm not knocking FreeNAS as it still has a lot of great features, but correct me if I'm wrong, it doesn't support clustering. If I am forced to reboot FreeNAS for some reason or another, that means I need to bring all my VMs offline. For us that is not a preferred scenario.

They are not the same. TrueNAS had high availability, specific hardware and other features that are not included in FreeNAS.

Sent from my Nexus 5X using Tapatalk

 

[mikesoultanian]

I reread your message and notice that you mentioned HA - I then checked their links and realized that they do have a HA solution, so yes, it does differ from the regular FreeNAS offering. Thanks for the info.

 

[mikesoultanian]

They are not the same. Truenas had high availability, specific hardware and other features that are not included in freenas.

Yeah, just noticed that - looks like we posted right at the same time ;) thanks!

 

[cyberjock]

It seems kinda odd to me that a disconnected drive disaster recovery would be considered undefined behavior for a storage appliance - that's the whole point of having a software RAID so that you're able to gracefully recover from a disaster such a something as simple as a cable failure.

That being said, I will definitely test this - from previous replies it sounded like it would require a reboot, which apparently it does not. I would think if my HBA supports hot-swap and it's able to detect my JBOD when it's unplugged and plugged back in (which it does), then FreeNAS *should* be able to as well. I do think it's a realistic expectation that there would be data corruption, but that's outside of FreeNAS' control and I respect that.



And with that I'm guessing that our needs fall outside of the abilities of FreeNAS. This has all been extremely helpful as I'll still need to keep this system running for the next year or so, but ultimately I'm finding that FreeNAS can't (and really isn't meant to) provide the continuous up-time that we require - it's looking like a clustered storage is what we need.

I work on TrueNAS HA daily (got my own test system too) and I can tell you that a SAS cable failure *will* trigger a failover, so uptime is maintained.

For FreeNAS, its harder to determine how to recover from a disconnected cable. If you are looking for reliability from failed cabling, there is a solution... multipathing. SAS multipathing was designed specifically to address issues like this. If you don't have or don't support that functionality, you might be able to online the disks with zpool online commands. I've never tried to do that before, so I'm not sure how well it would work.

As for power supply getting unplugged, you should have dual redundant power supplies to avoid that kind of issue as well.

I will agree that probably 90% of FreeNAS users have a single large pool, but 10% is still a lot of users (by numbers) with more than 1 zpool.

If you are in a RAID10 environment, I'd simply do stripes of disks that are in different jbods. You lose one jbod and you only lose 1 of every 2 disks in all of the vdevs. I've had customers deliberately do this.

Also, I'm not sure what controller you are using right now, but all of the 6Gb and 12Gb SAS cards I've worked with seem to handle disconnecting and reconnecting of jbods, disks, etc with hotswap and hotplugging working just fine. I've only seen the 12Gb SAS on more recent versions of FreeNAS though, so if you're using a version from 2016 you will be outside of my experience base.

 

[mikesoultanian]

I work on TrueNAS HA daily (got my own test system too) and I can tell you that a SAS cable failure *will* trigger a failover, so uptime is maintained.

For FreeNAS, its harder to determine how to recover from a disconnected cable. If you are looking for reliability from failed cabling, there is a solution... multipathing. SAS multipathing was designed specifically to address issues like this. If you don't have or don't support that functionality, you might be able to online the disks with zpool online commands. I've never tried to do that before, so I'm not sure how well it would work.

Plus if I understand SAS correctly, we have two SAS cables that each handle 4 lanes, so losing one cable would still allow for reduced functionality, right? I am just guessing on that one, though.

Do the TrueNAS boxes support multipathing? Honestly, I don't mind going with a paid solution, especially for support. What I don't want to be doing on my own is "try this command", "maybe this command will work". I'm managing too many different systems and being an expert at FreeNAS has been challenging - I either want documentation one how to recover from disaster situations or be able to call someone that can fix it for me :)

As for power supply getting unplugged, you should have dual redundant power supplies to avoid that kind of issue as well.

Yes, we do have that. I was merely just trying to highlight that stuff can happen, regardless of how much hardware redundancy we have in place. My worry is whether the software is able to gracefully recover (with or without intervention).

I will agree that probably 90% of FreeNAS users have a single large pool, but 10% is still a lot of users (by numbers) with more than 1 zpool.

If you are in a RAID10 environment, I'd simply do stripes of disks that are in different jbods. You lose one jbod and you only lose 1 of every 2 disks in all of the vdevs. I've had customers deliberately do this.

Which is very possible, but would be a mild hassle to set up and maintain long-term, especially making sure other admins pay careful attention that drives stay on the correct side of the mirror.

Also, I'm not sure what controller you are using right now, but all of the 6Gb and 12Gb SAS cards I've worked with seem to handle disconnecting and reconnecting of jbods, disks, etc with hotswap and hotplugging working just fine. I've only seen the 12Gb SAS on more recent versions of FreeNAS though, so if you're using a version from 2016 you will be outside of my experience base.

Yeah, I've been plugging/unplugging cables and the SAS controller and drive enclosures seem to do fine - ultimately my concern was how gracefully FreeNAS would handle these situations when an enclosure goes offline for whatever reason.

I will be talking to different vendors in the coming week and I will include TrueNAS as one of those vendors. Thanks for your help!

Thanks!
Mike