FreeNAS 11.2Beta2 Crash and now Locks up on Pool Import

[jgreco]

I am seriously considering this after seeing the response back from development team.

Wow. Sounds like they are not going to do anything that might fix this in the future.

What exactly would you like them to do? The problem is that cheap PC hardware can be problematic. The SATA port multipliers aren't particularly good. SAS1 expanders have issues too. The problem is that bad hardware often cannot be remediated in software. It is unfair to criticize the developers for that.

We've always tried hard to publish a list of the hardware that's actually known to work well, and the list is not particularly long. It's crappy that this is the way PC hardware is, but that's what it is.

 

[Chris Moore]

What exactly would you like them to do? The problem is that cheap PC hardware can be problematic. The SATA port multipliers aren't particularly good. SAS1 expanders have issues too. The problem is that bad hardware often cannot be remediated in software. It is unfair to criticize the developers for that.

We've always tried hard to publish a list of the hardware that's actually known to work well, and the list is not particularly long. It's crappy that this is the way PC hardware is, but that's what it is.

It is like another ticket that was immediately closed because the pool was more than 80% full when how full the pool is should have nothing to do with the system locking up when the user tried to delete files.
Even if it can't be solved easily, there should be a way for the system to gracefully handle the situation instead of crashing.


Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[jgreco]

It is like another ticket that was immediately closed because the pool was more than 80% full when how full the pool is should have nothing to do with the system locking up when the user tried to delete files.
Even if it can't be solved easily, there should be a way for the system to gracefully handle the situation instead of crashing.

I'm not interested in digging through all the tickets to identify which "another ticket" you are talking about. I'm discussing *this* ticket, which was closed with the following reasoning:

I don't think we can really do much with this. Port multipliers generally don't provide much chances for error recovery, since disks are too much dependent on each other. Even worse, your ASMedia SATA controllers don't even support FIS-based switching, that allow only one disk work at a time, and to timeout on one of disks blocks all others on that port multipliers too. If that is physically possible I woudl recommend you to move as much disks as possible from port-multipliers on onboard Intel SATA ports. Honestly I am surprised that the failed disk appeared the only problem of yours, since very few people on my experience have that configuration working reliably.

Because it appears to me that the ticket was closed due to it essentially being a crappy PC port multiplier problem, which is a conclusion I have a hard time seeing as unreasonable. We spent a lot of time in the early years on this forum with users who had undesirable hardware configurations, and port multipliers are right up there with Realtek ethernets and 4GB RAM systems and external USB drives as a quick path to rapid self-inflicted hair loss.

I am not aware of "a way for the system to gracefully handle the situation" which is why I called you out on what you said.

 

[Jerzy Sobski]

First part of the solution is the SAS controller with external ports. I linked to one earlier:
https://www.ebay.com/itm/LSI-Logic-...ort-6Gb-s-SAS-HBA-Cable-Included/382527261521
That should be fine.
From that, you will need external SAS cables, one for each drive tower. The card I linked to comes with one cable, but here is the kind of cable you need for the second:
https://www.ebay.com/itm/SFF-8088-M...1200-1220-MD3200-3220-MD3600i-2m/263857193638
Several companies make them and they come in different lengths, shorter the better, that one is two meters, and here is one that is one meter:
https://www.ebay.com/itm/Hitachi-Ex...O-SFF-8088-Cable-SC1-K1BS-3ft-1m/392009494826
The kind of connector is standard, but the retention clip release mechanism varies slightly from one manufacturer to another.
Then, to connect that cable into the drive tower, you will need an external SFF-8088 to internal SFF-8087 adapter like this:
https://www.ebay.com/itm/SilverSton...-Mini-SFF8087-to-SFF8088-adapter/253689280729
This one particular one: View attachment 25097
Is nice because it can be separated into two parts and you can use one for each of the drive towers. The hard part here is that you need to remove the SATA port multiplier hardware and find a way to mount this instead.
From there, and you will need to do this once for each drive tower, use a SFF-8087 cable, like this:
https://www.ebay.com/itm/0-7M-Internal-Mini-SAS-36-Pin-To-SFF-8087-Cable-Black-E2C2/253321506804
Again, many companies make them, the brand is not important, so if you can find a short cable that you can get faster, it is the kind of connector that matters.
Connect from the inside of that external/internal adapter to this SAS expander:
https://www.ebay.com/itm/Intel-RES2...der-Card-SAS-SATA-PCI-Express-x4/141788425753
The kind of expander matters here, because this one has a four pin Molex power input. I am hopeful that those drive towers have an extra Molex power cord that you can use to power this. View attachment 25098 Just put some electrical tape over the card edge, circled in red, because it isn't needed for what we are doing but we don't want it to short out on anything. It would be nice if there was a place to mount this, but if there isn't it just needs to not be grounded out on anything.
The 'input' cable coming from the server would go into the port closest to the PCI bracket, which you can remove if needed. The 'output' to the drives will need three of these cables:
https://www.ebay.com/itm/Lot-of-2-M...-Forward-Breakout-Internal-Cable/371681252206
They can be connected to the three ports on the opposite end of the card and will connect to the back of the drive cages in the tower where the SATA cables were connected. I made a kind of price list for the cost of all this:

Code:

LSI Logic SAS9200-16e External Dual Port 6Gb/s SAS HBA {Cable Included}  =  $ 59.00

Hitachi External Mini SAS 26P SFF-8088 TO SFF-8088 Cable SC1/K1BS 3ft/1m =  $  7.00

SilverStone Technology SA022 Dual Mini SFF8087 to SFF8088 adapter		=  $ 43.99

Internal Mini SAS 36-Pin To SFF-8087 Cable, 0.7M		  - $4.78  x  2  =  $  9.56

Intel RES2SV240 24 Port Expander Card					- $59.95  x  2  =  $119.90

Lot of 2 Mini SAS to 4-SATA Forward Breakout			 -  $12.99  x  3 =  $ 38.97


All totaled:  $  278.42

I am sure there are other ways to do it, but this was the most cost effective way I could figure to upgrade both drive towers to SAS. About $140 each.

I ended up purchasing all the items you had listed above and it should be arriving sometime sometime this week.

After I had placed the order I spent the weekend reviewing and looking at the possibility of moving all drives into a single unit and ended up deciding on purchasing the Norco RPC-4224 4u, along with better fans, MB: SuperMicro MBD-X9SRL-F-O, CPU: SuperMicro MBD-X9SRL-F-O, 6 Internal Mini SAS(SFF-8087) 36Pin Right Angle Male to Internal Mini SAS (SFF-8087) 36Pin Male Cable, 0.75 Meter.

For the Raid Expander I got the Intel RES2SV240
https://www.ebay.com/itm/Intel-RES2...=STRK:MEBIDX:IT&_trksid=p2057872.m2749.l26494

My question is: Which HBA would I need to make this work with Freenas and have access to all 24 drives? I also read in several posts that firmware would have to be reflashed onto the HBAs not knowing which economical HBA's are capable of this and are compatible with Freenas, I thought Id ask you on this.

In regards to the Intel Res2V240, can all 6 ports be used to connect the 24 drives since this could essentially be plugged into Motherboard, or would I have to reserves the one port to connect it to the HBA?

If that is that case, it be best to use a reverse SAS Cable to go from backplane of the 4 drives directly to the Motherboard sata ports?

Thanks

 

[Johnnie Black]

In regards to the Intel Res2V240, can all 6 ports be used to connect the 24 drives since this could essentially be plugged into Motherboard, or would I have to reserves the one port to connect it to the HBA?

At least one ports goes to the HBA, if you buy the commonly used LSI 9211 or 9207-8i you can use the other miniSAS port for the remaining 4 drives.

 

[Ericloewe]

could essentially be plugged into Motherboard

The PCIe edge connector is only for power, as an alternative to the Molex connector.

 

[Jerzy Sobski]

At least one ports goes to the HBA, if you buy the commonly used LSI 9211 or 9207-8i you can use the other miniSAS port for the remaining 4 drives.

Do either of these cards requiring to be reflashed with other firmware?


I found the following on Ebay. Would this one work:
https://www.ebay.com/itm/LSI-SAS-92...0&_nkw=9211-8i&_from=R40&rt=nc&LH_TitleDesc=0

 

[Johnnie Black]

Would this one work:

It's already in IT mode, so worst case it could need a firmware update, but that's very easy, easier than a bios update, you can even do it inside FreeNAS

 

[Chris Moore]

After I had placed the order I spent the weekend reviewing and looking at the possibility of moving all drives into a single unit and ended up deciding on purchasing the Norco RPC-4224 4u, along with better fans, MB: SuperMicro MBD-X9SRL-F-O, CPU: SuperMicro MBD-X9SRL-F-O, 6 Internal Mini SAS(SFF-8087) 36Pin Right Angle Male to Internal Mini SAS (SFF-8087) 36Pin Male Cable, 0.75 Meter.

Did you order these items already?
I sure hope not. That price on the system board is way more than what you should pay. I picked up two of them still in the factory packaging earlier this year for only $198 and free shipping. The ones I got were being disposed in an inventory clearance and were perfect with the factory seals still in place. If you shop eBay and buy a used one, you should get a deal like this:

You just have to be on the lookout for them when they come up. Then there is this one:

These already sold, however you can get a deal similar to this if you are patient.

 

[Jerzy Sobski]

Did you order these items already?
I sure hope not. That price on the system board is way more than what you should pay. I picked up two of them still in the factory packaging earlier this year for only $198 and free shipping. The ones I got were being disposed in an inventory clearance and were perfect with the factory seals still in place. If you shop eBay and buy a used one, you should get a deal like this:
View attachment 25167
You just have to be on the lookout for them when they come up. Then there is this one:
View attachment 25168
These already sold, however you can get a deal similar to this if you are patient.

I ordered it on Sunday. It seemed to be the about the lowest price I could find for sale at the time on Ebay and Amazon. Not being familiar with what this boards going rate on this I thought it was a reasonable price. When I went back to check on status of this order it had a tracking #.

Next time Ill have to look at history of what the items sold before pulling the trigger. Since the server has been down since last Monday, Im trying to rush and get it back up hopefully within the next week.

 

[heatvent]

I am currently running Freenas 11.2 Beta 2. Overnight Freenas reboot possibly from a crash and it attempted to reboot and got hung up during a Pool import. I have tried several times to turn of Server and reboot it. It gets to same point before it locks up (At least it appears to be locked up). Below is a screean capture of where it locks up.

Is there any work around to find out what is causing this and have it boot up completely.

View attachment 25072

I had a similar experience. I ended up creating a new 11.2 installion USB flash drive and then upgrading the existing Freenas 11.2 USB install. Specifically I also chose the option to reform the drive and keep the settings. Now boots up OK. Not sure why this worked but I am back in business.

 

[Jerzy Sobski]

I had a similar experience. I ended up creating a new 11.2 installion USB flash drive and then upgrading the existing Freenas 11.2 USB install. Specifically I also chose the option to reform the drive and keep the settings. Now boots up OK. Not sure why this worked but I am back in business.

Mine actually turned out to be a HD that went bad on a 15 drive pool. Since all my drives were in one pool handled by a port multiplexer, it would get entire system hung up due to the hard drive issue. Since the time this has occurred I have learned the Port Multiplexer was a bad idea and it was trouble waiting to happen.

Currently in process building a new system that will not require the port multiplexer to handle all the drives.

 

[jgreco]

Mine actually turned out to be a HD that went bad on a 15 drive pool. Since all my drives were in one pool handled by a port multiplexer, it would get entire system hung up due to the hard drive issue. Since the time this has occurred I have learned the Port Multiplexer was a bad idea and it was trouble waiting to happen.

More specifically, it isn't the entire system, but the other drives on that particular port that get screwed up while things are in progress on the failing drive and it is holding the channel. Very sorry you had to discover this the hard way, as it is *miserable*, especially because (as you say) it is trouble waiting to happen, and it does exactly that... it lurks and then stabs you after you think everything's great. Good luck on your new build. ;-)

 

[Jerzy Sobski]

First part of the solution is the SAS controller with external ports. I linked to one earlier:
https://www.ebay.com/itm/LSI-Logic-...ort-6Gb-s-SAS-HBA-Cable-Included/382527261521
That should be fine.
From that, you will need external SAS cables, one for each drive tower. The card I linked to comes with one cable, but here is the kind of cable you need for the second:
https://www.ebay.com/itm/SFF-8088-M...1200-1220-MD3200-3220-MD3600i-2m/263857193638
Several companies make them and they come in different lengths, shorter the better, that one is two meters, and here is one that is one meter:
https://www.ebay.com/itm/Hitachi-Ex...O-SFF-8088-Cable-SC1-K1BS-3ft-1m/392009494826
The kind of connector is standard, but the retention clip release mechanism varies slightly from one manufacturer to another.
Then, to connect that cable into the drive tower, you will need an external SFF-8088 to internal SFF-8087 adapter like this:
https://www.ebay.com/itm/SilverSton...-Mini-SFF8087-to-SFF8088-adapter/253689280729
This one particular one: View attachment 25097
Is nice because it can be separated into two parts and you can use one for each of the drive towers. The hard part here is that you need to remove the SATA port multiplier hardware and find a way to mount this instead.
From there, and you will need to do this once for each drive tower, use a SFF-8087 cable, like this:
https://www.ebay.com/itm/0-7M-Internal-Mini-SAS-36-Pin-To-SFF-8087-Cable-Black-E2C2/253321506804
Again, many companies make them, the brand is not important, so if you can find a short cable that you can get faster, it is the kind of connector that matters.
Connect from the inside of that external/internal adapter to this SAS expander:
https://www.ebay.com/itm/Intel-RES2...der-Card-SAS-SATA-PCI-Express-x4/141788425753
The kind of expander matters here, because this one has a four pin Molex power input. I am hopeful that those drive towers have an extra Molex power cord that you can use to power this. View attachment 25098 Just put some electrical tape over the card edge, circled in red, because it isn't needed for what we are doing but we don't want it to short out on anything. It would be nice if there was a place to mount this, but if there isn't it just needs to not be grounded out on anything.
The 'input' cable coming from the server would go into the port closest to the PCI bracket, which you can remove if needed. The 'output' to the drives will need three of these cables:
https://www.ebay.com/itm/Lot-of-2-M...-Forward-Breakout-Internal-Cable/371681252206
They can be connected to the three ports on the opposite end of the card and will connect to the back of the drive cages in the tower where the SATA cables were connected. I made a kind of price list for the cost of all this:

Code:

LSI Logic SAS9200-16e External Dual Port 6Gb/s SAS HBA {Cable Included}  =  $ 59.00

Hitachi External Mini SAS 26P SFF-8088 TO SFF-8088 Cable SC1/K1BS 3ft/1m =  $  7.00

SilverStone Technology SA022 Dual Mini SFF8087 to SFF8088 adapter		=  $ 43.99

Internal Mini SAS 36-Pin To SFF-8087 Cable, 0.7M		  - $4.78  x  2  =  $  9.56

Intel RES2SV240 24 Port Expander Card					- $59.95  x  2  =  $119.90

Lot of 2 Mini SAS to 4-SATA Forward Breakout			 -  $12.99  x  3 =  $ 38.97


All totaled:  $  278.42

I am sure there are other ways to do it, but this was the most cost effective way I could figure to upgrade both drive towers to SAS. About $140 each.

Ok. Ive gone through and did this setup but for some reason only 4 of the 15 drives are being recognized. Something is not working right and not sure what could be causing it. On boot up I go directly to the configuration menu for the card and select the 9200. It seems to hang for what feels like a long time and then moves into the submenu where I can select the expander. There should be 2 expanders but only one shows up and in that list only 4 of the drives are showing up. Not sure how to resolve this.

It seems like the drives in the SuperMicro are working fine. Need the drives in in the NORCO when I move the entire pool into a Raidz2.

I have also gone through all drives to check health of them and they are not the problem.

Below is the system Build:

HARDWARE SPECIFICATIONS & Storrage Rack 1
CPU: Intel Xeon Processor E5-2650 v2
MB: SuperMicro X9SRL-F FW:
Ram: (96 GB) 6 x 16 GB DDR3 PC3-12800R 1600 MHz ECC Registered
Chasis: Norco RPC 4224
PSU: CORSAIR RMx Series, RM1000x
PCIE-01: LSI SAS 9211-8i 8-port
PCIE-02: Intel RES2SV240 24 Port RAID Expander Card
PCIE-05: LSI Logic SAS9200-16e (Used to connect External 10 Bay Towers)
PCIE-07: HP NC364T PCI-E Quad Port Gigabit Server Adapter
Boot: USB 16GB
Pool 1: Raidz1 ( 3 Vdevs x 5 Drives (4TB EA))
Pool 2: Raidz2 ( 2 Vdevs x 8 Drives (4TB EA))
Cache: Intel SSD DC S3710 Series 220GB


Chasis Cooling:
- Center Chasis: 3 x Noctua NF-F12 iPPC
- Back Chasis: 2 x Noctua NF-A8 PWM
- CPU Cooler: Noctua i4 CPU Cooler

Storage RaCK 2
10-bay 3.5" SATA 6.0 Gb/s eSATA-Trayless Enclosure
Intel RES2SV240 24 Port RAID Expander Card

Storage RaCK 3
10-bay 3.5" SATA 6.0 Gb/s eSATA-Tray Enclosure
Intel RES2SV240 24 Port RAID Expander Card

 

[Jerzy Sobski]

Looks like the issue is with only one of the 10 bay towers. When I turn off the power to the one tower, the other tower works fine and picks up all the drives. Im now thinking it can be a problem with the expander or the backplane of the 10 bay that has been shut of for now. Its a much older model so good possibility it may be causing the issue.

 

[Chris Moore]

Looks like the issue is with only one of the 10 bay towers. When I turn off the power to the one tower, the other tower works fine and picks up all the drives. Im now thinking it can be a problem with the expander or the backplane of the 10 bay that has been shut of for now. Its a much older model so good possibility it may be causing the issue.

That is a good possibility. What kind is it?

 

[Jerzy Sobski]

That is a good possibility. What kind is it?

It was bought in 2009. Not sure the brand but was able to find some identifying # on the backplane. SS-305; I will be doing a test tonight to confirm by bypassing the backplane and connecting directly to the drives. I just have to pick some adapters from Molex to SATA power. Case used the Molex which was also used by the backplanes. Since Im going to try and go to the drives directly will need to get these adapters in order to power the drives.

Im now starting to think that this tower was what was causing all the issues to start with. Its the same tower where the HD's went out.

Also I noted this time around on boot that one of the LEDs for the drive lights is not coming on. Could possibly be a damaged board. This tower has been running since 2009.