FreeNAS Random Reboots

[Visseroth]

Since my update to 11 from 9.10.2 I've been experiencing random reboots.
Can anyone help me troubleshoot as to why? Seems the logs are trimmed back far enough I don't see a reason why or the reason wasn't written.
I didn't have this problem in 9.10.2

 

[Chris Moore]

My understanding of your configuration is that you have two RAIDz2 vdevs of 6 drives, at 4TB. All the hardware you have looks good to me, even if it is a little old...
Did you verify your upgrade? Something might have gone sideways. I wasn't able to upgrade. It just wouldn't work and I had to do a fresh install of the OS and reload my config database. It has been working pretty well since then.
Did you take any other troubleshooting steps? I had one of my systems random reboot when one of my SAS controllers went bad.

 

[Visseroth]

It may be a bit old, but it was within my means and does the job.
I did indeed verify the upgrade. Only thing I haven't done is a wipe of the OS and reload and reload the config. With all the bugs I'm seeing that may very well be my next step.
How did you know your controller went bad?

 

[Chris Moore]

It may be a bit old, but it was within my means and does the job.
I did indeed verify the upgrade. Only thing I haven't done is a wipe of the OS and reload and reload the config. With all the bugs I'm seeing that may very well be my next step.
How did you know your controller went bad?

Please don't take it as criticisim. My gear isn't much newer. I moved from the same processor you have to what I have now on my last upgrade cycle and I get about the same performance with less electricity used.
I knew that my controller was going bad because over the last year or so, when there had been an occasion to reboot the system, it had failed to be recognized as being in there but doing a full shutdown and reboot had made it work again. This time (watching the console) I was able to notice that the card and attached drives were not being recognized and the system would reboot at the point where it should have been mounting the pool. I replaced the card and everything has been fine since.

 

[Visseroth]

Dude, I just have to say thank you for being gentle about it, LOL!!!
The information you provided you HIGHLY valuable. I purchased this server last year hoping to actually bring my electrical costs down a tiny bit while improving performance and my storage pool size and really it's been a little buggy since the beginning and now I'm wondering if what you are saying is the reason I'm getting the attached error...


In order to get that error to go away I have to power the server off and then power it back on. Basically making all the hardware shutdown. Resets and/or reboots stop at the attached error.

 

[Chris Moore]

Dude, I just have to say thank you for being gentle about it, LOL!!!
The information you provided you HIGHLY valuable. I purchased this server last year hoping to actually bring my electrical costs down a tiny bit while improving performance and my storage pool size and really it's been a little buggy since the beginning and now I'm wondering if what you are saying is the reason I'm getting the attached error...
View attachment 20605

In order to get that error to go away I have to power the server off and then power it back on. Basically making all the hardware shutdown. Resets and/or reboots stop at the attached error.

It certainly looks like a hardware error. It makes troubleshooting more difficult that it only happens on a warm boot and not on a cold boot. As a troubleshooting step, I would suggest disconnecting everything possible from the system board and boot it up. Then run it through a few warm boot cycles to see if the error happens. You would go through this process of shutting the system down, add one item back, the warm boot a few times watching for errors, and repeat, until you can isolate one device that recreates the error. If the bare system board, with just processor and RAM is not random rebooting, then it isn't likely the source of the fault. If you can isolate a card that is the cause, it can be replaced. At least it isn't intermittent. A fault that is consistent, it easier to isolate.

 

[Visseroth]

Makes sense, and no, there are 5 LSI 9211-8i cards, so as soon as I have a chance to pull the server out and start testing I will and report back.
Let's see what I can get figured out.

 

[nightshade00013]

FIVE....Along with the CPU that thing can put out a TON of heat which could be an issue.

I absolutely have to ask though, why use 5 HBA's instead 1 with a SAS expander or two? It would drop the electricity use of the system as well as the heat generated by a ton.

 

[Visseroth]

The CPU was for horse power of course. The idea was to have it idle back, not consuming a bunch of power when it's not needed but when needed it'll do what it needs to do.
The server I picked has a SAS connector per slot and there are 36 bays.
So yea, I could have picked a server with a SAS expander backplane, but this is what was available when I was looking for a server in my price range.
So why so many? well each card handles 8 drives. 8*5=40 so that leaves one card only being half used.
I'm seriously thinking of switching to something like this SuperMicro SC216 as soon as I'm able which likely may not be for another year, maybe two. I purchased this one with the intent of using it for 5 to 8 years but I've had nothing but problems with the SuperMicro X8 series and now it seems that I may have a controller acting up.
So, hopefully that answers your questions.
Personally, I'd like to have a server that idles at 80W> peaking what ever it needs to accomplish the job it has been given, whether that be 200 to 400W, I don't care, but most the time this thing idles and I don't like the 350W draw. That is just unnecessary with the available technology.

 

[Chris Moore]

I'm seriously thinking of switching to something like this SuperMicro SC216 as soon as I'm able which likely may not be for another year, maybe two. I purchased this one with the intent of using it for 5 to 8 years but I've had nothing but problems with the SuperMicro X8 series and now it seems that I may have a controller acting up.
So, hopefully that answers your questions.

Naturally, you do what is right for you, but I would stay with the 3.5" disks and I feel they are a better value. That other chassis you linked to uses 2.5" disks which are a bit more expensive for the amount of data they can store.

You can put a different (newer) system board in the chassis you have and you can probably find relatively inexpensive SAS2 expander backplanes to go in place of what is in there now. You don't have to chuck the chassis to get a better ride when a new set of rims and tires will get you there.

 

[Visseroth]

I was just thinking about power savings. 2.5 inch drives, granted are slower, but do have better power utilization and you can get them up to 4TB now. Soon SSD drives should be dropping in price and for those that have the big bucks, it is possible to get 32TB SSDs, which is just insane.
But you make a valid point. I could slowly stock up the parts and just swap all the parts out one day when they have all been collected.
One thing I DON'T like about this system is the HD bay lights are on by default and dim with activity which is actually really annoying. To fix that the backplanes would have to be swapped. So I'll have to look and see what's available for this chassis.
Question is, what controller cards would I then be running. These LSI 9211-8i or something else?

 

[Chris Moore]

Question is, what controller cards would I then be running. These LSI 9211-8i or something else?

If you get the SAS expander backplane, you would run a cable that has SFF-8087 connectors on both ends from the card you have now to the backplane and the connection to the individual drive is done inside the backplane after the signal is processed by the controller chip on the backplane. This isn't technically accurate but it is kind of like a network switch. You just need one cable from the SAS card to the backplane to run all the drives, but you can run two cables from the card to the backplane for better performance. So, if you replace both backplanes with expander type, you need at most two SAS cards instead of the five you have now.
This is a custom cable for a Dell server, but here is what it could look like:
http://www.ebay.com/itm/Dell-YY6G3-...D-Controller-to-Backplane-Cable-/372032815591

PS. You could run one cable from your SAS card to each backplane and control all the drives with one card. If I recall correctly, one of those cards will control 256 hard drives. That is from memory, so it could be totally wrong, but it is way more than 36.

 

[Visseroth]

Good to know. Well I think I'll start digging into that and digging into swapping out my motherboard, cpu, ect. I'd like to get this thing idling down much lower than 350W.
So thank you for the information and tips! I think your recommendations are great and I'll start doing my homework. I'm also thinking of trying to swap out my Chelsio T420 for a 520.
But I'll make myself a laundry list and get start building a pile of parts.

 

[Chris Moore]

If I was buying a new system board (new to me) I would get one of these:
http://www.ebay.com/itm/Supermicro-...ard-Intel-C612-Chipset-Socket-R3/263210181322

 

[Visseroth]

It's a good price for a motherboard, for sure, but holy cow on the RAM. 64GB+ is $800 to $1800 bucks
I'd rather pay the electrical bill as it would take me about two years in savings to offset the costs and that's assuming at least a 50% reduction in energy consumption.
I could save just as much getting $500 worth of solar panels and grid tying with a grid tie inverter. Well I might be exaggerating a bit, but still. I'd be spending about 2 to $3000 just on the motherboard, RAM and CPU.

 

[Chris Moore]

It's a good price for a motherboard, for sure, but holy cow on the RAM. 64GB+ is $800 to $1800 bucks
I'd rather pay the electrical bill as it would take me about two years in savings to offset the costs and that's assuming at least a 50% reduction in energy consumption.
I could save just as much getting $500 worth of solar panels and grid tying with a grid tie inverter. Well I might be exaggerating a bit, but still. I'd be spending about 2 to $3000 just on the motherboard, RAM and CPU.

That is why I am not doing it, just wishing I could. A second best option might be to go with the next older revision, the X9 series board as they use DDR3 memory which is selling for a little less and the CPUs are also a little less all because it is the older tech.
Searching for the deal is half the fun.

 

[Visseroth]

Ahh, yea. If we're going to dream big I'm dreaming of a SSD array with 32TB drives. Probably 4 of them, lol.
I was just looking at the x9 options. Seems they use the same DDR3 that I'm using currently, which means I wouldn't need to get new RAM and I also found this for backplane options for the SC846 series.
So I'd be looking at a new MB, CPU and backplane but same everything else minus about 2 or 3 LSI cards which would give me some parts so when I start troubleshooting the cards to see if one of them is starting to act up I'll have a couple extras to swap which ever one is acting up.
The other option is to find a 846 case with the backplane I want. One may spend just as much.

 

[Chris Moore]

The other option is to find a 846 case with the backplane I want. One may spend just as much.

If that is something you are interested in, this is one I was looking at:
http://www.ebay.com/itm/Supermicro-...re-Barebones-24-Bay-SAS2-846EL1-/292252408126