I m a strange attractor

[Constantin]

Good morning,
Last night, I applied the latest 11.2 Stable update and then had the machine reboot. However, instead of booting, the system is locked in a non-ending cycle of boot.

At the bottom of the console screen, the console shows
"BMC Initializing..." with the 04 and 01 in the opposite corner followed by
"System Initializing..." for a few seconds whereupon the system hard resets

Any ideas what I can try first? I'm inclined to try a different, known-good, larger power supply first to ensure that power is not a limiting factor. However, the system has been fine for multiple reboots due to scheduled and unscheduled reboots in the past (the power company has a habit of turning off power now and then, the UPS allows a graceful exit).

(and in case you're wondering about the title, multiple IT support folk at work gave up on solving persistent, reproducible bugs that only allegedly affected my CPU in the whole company.)

 

[Basil Hendroff]

I had exactly this problem. It turned out to be a corrupted boot disk. I was able to boot from the secondary, of the mirrored boot disks and recover that way.

 

[Constantin]

Thank you!

That is going to be interesting. I only have one boot drive (the FreeNAS private-labeled SATADOM). I suppose the easiest way to test this hypothesis is to unplug the SATADOM and see if the pattern persists. If it is the SATADOM, how do I fix it?

I presume that in your case you simply disconnected the primary boot drive and let the secondary take over?

 

[Basil Hendroff]

That's correct My boot disks are USB sticks so this was easy to do. Possibly an easy way for you to confirm the corruption is to temporarily disconnect the SATADOM and build and boot FreeNAS from a USB stick. If it successfully boots, then you know the SATADOM boot drive is corrupted. You can then rebuild it by installing FreeNAS on to it and restoring the config (assuming you have previously saved it). If you haven't saved it, I believe it is stored in the system dataset. Search for this in other recent posts.

 

[Kevo]

If you are in fact using this board, AsRock C2750D4I, then you could be bit by one of two issues with those boards. When it happened to us I had a second box with a different motherboard and was able to swap drives and everything came up fine so I knew it was the board. The board was still barely within the warranty timeframe so AsRock swapped it for us.

From memory the two potential issues are a failure/wear out of one of the bios chips, maybe the ipmi bios, and an issue with the power rails going bad. Support at AsRock should be able to help you out, or if it was purchased and has warranty from iXSystems, they should be able to help out as well.

We actually needed to get up and running pretty quick, so we went ahead and bought a new board and then when the RMA came back we built a 3rd box with it.

 

[Constantin]

Thank you both. I will try to look into this further come evening. I hope it's not the wrong BIOS revision leading to a BIOS flash failure. I already had my original C2750D4I board replaced once due to the AVR54 clock generator issue. I sure would hope that the board that iXsystems sent back would have the right BIOS revision on it to prevent the BIOS flash from getting worn out.

 

[Constantin]

OK, checked out the two firmware/flash chips. The BMC F/W is P00.35.00 and the Flash is P2.90. I do not recall ever touching either re: flashing, so I'll assume for now that the chips are as they were when they left iXsystems.

Removing the SATADOM had no effect. The system will boot for approximately 57 seconds and then it'll hard reset. You can see the SATADOM dangling in the bottom of the frame. Here is a video of the boot process from the motherboard perspective: https://share.icloud.com/photos/08QkXFrukYARRXV4SoQ5pcSbQ

Should I try changing the power supply just to rule that out?

 

[Basil Hendroff]

Coincidence? https://www.ixsystems.com/community...-your-asrock-c2550d4i-or-c2750d4i-last.45445/. I guess there's no harm ruling out the power supply anyway.

 

[Constantin]

I've gone ahead and contacted iXsystems for further guidance. I'll change out the PS today to eliminate that possibility but I doubt that's the issue since all the drives are well-spun up by the time the system hard resets. In other words, the period of highest PS-stress is well behind the rear-view mirror by the time the system abruptly and reliably shuts down at the 57s mark, SATADOM or no SATADOM.

I've been contemplating switching to a Xeon-D system for a long time now so this may push me over the edge and I may simply elect to replace the ASRock Avoton board with a SuperMicro Xeon-D board regardless of the repair / replacement outcome. I suppose that will make selling the mini XL a bit easier as it will be a complete system, ready to go.

 

[Constantin]

Well, color me a continued satisfied customer of iXSystems. They took down the info, verified warranty eligibility, and now I am in the queue for my second replacement board in a about a year. Due to the current backlog, iXSystems announced it will take a few weeks to get me a new board.

However, I am a bit perplexed that the first replacement board allegedly has the same clock generator issue as the first one. Given that my replacement board was shipped in mid-December 2017, that's about 9 months after the Intel bug was acknowledged and a fix announced. Perhaps it took the ASRock and/or Intel a lot longer to actually fix the issue?

I guess I should place that order for the Xeon-D system. The extended 3-year warranty that iXsystems offers on the Mini XL will run out this September. I'd rather opt for an extended life board from SuperMicro at this point.

 

[Basil Hendroff]

That is good news! Doesn't the replacement board get another 3 years warranty?

 

[Constantin]

My expectation is that it’s three years from purchase of the mini XL.

iXSystems may or may not allow a different warranty on the replacement board.