FreeNAS 11.2 U2 server keeps restarting

[strudel]

Hi,

Since 2 days, my FreeNAS 11.2-U2 server keeps rebooting. Just before it reboots, I get this error message (see attached picture).

The process is always the same : boot -> server OK -> x amount of time/or whenever I try to access the server via the Web UI -> reboot

How can I fix this? All my data is backed up.

Thanks

 

[droeders]

Based on the panic trace, I would bet that your boot/root pool is corrupted.

You didn't list your hardware specs. Are you booted off USB flash drives? Do you have a backup of your configuration that you can use to re-install FreeNAS and restore from?

 

[scrappy]

Interesting. I was experiencing similar panics starting a few weeks ago. I thought it was my mirrored USB Thumb drives failing me so I replaced them with some old 2.5" laptop hard drives and FreeNAS didn't crash again...for about a week. Then the same thing happened. It also happened around the start of a scheduled zpool scrub if I recall. The next time I restored FreeNAS, I found out by happen-chance that it didn't crash again until I re-imported one of my encrypted zpools. Probably not even a minute after import the message (in my posted image) popped up before the server's watchdog service forced a reboot. Since that time I rebuilt that zpool (without encryption) and so far FreeNAS has been running smoothly with no crashes - but it's only been a few days. Also, I have no idea whether or not having an encrypted zpool had anything to do with FreeNAS crashing, but it would appear there was some correlation with FreeNAS crashing at the same time of my importing that particular zpool.

 

[strudel]

Based on the panic trace, I would bet that your boot/root pool is corrupted.

You didn't list your hardware specs. Are you booted off USB flash drives? Do you have a backup of your configuration that you can use to re-install FreeNAS and restore from?

I boot off 2 mirrored USB sticks (brand new), I changed my boot device like 2 weeks ago. I have a backup of my config and my data. I think there's some issues with my main pool as I got DEGRADED errors in the past, could that be it? Could it be related to a hardware (RAM/CPU) issue?

 

[strudel]

Interesting. I was experiencing similar panics starting a few weeks ago. I thought it was my mirrored USB Thumb drives failing me so I replaced them with some old 2.5" laptop hard drives and FreeNAS didn't crash again...for about a week. Then the same thing happened. It also happened around the start of a scheduled zpool scrub if I recall. The next time I restored FreeNAS, I found out by happen-chance that it didn't crash again until I re-imported one of my encrypted zpools. Probably not even a minute after import the message (in my posted image) popped up before the server's watchdog service forced a reboot. Since that time I rebuilt that zpool (without encryption) and so far FreeNAS has been running smoothly with no crashes - but it's only been a few days. Also, I have no idea whether or not having an encrypted zpool had anything to do with FreeNAS crashing, but it would appear there was some correlation with FreeNAS crashing at the same time of my importing that particular zpool.

Let me know if your problem is solved, my pool is not encrypted so I don't think that's the issue.

 

[scrappy]

Let me know if your problem is solved, my pool is not encrypted so I don't think that's the issue.

I'll give an update here after a week or so of continuous uptime. I manually restarted it a few days ago for unrelated reasons, but so far my server has been running smoothly after the system restore, zpool rebuild + zpool scrubs. If it stays this way I'll feel more convinced the forced restarts were being caused by my old zpool whether it was corruption or something else.

 

[droeders]

I think there's some issues with my main pool as I got DEGRADED errors in the past, could that be it? Could it be related to a hardware (RAM/CPU) issue?

Please read and heed these forum rules - both for this and subsequent posts:

https://www.ixsystems.com/community/threads/forum-guidelines.45124/

I can't say what the root cause is, but the panic that you posted looks related to ZFS. I definitely don't like the sound of unexplained DEGRADED errors. Did you get to the root cause of these when they happened?

Let's restart with the forum rules that you have now read. :) What are your hardware details?

 

[scrappy]

Follow up: My server has been running for a week now with no crashing/forced reboots.

Re-Follow up: Nevermind. My FreeNAS system rebooted itself over and over again throughout the early morning with my first ipmi event log showing a hard reset occurred at 3:05 am with 18 more hard resets to follow. At 3:17 am I started receiving UPS report emails — 12 emails in total of "COMMBAD - ups" and then "COMMOK - ups" which has not previously happened to me. At 3:45 am an email was sent reporting a scheduled scrub on my freenas-boot disk started. The system also had other scheduled tasks like zfs replication happening around this same time in the morning.

This whole ordeal has been quite infuriating. When I get home tonight I will try installing a prior version of FreeNAS and hopefully that fixes this particular problem.

 

[scrappy]

Over the past 24 hours I have rebuilt my FreeNAS system based on version 11.1-U7. The hard lesson learnt here is to never update a heavily relied upon FreeNAS server until at least 3 or 4 (maybe more?) minor updates have released to address various bugs. Since I no longer have my prior system + system logs that might've told a story of what happened before each crash, I can only guess the likely culprit(s) were:

-ZFS replication tasks to another zpool on same server
-ZFS periodic snapshot tasks
-ZFS scrub tasks

All these tasks listed were set to occur early in the morning on my server - the same time it had begun crashing continuously. I remember that morning after waking up when FreeNAS had just came back online after its 19th forced reset (according to my ipmi event log) and seeing the ZFS replication task icon start moving in the top right corner of the screen. A moment later FreeNAS went down its last time before I forced my server to stay turned off.