My system just crashed and I need help figuring out why.

[CompuGlobalHyperMegaNet]

First of all, specs or in my sig.

I currently have two pools, tank, which is my primary pool for the system in question and dozer which is from my backup system.



Second, here's what I was doing when I experienced the crash (in roughly chronological order).

1. A few days ago, I copied the data from dozer to tank using zfs send | recv. I then Exported dozer and powered down my primary server order to remove the disks but I encountered an issue where my system would reboot upon being shutdown via the webGUI. I went into the BIOS and checked that headless mode was enabled and made sure that there was no wake-on-LAN shenanigans going on. I'm not a 100% sure the issue is fixed but the system did shutdown fully, I removed the disks from the detached pool and rebooted.

2. I've been reorganising some datasets from dozer today, that I had copied from the now detached pool using send recv. I used the variations on the following command to move my various datasets... "zfs rename tank/dozer_backup/data tank/data". This included moving the jails dataset from dozer to tank.

3. After renaming the datasets, I changed the jails root configuration, updated my jail storage paths and fired up my Deluge, Plex and Syncthing Jails. The webGUI for my server's Syncthing wouldn't load, so I had to edit the rc.conf using vi. After loading my trusty vi cheatsheet bookmark, I updated rc.conf and closed the shell.

4. This is where my memory fails me... I either tried to shutdown the Syncthing Jail (from the FreeNAS WebGUI's Jails page) or restart it. The system became unresponsive, as did the PuTTY window I had open, luckily I had the good old IPMI iKVM window open which was going crazy and spamming text faster than I could read. Fortunately I managed to grab the following screenshot before the system rebooted. From what I could tell, all the other messages that appeared in the iKVM were the same basic format.

So, what happened? Did I just pop my kernel panic cherry?

 

[Chris Moore]

Did the system boot back up and work correctly or is it still in a failed state?

 

[CompuGlobalHyperMegaNet]

It's working as normal.

 

[Chris Moore]

So, what happened? Did I just pop my kernel panic cherry?

There are some changes that you can make to a jail configuration that will crash the system. I ran into that a lot when I was using more jails. For example, PHP Virtual Box. That thing crashed my system a lot. Now that I am only using Plex, I hardly ever have a crash. You might want to consider upgrading to the latest version of FreeNAS if there is no compelling reason not to.

 

[CompuGlobalHyperMegaNet]

There are some changes that you can make to a jail configuration that will crash the system.

I didn't know that... I'll have to do a bit of research I think.

I ran into that a lot when I was using more jails. For example, PHP Virtual Box. That thing crashed my system a lot. Now that I am only using Plex, I hardly ever have a crash. You might want to consider upgrading to the latest version of FreeNAS if there is no compelling reason not to.

I currently have only two copies of my data, one of which is on the dozer pool (4 disks consisting of 2x8TB mirrors) which has a disk going bad. Before I stress the questionable disk by resilvering, I want to setup CrashPlan for Small Business on my backup server (which is running 11.1) and finally comply with the 3-2-1 Rule (sort of). Once that's done, the plan was to upgrade my primary tank system to 11.x... I'm not sure whether that's a compelling reason to upgrade now or not.

Either way, before I upgrade, I'd like to get at least a basic idea of what happened as I've never had it before and I'd like to take the opportunity to learn something new.

 

[Chris Moore]

I am guessing that the things you changed in rc.conf were network related. That's what usually did it for me. If I changed network information in a jail, it didn't always crash but it happened almost half the time. There were also odd memory issue where FreeNAS didn't reserve memory for the jails. I have not tried to make a new iocage jail yet, but it is supposed to be better.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[CompuGlobalHyperMegaNet]

The alteration I made to the rc.conf was changing synthing_enable="NO" to "YES" (copy of rc.conf below for reference).

Code:

root@syncthing_1:/ # vi /etc/rc.conf
portmap_enable="NO"
sshd_enable="NO"
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
hostname="syncthing_1"
devfs_enable="YES"
devfs_system_ruleset="devfsrules_common"
inet6_enable="YES"
ip6addrctl_enable="YES"
syncthing_enable="YES"

 

[Chris Moore]

I am guessing, based on the kind of behavior I saw when I was using a jail to run PHP Virtual Box, when you started this back up it probably tried to reinitialize the network stack and that caused problems that were not able to be handled. I think it is a bug, but I don't know if it would behave any differently in a newer version of FreeNAS. I know they have fixed many things with the move to version 11. The other thing is, this may not get fixed because they are moving to the iocage jails. I don't know if that impacts the functionality of the virtual network stack, but it is supposed to be better in some way that is (as yet) not clear to me.

 

[CompuGlobalHyperMegaNet]

Just a little update. I've just had another crash with the same spamming of the IPMI iKVM Shell. This time though, I (sort of) hadn't directly changed any network related settings on my FreeNAS system itself. I've been trying to modify the config of my OpenVPN server on my pfSense box but for some reason, the dang OpenVPN client just won't get an ip. I refreshed / restarted / reset a few things of pfSense and I also restarted the Deluge Jail, serveral times in fact, so I did change network settings on my FreeNAS system but I have restarted the Jail one too many as my FreeNAS box crashed.

On the bright side, it looks like I've got a pretty good idea of what kind of monkeying around causes the crashes.

Any tips on avoid crashes when modifying network settings on FreeNAS?

(or perhaps I should hold of with fiddling about with things until I've upgraded to 11.x)

 

[Chris Moore]

Upgrade to 11 will change the underlying OS from FreeBSD 10 to FreeBSD 11 which should give you some stability improvement.
No guarantees.