"zpool status" doesnt complete

[airflow]

Hi,

I have a FreeNAS with a minor problem here. Hardware is ASRock E3C226D2I, Intel Core i3-4130T, 2x 2.90GHz, Kingston ValueRAM DIMM Kit 16GB (ECC capable).

The boot-device are two mirrored USB-sticks. One of those "has experienced an unrecoverable error" (but the boot-zpool itself is ONLINE). I decided to take the faulty USB-stick, format it, tried to write data on it to test it (worked fine), and to put it back on the FreeNAS into the boot-mirror (I had to use the "replace"-function in the GUI to do that).

Resilvering of the UBS-stick started and seemed to work fine. The next day, I wanted to check the progress by issuing "zpool status" on the CLI. When I do that, this command doesnt complete. It just hangs seemingly forever, I cannot even stop the programm with Ctrl-C. In the Reporting-Stats in the GUI I see that nothing is read/written to the sticks anymore (except normal activity).

Code:

[root@fractal] ~# zpool status
^C

In the GUI, the section for "Boot" within "System" doesnt load. What should I do here? My only ideas are 1) rebooting the box and recheck or 2) remove the USB-stick again physically and see what happens.

Thanks,
airflow

 

[dlavigne]

Alternately, do a fresh mirrored install onto the 2 sticks and restore your config.

 

[airflow]

I tried now the steps I gave before. First I tried to physically remove the problematic USB-stick, which didn't resolve the problem. The commands were still hanging and there was no change in behaviour. Then I tried to reboot the system via the GUI, which also didn't work, neither the "reboot" command from CLI (there was no output on the shell whatsoever).
So I had to a hard reset to try to resolve the situation. But the system could not boot any more after that. No matter whether the problematic stick was inserted or not or if I choose one or the other to boot in the boot menu, grub refused to boot with the error message that there was "no such device <id>".
When I searched the forums, I found this thread, which basically had the exact same problem and also a way for resolution (which involves using the rescue-system from boot-media and scrubbing the pool). There seems to be a bug in FreeNAS which prevents it from handling this situation more gracefully.

 

[gpsguy]

Flash drives are cheap - install FreeNAS on one or two NEW ones and restore your config. Or buy a small, inexpensive SSD drive and install FreeNAS on it.

 

[airflow]

You are right and it's true that 1) FreeNAS can be easily re-imaged and saved configuration can be used and 2) it's not worth the time to spend it to get a faulty USB-stick back to life. i agree on both, but you are missing the point, because the problem is that the system failed to boot when one of the two redundant USB-sticks made problems. As there was a redundant boot-device in place with one of the USB-sticks still perfectly functionable (which still had all the data to boot on it), it should have worked to boot from that, no? Because if that's not properly working you can skip the part with dual boot-devices and just recover in cause of single-bootdevice failure with the steps you said.

 

[nojohnny101]

because the problem is that the system failed to boot when one of the two redundant USB-sticks made problems. As there was a redundant boot-device in place with one of the USB-sticks still perfectly functionable (which still had all the data to boot on it), it should have worked to boot from that, no? Because if that's not properly working you can skip the part with dual boot-devices and just recover in cause of single-bootdevice failure with the steps you said.

I think you had more than just a "bad usb drive in a mirrored pair fail and thus crashed the system". Mirroring boot drives is what addresses this problem. I know it didn't work in your instance, which should fairly obviously point out that there is a different problem at work here.

I (and thousands of others I'm sure) have run mirrored USB sticks, had one fail, and encountered the expected behavior of still being able to boot/function, but in a degraded state until the bad USB drive was replaced with a new one (or a fresh install with new good USB drives was made, as @dlavigne had suggested).