Rolling back freenas-boot manually

[mortar]

I originally installed freenas to a pool on two 16GB usb sticks. Many an update went fine. The latest (FreeNAS-9.3-STABLE-201602031011) update however froze in the webgui. I do all my updates manually as the nas box is not connected to the Internet. I ran the update again, and it looked like it passed. After a reboot, however, I'm greeted with GRUB rescue mode , after an error about i386-pc not being found.

I booted with the installation ISO, hoping to upgrade, keeping my configs, but the installer only offers to wipe the drives and do a full install. I also zfs imported freenas-boot in installer shell to see if there's anything there and I cannot locate the grub files. I mounted a last known good /ROOT/FreeNAS-FreeNAS-9.3-STABLE-somethingsomething and it seemed to contain files, but nothing under boot/grub.

Can I somehow, using the installer shell, rollback to the last known good snapshot (in this case 20151212) of freenas-boot and enable it to boot again or do I have to backup the configs (which files?) and do a clean reinstall?

 

[BigDave]

The latest (FreeNAS-9.3-STABLE-201602031011) update however froze in the webgui.

Did the machine freeze when you were downloading and applying the update?
You may have a bad USB stick...
The install ISO only does installation, not updates.
You need to roll back to your last working instance,
upon getting back to the GUI, save your config.
For future reference, ALWAYS save a copy of your config file BEFORE
performing ANY changes to your machine. This includes a reboot!!!

 

[dlavigne]

I do all my updates manually as the nas box is not connected to the Internet. I ran the update again, and it looked like it passed. After a reboot, however, I'm greeted with GRUB rescue mode , after an error about i386-pc not being found.

Can you clarify how you update manually (eg what steps do you perform)?

 

[mortar]

Did the machine freeze when you were downloading and applying the update?
You may have a bad USB stick...
The install ISO only does installation, not updates.
You need to roll back to your last working instance,
upon getting back to the GUI, save your config.
For future reference, ALWAYS save a copy of your config file BEFORE
performing ANY changes to your machine. This includes a reboot!!!

Nope, the machine did not freeze. The update progress bar froze (on Firefox 44). I update by downloading the update from the freenas website and use the webgui to select the file and paste the sha256 sum from the website to the same dialog. Feel free to suggest a better procedure for a non-Internet-connected box if there is one.

The main menu of the install ISO has an option "Install/Upgrade". Does that "upgrade" part mean nothing anymore then? I suppose it at least has been an actual feature of the installer at some point and not just wishful thinking.

Any more specific suggestions on the rollback procedure? If I just rollback or rather copy one of the zfs clones that have been created by updates (why are there clones AND spnapshots, btw.?) will I get a working GRUB back, too? There are two snapshots of GRUB, but they are both "Preinstall snapshots" iirc. I'll have to recheck the options, so I can ask more specifically.

There really, REALLY should be a mnual recovery section in the manual that's kept up-to-date. I found some information from various #bug reports, but they conflict and are apparently not representative of the current version, eg. what are the config files to backup in case I want to do it manually from the command line, etc.

 

[BigDave]

I was wrong (see strikethrough above), you can use steps in section 2.5 in the
manual to update from the installer ISO.

If something goes wrong, the Installer package should see the previous
install and allow you to revert back to that instance, see section 2.5.5

Sorry for the misinformation

 

[mortar]

So, solved this one myself, I hope.

1. Backed up the OS sticks in case I fck up using
   Code:

   dd if=/dev/adaXX of=FreeNAS-stick-a.img bs=20M

   
   
2. Booted into the installation ISO FreeNAS-9.3-STABLE-201602031011 and entered shell
3. Imported the freenas-boot pool using
   Code:

   zpool import -R /mnt freenas-boot

4. Looked for the best candidate for rolling back to using
   Code:

   zfs list

   and
   Code:

   zfs list -t snapshot

   There are entries like freenas-boot/ROOT/default and freenas-boot/ROOT/Initial-Install and freenas-boot/ROOT/FreeNAS-FreeNAS-9.3-STABLE-YYYYMMDDXXXX. Only one of the FreeNAS-FreeNAS-9.3.STABLE-XXX entries had any snapshots.
   
5. Mounted one of the FreeNAS-FreeNAS volumes using
   Code:

   zfs mount FreeNAS-FreeNAS-9.3-STABLE-XXX

6. Was unsure whether grub should already be there or a separate volume. Got nothing using
   Code:

   ls -la /mnt/boot/grub

7. So, mounted a grub volume using
   Code:

   mount -t zfs freenas-boot/grub /mnt/boot/grub

   Still, got nothing using
   Code:

   ls -la /mnt/boot/grub

   
   
8. Grub was apparently wiped out by the update (see first post). So, went to see if there are any snapshots in there using
   Code:

   cd /mnt/boot/grub/.zfs/snapshots

9. There were a couple of Preinstall snapshots from the first upgrade I ever installed, but nothing even remotely recent. Since that seemed to be the only chance to restore grub, copied the most recent snapshot back to the main grub volume using
   Code:

   cd /mnt/boot/grub/.zfs/snapshots/Preinstall-XXXX; cp -R * /mnt/boot/grub

10. Compared some of the config database files in the ROOT volume snapshots (under /mnt/.zfs/snapshots) to the current one and the current one seemed to be ok. So, selected the mounted ROOT volume (FreeNAS-FreeNAS-9.3-STABLE-XXX) as the one to boot into. The config database is in
    Code:

    /mnt/data/freenas-v1.db

    
    
11. The grub boot menu would now point to ROOT/default and some totally outdated boot environments, so, I edited it using
    Code:

    vi /mnt/boot/grub

    Using search and replace, replaced one boot entry's "Initial-Install" with "FreeNAS-FreeNAS-9.3-STABLE-XXXXX" (the one I mounted). In this case ROOT/default was also hosed and for some weird reason contained some old, old snapshot - possibly the same as the grub one, so, it was unusable.
    
12. Booted and got the system back
13. As an exercise, backep up the config using the webgui, did a full reinstall using the ISO and restored the config and it appears to work.

Based on my experiences so far, I have to say, I would never consider using FreeNAS for anything but non-critical stuff in a non-production environment.

Fwiw, the installer DID NOT see the previous installed version on the USB sticks until GRUB was restored, and even after that, it found an ancient version because ROOT/default had something ancient in it. I tried the above procedure once with ROOT/default instead of ROOT/FreeNAS-FreeNAS-9.3-STABLE-XXXX..

Fwiw2, the USB sticks were fine as neither zpool scrub or full stick dd read and write reported errors or caused errors in dmesg.

 

[BigDave]

1. As an exercise, backep up the config using the webgui, did a full reinstall using the ISO and restored the config and it appears to work.

Based on my experiences so far, I have to say, I would never consider using FreeNAS for anything but non-critical stuff in a non-production environment.

You suffered this experience only because you had no saved configuration file, surely you don't hold FreeNAS
responsible for what happened? IMHO no one should be using a $10 flash drive for a critical production environment anyway!

 

[mortar]

You suffered this experience only because you had no saved configuration file, surely you don't hold FreeNAS
responsible for what happened? IMHO no one should be using a $10 flash drive for a critical production environment anyway!

Well, I kind of do hold FreeNAS responsible (and certainly mean no offense to you, whatever your affiliation with FreeNAS may be). User error caused additional time lost in finding out how to recover and some additional steps taken to do it, compared to if I had backed up my configs. Reinstall was inevitable in any case and it was caused by the failure of FreeNAS to maintain its great-on-paper rollback system functional, which should have allowed a safe and painless rollback straight from GRUB (had GRUB not been completely destroyed by the failed upgrade). Additionally, my bad experiences with FreeNAS are not limited to this incident, but others as well. Simple things, that should have just worked imho. Only one failure was caused by "bad" hardware, which was also admitted to work elsewhere, but fail on FreeBSD because of driver support (Marvell SATA controller). I sure hope this fresh install fares better. ZFS is just too nice to give up :)

And it was not a $10 flash drive - more like $7 :P But this failure can't be blamed on the drives, although you're right about not using those things in production.