Upgrade from 9.2.1.7 to 9.3 stable fails

[andyl]

Hi all,

I have a Microserver N40l running FreeNAS 9.2.1.7 which I'd like to upgrade to FreeNAS 9.3

Under 9.2.1.7, I'm using an old 2Gb generic USB2 stick as my boot media. I'd like to replace this with a mirrored pair of 8Gb branded USB2 sticks for 9.3

As I understand it, the best procedure to follow is to conduct a new install of 9.3 to the new media and then import my config backup from 9.2.1.7. I've read on this forum that just conducting an upgrade and replacing the boot media at the same time will leave the partitions on the new boot media left at the old size (sorry - I looked for the post, but can't seem to find it). I don't believe I'll be able to install 9.3 onto a 2Gb stick.

So I followed the procedure as below;

1. Backup existing config.
2. Create new install media.
   I followed the procedure as described in the 9.3 user manual to create install media from the 9.3 install ISO to a third USB stick (Win32DiskImager).
3. Boot off install media
   I shutdown the server and removed the 9.2.1.7 boot media.
   I inserted the install media and the new (currently blank) boot media into the server. On startup, the server entered the 9.3 install wizard as described in the 9.3 manual. I installed 9.3 using the defaults.
4. Log into web GUI.
   The GUI launched an import wizard. I got a few screens into this before exiting it - I had imported my existing volume, but not made any changes - I think.
5. Import config backup.
   
6. Server restarts.
   The server restarted into 9.3 and performed a DB upgrade of some sort, before seeming to get stuck in a cycle. The options menu on the server console kept scrolling and I received frequent errors on the console indicating that /var/log/messages could not be found. The web GUI was not available. Logging into the server via SSH, I could see that the ZFS volume had not been imported.

I was able to rollback to 9.2.1.7 by shutting down the server and replacing the new boot media with the old, at which point the server came up into 9.2.1.7 and is running happily without any apparent data loss.

Anybody able to offer any pointers?
Am I over-complicating matters doing an install rather than an upgrade?

Thanks for reading.

Cheers,
Andy.

 

[Robert Trevellyan]

You might have better luck updating to the 9.2.1.9 update before attempting the upgrade.

 

[andyl]

You might have better luck updating to the 9.2.1.9 update before attempting the upgrade.

Thanks, I'll try that.

 

[joeschmuck]

If you are having issues still after the below steps, post your system hardware specs, how much RAM do you have specifically?

Also you shouldn't have to use 9.2.1.9, I do not believe there are any differences from an upgrade perspective between the two, however it's worth a try.

The easy way to do the upgrade would be to take your new USB Flash drive, install 9.2.1.7 to it and restore your config file, reboot. Things should work well and now you are running using an 8GB boot flash drive. Now do an upgrade to 9.3.

Good Luck.

 

[andyl]

If you are having issues still after the below steps, post your system hardware specs, how much RAM do you have specifically?.

8Gb of ECC memory, 4 of 2Tb drives for a post-RAID capacity of 5.1Tb.

Also you shouldn't have to use 9.2.1.9, I do not believe there are any differences from an upgrade perspective between the two, however it's worth a try.

Oops - too late, already done :)

The easy way to do the upgrade would be to take your new USB Flash drive, install 9.2.1.7 to it and restore your config file, reboot. Things should work well and now you are running using an 8GB boot flash drive. Now do an upgrade to 9.3.

Hmm, thanks - I'll try that if the upgrade doesn't work from 9.2.1.9 to 9.3

Good Luck.

Thanks Joe!

 

[andyl]

You might have better luck updating to the 9.2.1.9 update before attempting the upgrade.

Thanks, Robert. I tried this, the 9.2.1.9 update installed fine, but the upgrade to 9.3 ended up with the same result.

Cheers,
Andy.

 

[andyl]

The easy way to do the upgrade would be to take your new USB Flash drive, install 9.2.1.7 to it and restore your config file, reboot. Things should work well and now you are running using an 8GB boot flash drive. Now do an upgrade to 9.3.

Thanks for the suggestion Joe, but this didn't work either :(

I built a 9.2.1.9 boot disk on one of the new 8Gb sticks using Win32DiskImager and FreeNAS-9.2.1.9-RELEASE-x64.img.xz (I unzipped the archive).

I booted off this and all was well, the web GUI and shares were available.

Then I ran the GUI upgrade against it. This proceeded for a good while, but then gave disk space errors and failed to find /var/log/messages.
Is it possible that the upgrade process does not increase the size of the 2Gb partition on the 8Gb disk? The install could then run out space, maybe?

Thanks for you assistance!

 

[dlavigne]

For your last attempt, I think you are hitting this:

If the burned image fails to boot and the image was burned using a Windows system, wipe the USB stick before trying a second burn using a utility such as Active@ KillDisk. Otherwise, the second burn attempt will fail as Windows does not understand the partition which was written from the image file. Be very careful that you specify the USB stick when using a wipe utility!

As for the "/var/log/messages could not be found" message, when you get back to 9.2.x, check System -> System Dataset to see what it is pointing to.

 

[andyl]

For your last attempt, I think you are hitting this:

If the burned image fails to boot and the image was burned using a Windows system, wipe the USB stick before trying a second burn using a utility such as Active@ KillDisk. Otherwise, the second burn attempt will fail as Windows does not understand the partition which was written from the image file. Be very careful that you specify the USB stick when using a wipe utility!

I think I got past this with diskpart/clean under Win7. I imaged the 8Gb USB stick with 9.2.1.9 and got it to boot okay. I did have problems until I performed the diskpart/clean - Win32DiskImager would claim to have imaged ok, but the system would fail during boot.

As for the "/var/log/messages could not be found" message, when you get back to 9.2.x, check System -> System Dataset to see what it is pointing to.

/var/log/messages ends up on the USB stick after install, doesn't it?
Or is it that the volume is failing to import somehow?

I already have 9.2.1.9 working again. I was able to put in the original 2Gb USB stick, and boot back into 9.2.1.9 - all is working as before. The system dataset appears to be set correctly - it refers to the correct (and only) volume.

According to diskpart, I have the following partitions on the 8Gb stick after the 9.2.1.9 -> 9.3 upgrade;

Code:

DISKPART> list partition


  Partition ###  Type              Size     Offset
  -------------  ----------------  -------  -------
  Partition 1    Unknown            512 KB    17 KB
  Partition 0    Unknown           7631 MB   532 KB

DISKPART>

GPartEd concurs; two partitions, one of 512Kb and one of 7.45Gb.

I had thought the disk may be partitioned too small for the installation to complete, but it doesn't look that way here...

So now I'm stumped. :(

Any assistance greatly appreciated.
Thanks dlavigne; please let me know if I've misunderstood anything you said :)

 

[joeschmuck]

Okay, here is something to try...

Which version of 9.3 are you trying to upgrade to? Consider using this one to start with...
http://download.freenas.org/9.3/STABLE/201503131809/x64/

Also, to remove your imaging process as part of the problem, consider using the ISO to create a bootable CD and go that route.

 

[joeschmuck]

Then I ran the GUI upgrade against it. This proceeded for a good while, but then gave disk space errors and failed to find /var/log/messages.
Is it possible that the upgrade process does not increase the size of the 2Gb partition on the 8Gb disk? The install could then run out space, maybe?

Out of curiosity, which version of 9.3 did you try to upgrade to? If you tried a current version, again, try the process again and use the version I specified above. You may need to step through a few upgrade steps to get there but once you get to any 9.3 that works, make another configuration backup. There was one version of 9.3 which failed to automatically update properly so turn off Automatic updates once you get the first version of 9.3 to work. Do this manually at least until you get to one dated in Aug 2015.

 

[andyl]

OK, I'll try that...

For info, I have been using;
FreeNAS-9.3-STABLE-201509022158.GUI_Upgrade.txz
FreeNAS-9.3-STABLE-201509022158.iso

Thanks again for the help :)

 

[joeschmuck]

I am not specifically aware of an upgrade path issue with 9.3 (I'm old and forget things) but we have seen it before several times so I'm hoping that is the issue to get you over this hump.

 

[Nick2253]

I recently went through this process, and here's how I did it, if it helps:

• 
  
  
  • Install same version of FreeNAS on larger USB stick.
  • Import config into FreeNAS on larger stick.
  • Run the 9.3 installer (I used the ISO with my IPMI interface, but you could use a USB CD drive or a thumb drive to boot into the installer. I happened to use the exact version joe recommends).
  • The installer prompted me to upgrade my system to 9.3. I let it go to town.
  • After all was said and done, I rebooted, and I was at 9.3.

I never bothered with the GUI upgrade. I just used the full installer, and that worked for me.

EDIT: NVM, I put on my reading comprehension hat, and I realize you've already gone through these steps.

 

[andyl]

EDIT: NVM, I put on my reading comprehension hat, and I realize you've already gone through these steps.

Lol, Thanks Nick.
I haven't tried the version that Joe recommends, so I'll try that one next....

 

[andyl]

So I tried upgrading a working 9.2.1.9 install on the new 8Gb USB stick to 9.3 using the FreeNAS-9.3-STABLE-201503131809.iso image mounted as an ISO through the server's iLO.
No good, I'm afraid...

As before, the 9.2.1.9 install works fine, I'm happy there's no issue with this build, but after the 9.3 install completes I get

I've included a couple of interesting errors that I got along the way...


The screenshot above ("Before-Reboot.png") occurred after the server had completed the install from booting off FreeNAS-9.3-STABLE-201503131809.iso, but before it had rebooted into the 9.3 install on the 8Gb USB.



The screenshot above ("After-Reboot.png") occurred after the server had rebooted into the new install, and is the typical error I receive.

As ever, thanks for the assistance, and if there's anything I've missed or misunderstood - please let me know.

Cheers,
Andy.

 

[joeschmuck]

Lets try this last one (I hope it's the last one)...

1) Grab the current FreeNAS 9.3.1 and use your magic to install it to one of your 8GB USB Flash drives.
2) Boot your system and answer all the questions, do not exit prematurely. Configure what you can.
3) Do Not import your configuration.
4) Reboot.

Does it work without error messages? I'm trying to establish if your hardware is working properly with FreeNAS 9.3.1, not if the importing of your 9.2.1.7 configuration file works correctly, I think we have established that it may have some corruption in it or something else in going on we haven't solved yet.

If you system operates without all those error messages, do not import your config file. I highly recommend you reconfigure from scratch, setting up shares and the like, ensuring things can work without issues. Also, do not upgrade your pools, ignore that warning for now or you will not be able to switch back to 9.2.1.x. If you have plugins then I'd take some notes on settings so you could recreate them again because you will likely need to recreate them under the new 9.3.1 version. It's not a perfect solution but it may be your only option if you want to upgrade to 9.3.1.

I wish I had other options to try but I think you have tried about everything.

Good Luck

 

[andyl]

I did as you suggest and got some strange results. It looks like the volume does not import properly.

Looking under Storage/Volumes/View Volumes after the intall, there was nothing present, but I could browse the volume via the console command prompt and df -kh gives me accurate info about the volume.

I rebooted, and the volume did not automatically import. Running a manual import appears to complete successfully, but there is still nothing in the View Volumes pane.

I cannot add shares, because the web GUI seems to think there is no volume to share.

I'm baffled. At this point, I think destroying the volume and restoring from backup may be my only option.

Thanks for your help.

 

[joeschmuck]

I wouldn't think you need to destroy the volume, but it you do, that might be the fastest recovery option for you. Did you use the Auto-Import feature, also are you using ZFS or are you using UFS for the file system? Can you post the output of "zpool status" and "zfs list". If you by chance are using UFS, well FreeNAS 9.3 no longer supports UFS file system so you would have to remain at 9.2.1.9 as the latest supported version for that file system.

 

[andyl]

The server is using ZFS.

I'm guessing I need to be running the broken 9.3 install when I run the commands you mention, and I'm currently booted into 9.2.1.9 and using the server.

Just in case, here are the command outputs run under 9.2.1.9;

Code:

[root@bitbucket] ~# zfs list
NAME                                                   USED  AVAIL  REFER  MOUNTPOINT
Vol1                                                  5.17T  18.9G  5.17T  /mnt/Vol1
Vol1/.system                                          48.1M  18.9G   279K  /mnt/Vol1/.system
Vol1/.system/cores                                    24.0M  18.9G  24.0M  /mnt/Vol1/.system/cores
Vol1/.system/rrd-277b94c280184a779514e6fd3f0cca44      209K  18.9G   209K  /mnt/Vol1/.system/rrd-277b94c280184a779514e6fd3f0cca44
Vol1/.system/rrd-587e094a2524450e802b560a401fce7b      209K  18.9G   209K  /mnt/Vol1/.system/rrd-587e094a2524450e802b560a401fce7b
Vol1/.system/samba4                                   4.91M  18.9G  4.91M  /mnt/Vol1/.system/samba4
Vol1/.system/syslog-277b94c280184a779514e6fd3f0cca44   570K  18.9G   570K  /mnt/Vol1/.system/syslog-277b94c280184a779514e6fd3f0cca44
Vol1/.system/syslog-587e094a2524450e802b560a401fce7b  18.0M  18.9G  18.0M  /mnt/Vol1/.system/syslog-587e094a2524450e802b560a401fce7b
[root@bitbucket] ~# zpool status
  pool: Vol1
state: ONLINE
  scan: scrub repaired 0 in 14h0m with 0 errors on Wed Sep 23 13:16:11 2015
config:

        NAME                                            STATE     READ WRITE CKSUM
        Vol1                                            ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            gptid/ba7a3326-3b58-11e2-ab99-e8393520a421  ONLINE       0     0     0
            gptid/baeba256-3b58-11e2-ab99-e8393520a421  ONLINE       0     0     0
            gptid/d798b345-4ca1-11e5-993a-e8393520a421  ONLINE       0     0     0
            gptid/67b3e427-7309-11e4-be82-e8393520a421  ONLINE       0     0     0

errors: No known data errors

I did do a "zpool status" on 9.3, after I'd re-imported the volume and it looked pretty normal to me.

I'll boot into 9.3 tomorrow and run the commands again....