Error upgrading from 11.2-U7 to 11.3-STABLE

[ghostwolf59]

I have now attempted to upgrade my 11.2-U7 system to 11.3-STABLE but upgrade more or less stalls with "CAM status: CCB request completed with an error" *just keep trying and trying.

Reverting back to 11.2-U7 I cant find any logs on the error, so this is as much as I can provide atm.

Reading some old posts the CCB error seem to point to a failing USB, but not sure if its just that.

The thumb drive I use is only about a week old *64GB Sandisk Ultra"

No issues booting into 11.2-U7 from that same stick - Anyone that could offer some advice on this *and/or where the logs from the failing 11.3 might have ended up on the system that would allow me to provide a more detailed message?

Should also me said that I were forced to CTRL+BREAK out from the stuck process so I could reboot the system again.

 

[SweetAndLow]

Sounds like a bad USB device. Can't handle writing the new image.

 

[ghostwolf59]

Sounds like a bad USB device. Can't handle writing the new image.

Will try a brand new stick to see if that work

 

[ghostwolf59]

Aaaarg!!! New stick - installed 11.3 fresh from iso - Installed, BUT(!) all jails gone = applied saved config = no change *all jails gone*
Then reverted back to a fresh 11.2.-U7 install - applied saved config = all my jails gone !!!!!

FFFFFFFFFFS!!!!

 

[Fredda]

I just read about this here in by user @tobtschik in the German subforum.
I seems the upgrade moves the config for iocage and the jails and you need to restore them.

For iocage

Code:

cd /mnt/<pool>/iocage/
mv defaults_backup.json defaults.json 

For the jails

Code:

cd /mnt/<pool>/iocage/jails/<jailname>
mv config_backup.json config.json 

 

[ghostwolf59]

I just read about this here in by user @tobtschik in the German subforum.
I seems the upgrade moves the config for iocage and the jails and you need to restore them.

For iocage

Code:

cd /mnt/<pool>/iocage/
mv defaults_backup.json defaults.json

For the jails

Code:

cd /mnt/<pool>/iocage/jails/<jailname>
mv config_backup.json config.json

And where would one stand in terms of saved config from 11.2-U7 in regards to this?
Assume you *like I just decided to do again* installs 11.3-STABLE as a fresh install - Last attempt scrambled the entire user interface once I tried to apply the 11.2-U7 saved config.
Is the backed up json all I need or should I still try to apply my old config?
...or is it a case where I once move back to 11.2-U7 and cant find any jails, that I need to move this backed up json across?

Read the upgrade notes and it was straight forward, but face value seem to be more and its creating a bit of a mess @ my corner right now *tossing with the idea of installing 11.3 and manually re-create all my jails from scratch *dont want to do that so....

 

[ghostwolf59]

Ok, 11.3 freshly installed and havent applied saved config - no pools, no jails... no nothing really
Should I try to apply the 11.2-U7 saved config or should I revert back to 11.2-U7 first?

 

[Fredda]

Ah, sorry, this was something I forgot to translate from the original post. The commands to restore the json files were done to bring back the jails after downgrading to 11.2-U7, not to get them running in 11.3.

 

[tobtschik]

I would install 11.2-U7, import your pool. Then roll back the iocage configs, activate your pool for iocage and you should be good to go. Everything should be stored in the iocage configs. From there on you could give it another try or wait for the developers to fix this problem. Is there any particular feature from 11.3 you want to use?

 

[ghostwolf59]

I would install 11.2-U7, import your pool. Then roll back the iocage configs, activate your pool for iocage and you should be good to go. Everything should be stored in the iocage configs. From there on you could give it another try or wait for the developers to fix this problem. Is there any particular feature from 11.3 you want to use?

Im a bit dúmb, so could you kindly outline command and location to get this done please!
Im back on 11.2-U7 now - nothing applied in terms of backed up config.... could you outline the steps I need to take to get it all back
Only thing that seem to be running now is my pools, ssh & smb shares - No jails *not even defined where the system asks me to create a iocage from scratch

 

[ghostwolf59]

Moved the the backed up json and all my jails reappeared - but none can be started - just applied my backed up config and system currently rebooting - hopefully it will resolve itself

 

[ghostwolf59]

Nop! nothing changed - jails appear, but throws errors and wont start !!!! - same error on all jails from what I can see


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 165, in call_method
result = await self.middleware.call_method(self, message)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1107, in call_method
return await self._call(message['method'], serviceobj, methodobj, params, app=app, io_thread=False)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1052, in _call
return await self._call_worker(name, *args)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1072, in _call_worker
return await self.run_in_proc(main_worker, name, args, job)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 995, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 971, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
RuntimeError:
Stopped plex due to VNET failure

 

[tobtschik]

I can only assume what’s going on there. The last line „VNET failure“ seems familiar. I think I had that once during my migration from warden to iocage. If I remember correctly, bridge0 was missing back then. I fixed it by setting up a tunable (which should not be needed). Infos can be found here: https://www.ixsystems.com/community/threads/iocage-vnet0-and-bridge0.59964/post-425631

Before setting up the tunables you might wanna ssh into your FreeNAS and check ifconfig.

Others had the exact same errors with wrong time and timezone settings (wtf?! see here https://redmine.ixsystems.com/issues/52548).

 

[ChiknNutz]

No help here but...UGH, gives me serious pause on moving to 11.3. I want to so I can get Sabnzbd via plugin again, but seems there are still a lot of folks having issues with 11.3. Especially when I see some folks that also have issues trying to move back to 11.2 after having issues with 11.3...not good.

 

[ghostwolf59]

This upgrade have turned into a major fuckup !!!

Decided to upgrade to 11.3 once I had a fresh install of 11.2-U7 with applied saved config - no jails running but at least shown
On completion of 11.3 upgrade all my jails have been tagged as CORRUPT
Unable to add static ip, network interface, gateway etc (save button disabled)
Drop down ipv4 interface show "-" and "N/A" (nothing else)
freebsd version is disabled and show "n/a"

Unable to upgrade the jail

Not sure if this has something to do with me manually upgrading these jails to freebsd 11.3-RELEASE-p5 while on 11.2-U7

One test jail I setup prior to upgrading to 11.3 still valid and points to freebsd 11.3-RELEASE-p6

 

[ghostwolf59]

A complete stuff up looking at the end result - now all my jails is unusable, resulting in that I went down deleting the entire iocage *taken some backups though* - but when trying to recreate a jail the system didnt even allow me to look for package updates *only looking for local* and even though I made sure it could see outside it still just went local looking for updates and cant connect to freebsd on the web.
Tried the same on root level and the same thing occurred - This update is nothing but a joke - now going back to 11.2-U7 and will stay there until I get some real confirmation on successful migration.
This process also killed two brand new thumb drives - One is perfectly fine under windows, and bios on the nas box sees it, but the update process dont *using 11.3

Makes me question the QA and level of testing before rolling this 11.3-STABLE release as "stable" - its far from stable @ my end I tell you!!!!

 

[tobtschik]

What I will take from all this: use a NAS as NAS and do Containers/VMs on another Host. Even when using it at home. I will migrate to Proxmox since most of the things I want to do are other than sharing files over SMB/NFS, so this will be much better suited for my needs.

 

[ghostwolf59]

So clean 11.2-U7 install - defined three jails with nothing in it other than creating it
Again applied 11.3-STABLE freenas upgrade. Booted and then decided to upgrade to 11.3 freebsb since the jail still sits on 11.2 and this following error is thrown - this is beyond a joke really !!!!

Very useful error message I say

 

[SweetAndLow]

So clean 11.2-U7 install - defined three jails with nothing in it other than creating it
Again applied 11.3-STABLE freenas upgrade. Booted and then decided to upgrade to 11.3 freebsb since the jail still sits on 11.2 and this following error is thrown - this is beyond a joke really !!!!

Very useful error message I say

View attachment 35480

Your jails are currently upgraded to the highest patch level for 11.2. There is no more upgrading them. If you saw the output before you got this error it tells you that. Yes this error is a bug but it's not hurting you in any way.

The GUI upgrade button runs iocage update cmd. If you want to upgrade your iocage jails to the next release you need to run iocage upgrade on the cli.

 

[ghostwolf59]

Your jails are currently upgraded to the highest patch level for 11.2. There is no more upgrading them. If you saw the output before you got this error it tells you that. Yes this error is a bug but it's not hurting you in any way.

The GUI upgrade button runs iocage update cmd. If you want to upgrade your iocage jails to the next release you need to run iocage upgrade on the cli.

"Not hurting you in any way"... - Absolutely true if you happy with jails tagged "CORRUPT" that wont start or do anything with!
As for my jails stuck on 11.2, this is incorrect - All my jails had been manually upgraded to 11.3-RELEASE-P5.

I ended up recreating all my jails once I updated my 11.2-U7 to 11-3. Gui on jail setup cant be used as it always creates errors where jail sometime starts, but shell spits the dummy complaining about root acccess *or lack thereof. If a jail start it cant be stopped using the gui.
Only way I found around these issues were to create a new iocage on another store, then force the deletion of the initial iocage pool, then revert back to create a brand new iocage on the store initially pointed to.
Then I found that previous jail processes from the deleted pool still were running resulting in further issues when creating jails on the newly created iocage.... Repeated the switch of iocage, delete the iocage pool, reboot the entire system then start over again.

Finally I have 90% of my jails up and running properly on the 11.3 upgrade - struggling with nextcloud but hopefully it will sort itself.