FreeNAS 9.3 - Update through WebUI fails

Status
Not open for further replies.

Hi-Liter

Explorer
Joined
Oct 6, 2011
Messages
83
Hi,
I was trying to update my FreeNAS 9.3 (FreeNAS-9.3-STABLE-201501090144 - that's what it says now in system information) using the new update functionality in the WebUI. It performed the update, but when the machine rebooted, I noticed that the update apparently failed and a file /data/update.failed was created. These are the contents of the file:

Running migrations for api:
- Nothing to migrate.
- Loading initial data for api.
Installed 0 object(s) from 0 fixture(s)
Running migrations for freeadmin:
- Nothing to migrate.
- Loading initial data for freeadmin.
Installed 0 object(s) from 0 fixture(s)
Running migrations for support:
- Nothing to migrate.
- Loading initial data for support.
Installed 0 object(s) from 0 fixture(s)
Running migrations for directoryservice:
- Nothing to migrate.
- Loading initial data for directoryservice.
Installed 0 object(s) from 0 fixture(s)
Running migrations for services:
- Migrating forwards to 0157_auto__del_field_iscsitarget_iscsi_target_logical_blocksize.
> services:0156_auto__add_field_iscsitargetextent_iscsi_target_extent_blocksize__add_f
! Error found during real run of migration! Aborting.

! Since you have a database that does not support running
! schema-altering statements in transactions, we have had
! to leave it in an interim state between migrations.

! You *might* be able to recover with:
! The South developers regret this has happened, and would
! like to gently persuade you to consider a slightly
! easier-to-deal-with DBMS (one that supports DDL transactions)
! NOTE: The error which caused the migration to fail is further up.
Error in migration: services:0156_auto__add_field_iscsitargetextent_iscsi_target_extent_blocksize__add_f
Traceback (most recent call last):
File "/usr/local/www/freenasUI/manage.py", line 42, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.7/site-packages/south/management/commands/migrate.py", line 111, in handle
ignore_ghosts = ignore_ghosts,
File "/usr/local/lib/python2.7/site-packages/south/migration/__init__.py", line 220, in migrate_app
success = migrator.migrate_many(target, workplan, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 256, in migrate_many
result = migrator.__class__.migrate_many(migrator, target, migrations, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 331, in migrate_many
result = self.migrate(migration, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 133, in migrate
result = self.run(migration, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 114, in run
return self.run_migration(migration, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 84, in run_migration
migration_function()
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 60, in <lambda>
return (lambda: direction(orm))
File "/usr/local/www/freenasUI/../freenasUI/services/migrations/0156_auto__add_field_iscsitargetextent_iscsi_target_extent_blocksize__add_f.py", line 22, in forwards
for target in orm['services.iSCSITarget'].objects.all():
File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 96, in __iter__
self._fetch_all()
File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 857, in _fetch_all
self._result_cache = list(self.iterator())
File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 220, in iterator
for row in compiler.results_iter():
File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 713, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 452, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such column: services_iscsitarget.iscsi_target_logical_blocksize

When I go to the "Update" section of the system settings now, it says there are no updates. Apparently, the database could not be migrated - and now I think I am left with an inconsistent DB state. Anyone knows what to do?

Another thing: If I wasn't present when the system rebooted, I wouldn't even know about the update failure. The WebUI doesn't show any errors (I would have expected to see something in the "Alerts" in the top right corner).
 
D

dlavigne

Guest
Please create a bug report at bugs.freenas.org and post the issue number here.

Regarding the UI not showing an alert due to a failed update, please create a separate feature request at bugs.freenas.org and post that issue number here as well. Thanks!
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Do you have at least 8GB of RAM? I've seen the kind of failure you are experiencing first-hand. It happens when the update script aborts inappropriately (in the two cases I saw it was failing due to insufficient RAM).
 

Hi-Liter

Explorer
Joined
Oct 6, 2011
Messages
83
Yes I do, my System:
Supermicro X10SLH-F
Intel Xeon E3-1220 v3
2x Samsung 8GB DDR3-1600 UDIMM (ECC)
6x WD Red 3TB
 
Last edited:

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
What's your boot device? What happens if you try to verify the install or do a scrub of the boot device? Do you get any errors?
 

Hi-Liter

Explorer
Joined
Oct 6, 2011
Messages
83
My boot device currently is a small external (USB) SSD drive (I plan on going back to a USB flash drive, my previous one failed and I only had the SSD drive), I just installed the system a few days ago, didn't do a scrub yet.
"Verify install" reports that everything is ok.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I wouldn't be surprised if there's some spooky magic going on with your external USB SSD. The bridges that some use make for weird problems and such. I'd just keep things like they are and when you get your replacement flash drive, do a config restore and see what happens.
 

Hi-Liter

Explorer
Joined
Oct 6, 2011
Messages
83
Ok, will do!
One more question (don't know if this is a bug): I have checked the automatic update checkbox, but the list of updates to be performed never shows any updates, I just see them if I do a manual check. Is this intended (e.g. only major version updates are shown in the list of updates to perform)?
 

xnaron

Explorer
Joined
Dec 11, 2014
Messages
98
I had the same issue happen when I upgraded

Boot device is a usb 16GB flash drive

System is Supermicro x10sl7-f with 16GB ram and xeon cpu

I performed a verify install and it said "All files verified successfully!"

Code:
Installed 0 object(s) from 0 fixture(s)
Running migrations for services:
- Migrating forwards to 0157_auto__del_field_iscsitarget_iscsi_target_logical_blocksize.
> services:0156_auto__add_field_iscsitargetextent_iscsi_target_extent_blocksize__add_f
! Error found during real run of migration! Aborting.

! Since you have a database that does not support running
! schema-altering statements in transactions, we have had
! to leave it in an interim state between migrations.

! You *might* be able to recover with:
! The South developers regret this has happened, and would
! like to gently persuade you to consider a slightly
! easier-to-deal-with DBMS (one that supports DDL transactions)
! NOTE: The error which caused the migration to fail is further up.
Error in migration: services:0156_auto__add_field_iscsitargetextent_iscsi_target_extent_blocksize__add_f
Traceback (most recent call last):
  File "/usr/local/www/freenasUI/manage.py", line 42, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
 
Joined
Jan 15, 2015
Messages
25
I'm receiving this error message when I try to Apply Pending Updates.... I tried three different usb stick (made from scratch) and tried in a Virtual Box Environment for confirm the problem, and two different machines.
I installed FreeNAS-9.3-STABLE-201501150211


Jan 15 16:51:43 freenas updated.py: [freenasOS.Manifest:349] Signature check succeeded
Jan 15 16:51:43 freenas updated.py: [freenasOS.Update:641] Upgrade package freenas-pkg-tools-9.3-STABLE-10199d3 to freenas-pkg-tools-9.3-STABLE-82d7111
Jan 15 16:51:43 freenas updated.py: [freenasOS.Update:38] RunCommand(/usr/local/sbin/beadm, ['create', u'FreeNAS-9.3-STABLE-201501150211'])
Jan 15 16:51:45 freenas updated.py: [freenasOS.Update:38] RunCommand(/sbin/zfs, ['snapshot', u'freenas-boot/grub@Pre-Upgrade-FreeNAS-9.3-STABLE-201501150211'])
Jan 15 16:51:46 freenas updated.py: [freenasOS.Update:111] Unable to create grub snapshot Pre-Upgrade-FreeNAS-9.3-STABLE-201501150211
Jan 15 16:51:46 freenas updated.py: [freenasOS.Update:38] RunCommand(/usr/local/sbin/beadm, ['mount', u'FreeNAS-9.3-STABLE-201501150211', '/tmp/tmpvV2WT0'])
Jan 15 16:51:46 freenas updated.py: [freenasOS.Update:38] RunCommand(/sbin/umount, ['mount', u'FreeNAS-9.3-STABLE-201501150211', '/tmp/tmpvV2WT0'])
Jan 15 16:51:46 freenas updated.py: [freenasOS.Update:38] RunCommand(/usr/local/sbin/beadm, ['unmount', '-f', u'FreeNAS-9.3-STABLE-201501150211'])
Jan 15 16:51:46 freenas updated.py: [freenasOS.Update:38] RunCommand(/usr/local/sbin/beadm, ['destroy', '-F', u'FreeNAS-9.3-STABLE-201501150211'])
Jan 15 16:51:48 freenas updated.py: [freenasOS.Update:38] RunCommand(/usr/local/sbin/beadm, ['destroy', '-F', u'FreeNAS-9.3-STABLE-201501150211'])
Jan 15 16:51:48 freenas updated.py: [tools.updated:167] Exception on update
Jan 15 16:51:48 freenas updated.py: [tools.updated:12] Traceback (most recent call last):
Jan 15 16:51:48 freenas updated.py: [tools.updated:12] File "/usr/local/www/freenasUI/tools/updated.py", line 165, in <module>
Jan 15 16:51:48 freenas updated.py: [tools.updated:12] main(handler, args)
Jan 15 16:51:48 freenas updated.py: [tools.updated:12] File "/usr/local/www/freenasUI/tools/updated.py", line 123, in main
Jan 15 16:51:48 freenas updated.py: [tools.updated:12] install_handler=handler.install_handler,
Jan 15 16:51:48 freenas updated.py: [tools.updated:12] File "/usr/local/lib/freenasOS/Update.py", line 746, in ApplyUpdate
Jan 15 16:51:48 freenas updated.py: [tools.updated:12] raise e
Jan 15 16:51:48 freenas updated.py: [tools.updated:12] Exception: Unable to mount boot-environment FreeNAS-9.3-STABLE-201501150211
Jan 15 16:51:48 freenas updated.py: [tools.updated:173] Exiting daemon process
 

DrumNBisco

Dabbler
Joined
Nov 5, 2013
Messages
15
Just did a fresh install of 9.2.1.9 on a 8GB flash drive, imported my previous config and upgraded to 9.3 yesterday.

I was getting a similar error when I was trying to update via the GUI and had to edit the Update.py file to fix it.

Code:
Jan 15 14:39:44 freenas updated.py: [freenasOS.Update:699] Unable to mount boot-environment FreeNAS-9.3-STABLE-201501151844
Jan 15 14:40:18 freenas updated.py: [freenasOS.Update:741] Update got exception during update: Unable to mount boot-environment FreeNAS-9.3-STABLE-201501151844
Jan 15 14:40:19 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Unable to mount boot-environment FreeNAS-9.3-STABLE-201501151844]


I found bug 7517 with a syntax error in update.py
https://bugs.freenas.org/issues/7517#change-34024

Per the bug report, I edited /usr/local/lib/freenasOS/Update.py and changed "arts" to "args". I don't remember the specific line, but just search for "arts" and you'll find it.

After I saved the file, I ran the GUI update again and it was successful. If anyone needs me to post any additional logs let me know, fortunately I did not get error Bug #7521: Update process gets EOPNOTSUPP in mounted /dev added

System specs are
Supermicro X10SLL-F-O
Intel Xeon E3-1220V3 Haswell @ 3.1 Ghz
24 GBs ECC Ram
4 Toshiba 2TB 7200 RPM 64MB cache hardrives in RAIDZ2
 

Joe-freenas

Dabbler
Joined
Jan 15, 2015
Messages
16
Previous to latest update introduced a bug which makes new updates fail.
They already put a patch in the latest nightly build, but if updates cannot be appplied....what to do!?
Thanks to ZFS, the solution is easy...simply reboot and choose the previous boot environment at boot time and then simply apply latest update and presto!
Thanks ZFS, thanks FreeNAS team and you are welcome! ;-)
 

DrumNBisco

Dabbler
Joined
Nov 5, 2013
Messages
15
True, however that only works if you were on the previous build in the first place....
 

Joe-freenas

Dabbler
Joined
Jan 15, 2015
Messages
16
True, however that only works if you were on the previous build in the first place....

Yep, only people that update often or updated to that version will see the issue. Once the patch is out, everybody will bypass the 'bad' version and update properly.

Sometimes it does suck to be on the edge eh...
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Yep.. I got boned with this bug on my FreeNAS Mini. :P

I haven't had time to fix it yet.. probably tomorrow or the weekend.
 
Status
Not open for further replies.
Top