11.2-u1 Reporting: Temp Graphs Empty on 2 of 4 drives in pool

[181228]

I've got a fresh, bare-metal install of 11.2-u1. It seems to be working OK, with a few minor glitches.

When I look at 'Reporting > Disk > Disk Temperature', temperature graphs are being generated for ada0, ada3 but not ada1, or ada2.

When I perform 'smartctl -a /dev/adaX | grep Temp" for each of the drives, I see that all four drives in the pool are reporting temperatures via SMART, but FreeNAS is only reporting temps for two of the drives instead of for all four. The two drives that are reporting show the current temperature in the lower right hand corner of their plots, but the two drives that are not reporting show 'nan' instead of a temperature.

I understand that 'nan' indicates that the poll response was 'not a number', but I don't understand why the scripts are failing to poll the correct data when I'm capable of pulling it using SMART.

Ideas?

edit: all four devices are idential physical hard disks, aka "spinners". They're even the same brand and model.

 

[Constantin]

Are the drives that are not reporting perhaps SSDs?

I ask because the hard drive temp graph only works my spinners. Per the UI, the Intel SSDs don't report, the Supermicro SATADOM does.

EDITED to clarify where hard drive temps are being reported.

 

[181228]

They're spinners. What makes this most confusing is that they're all the same brand / model, yet only 2 of the 4 are being plotted correctly.

 

[181228]

example pics attached.

 

[Constantin]

That's really odd and I hope someone more knowledgable can chime in re: a potential solution. In the meantime, the best solution may be to file a bug report?

 

[181228]

https://redmine.ixsystems.com/issues/66448

 

[Constantin]

Good luck! I hope the devs can replicate the issue and squash the bug.

 

[pschatz100]

I'm running 11.2-RELEASE, and I get the temperature for ada3 only. The system reports "NA" for ada1, ada2, and ada4. I have three WD Reds and one HGST. The temp being reported is from one of the WD Reds.

 

[Constantin]

Please add your voice to the redmine ticket! The more data the devs have, the better!

 

[181228]

Now I'm seeing alerts about locked databases popping up in my email inbox. By any chance do you have a similar problem? I'm just wondering if this could be a contributing factor.

I had problems with database lockups when I tried to upgrade my fresh-install of 11.2-RELEASE to 11.2-U1. I built a brand new 11.2 box on bare metal, and immediately proceeded to update to 11.2-U1 before doing anything with it. The fresh install resulted in immediate database lockup errors.
See this thread:

https://forums.freenas.org/index.ph...d-after-11-2-release-upgrade-31f889bbf.72386/

Those db lockup errors went away with a reboot. So I started populating my zpool with data and noticed the defective temp graphs...

Now my inbox is telling me that I've got database lockups again; in other words, the system is defective to the point of not being trustworthy, as it can't even perform volume status checks:

New alerts: * Unable to run alert source 'VolumeStatus' Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params) File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 412, in execute execute = self.locked_retry(Database.Cursor.execute, query, params) File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 403, in locked_retry raise e File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 389, in locked_retry rv = method(self, *args, **kwargs) sqlite3.OperationalError: database is locked The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 358, in __run_source alerts = (await alert_source.check()) or [] File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/volume_status.py", line 15, in check for pool in await self.middleware.call("pool.query"): File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1113, in call return await self._call(name, serviceobj, methodobj, params, app=app, pipes=pipes, io_thread=True) File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1049, in _call return await methodobj(*args) File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 664, in nf return await f(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/middlewared/service.py", line 266, in query 'datastore.query', self._config.datastore, [], datastore_options File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1113, in call return await self._call(name, serviceobj, methodobj, params, app=app, pipes=pipes, io_thread=True) File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1063, in _call return await run_method(methodobj, *args) File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs)) File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 668, in nf return f(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 175, in query select=options.get('select'), File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 88, in __queryset_serialize for i in qs: File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 256, in __iter__ self._fetch_all() File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 1087, in _fetch_all self._result_cache = list(self.iterator()) File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 54, in __iter__ results = compiler.execute_sql() File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql cursor.execute(sql, params) File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params) File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 94, in __exit__ six.reraise(dj_exc_type, dj_exc_value, traceback) File "/usr/local/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise raise value.with_traceback(tb) File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params) File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 412, in execute execute = self.locked_retry(Database.Cursor.execute, query, params) File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 403, in locked_retry raise e File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 389, in locked_retry rv = method(self, *args, **kwargs) django.db.utils.OperationalError: database is locked Alerts: * Unable to run alert source 'VolumeStatus' Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params) File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 412, in execute execute = self.locked_retry(Database.Cursor.execute, query, params) File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 403, in locked_retry raise e File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 389, in locked_retry rv = method(self, *args, **kwargs) sqlite3.OperationalError: database is locked The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 358, in __run_source alerts = (await alert_source.check()) or [] File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/volume_status.py", line 15, in check for pool in await self.middleware.call("pool.query"): File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1113, in call return await self._call(name, serviceobj, methodobj, params, app=app, pipes=pipes, io_thread=True) File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1049, in _call return await methodobj(*args) File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 664, in nf return await f(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/middlewared/service.py", line 266, in query 'datastore.query', self._config.datastore, [], datastore_options File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1113, in call return await self._call(name, serviceobj, methodobj, params, app=app, pipes=pipes, io_thread=True) File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1063, in _call return await run_method(methodobj, *args) File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs)) File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 668, in nf return f(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 175, in query select=options.get('select'), File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 88, in __queryset_serialize for i in qs: File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 256, in __iter__ self._fetch_all() File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 1087, in _fetch_all self._result_cache = list(self.iterator()) File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 54, in __iter__ results = compiler.execute_sql() File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql cursor.execute(sql, params) File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params) File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 94, in __exit__ six.reraise(dj_exc_type, dj_exc_value, traceback) File "/usr/local/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise raise value.with_traceback(tb) File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params) File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 412, in execute execute = self.locked_retry(Database.Cursor.execute, query, params) File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 403, in locked_retry raise e File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 389, in locked_retry rv = method(self, *args, **kwargs) django.db.utils.OperationalError: database is locked


Gone alerts: * Unable to run alert source 'VolumeStatus' Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params) File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 412, in execute execute = self.locked_retry(Database.Cursor.execute, query, params) File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 403, in locked_retry raise e File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 389, in locked_retry rv = method(self, *args, **kwargs) sqlite3.OperationalError: database is locked The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 358, in __run_source alerts = (await alert_source.check()) or [] File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/volume_status.py", line 15, in check for pool in await self.middleware.call("pool.query"): File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1113, in call return await self._call(name, serviceobj, methodobj, params, app=app, pipes=pipes, io_thread=True) File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1049, in _call return await methodobj(*args) File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 664, in nf return await f(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/middlewared/service.py", line 266, in query 'datastore.query', self._config.datastore, [], datastore_options File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1113, in call return await self._call(name, serviceobj, methodobj, params, app=app, pipes=pipes, io_thread=True) File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1063, in _call return await run_method(methodobj, *args) File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs)) File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 668, in nf return f(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 175, in query select=options.get('select'), File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 88, in __queryset_serialize for i in qs: File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 256, in __iter__ self._fetch_all() File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 1087, in _fetch_all self._result_cache = list(self.iterator()) File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 54, in __iter__ results = compiler.execute_sql() File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql cursor.execute(sql, params) File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params) File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 94, in __exit__ six.reraise(dj_exc_type, dj_exc_value, traceback) File "/usr/local/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise raise value.with_traceback(tb) File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params) File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 412, in execute execute = self.locked_retry(Database.Cursor.execute, query, params) File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 403, in locked_retry raise e File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 389, in locked_retry rv = method(self, *args, **kwargs) django.db.utils.OperationalError: database is locked

http://redmine.ixsystems.com/issues/66459

I hate to admit it, but 11.2 is just not trustworthy and I'm going to rebuild the box with something else. I've got decades of accumulated data that I can't trust to such a bug ridden platform. It's unfortunate, but 11.2 is just not ready for deployment and I can't expend any more time on a package that should still be in beta.

 

[181228]

Please add your voice to the redmine ticket! The more data the devs have, the better!

i agree. developers place more faith in bugs that have been verified as applying to many people. they tend to blow off things that they think are a fluke.

 

[Apollo]

Are the drives that are not reporting perhaps SSDs?

I ask because the hard drive temp graph only works my spinners. The SSDs don't report, the Supermicro SATADOM does.

Not true.
SSD's do report temperature as well.

 

[Constantin]

Not true.
SSD's do report temperature as well.

See below. This is my experience with the UI. I'm glad yours is different. Note: I'm not asserting that the drives aren't reporting temperatures, I'm merely pointing out how the UI is not reporting them.

My L2ARC drive is reporting in the UI (Samsung EVO 840 series)

...while the two SLOGs (S3710 from Intel) do not.

The SATADOM also reports elsewhere.

 

[inis]

I created a new issue for this. Same issue persist in 11.2 u2

https://redmine.ixsystems.com/issues/77079