Daily security run output email since upgrading to Freenas 11.1

[Joe Fenton]

Hi all,

I've been receiving the following information in daily security run emails from my FreeNAS box since I recently upgraded to 11.1.

I've looked around and people have suggested that they mean the HDDs are failing, but I have SMART self testing turned on, and have received no other warnings. There are also no critical or other warnings through FreeNAS warning me of drive failure, which I've had in the past.
The message seems to come from every drive, ada0, ada1, ada2, ada3, which sounds suspect that they would all have problems overnight, unless of course previously some messages were suppressed.
I haven't found any obvious conclusion as to why they might be happening. Other posts have suggested that it was a bug, but that it should be fixed by now in FreeNAS.

My OS version is FreeNAS-11.2-U4, it is a Intel(R) Atom(TM) CPU C2550 @ 2.40GHz (4 cores) and 32GB RAM.

Anyone got any ideas?

Code:

freenas.local kernel log messages:
> ahcich2: Timeout on slot 10 port 0
> ahcich2: is 00000008 cs 00000000 ss 00000000 rs 00000400 tfd 40 serr 00000000 cmd 10008a17
> (ada0:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 30 5a 59 40 c4 00 00 00 00 00
> (ada0:ahcich2:0:0:0): CAM status: Command timeout
> (ada0:ahcich2:0:0:0): Retrying command
> ahcich2: Timeout on slot 23 port 0
> ahcich2: is 00000008 cs 00000000 ss 00000000 rs 00800000 tfd 40 serr 00000000 cmd 10009717
> (ada0:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 f0 76 59 40 c4 00 00 00 00 00
> (ada0:ahcich2:0:0:0): CAM status: Command timeout
> (ada0:ahcich2:0:0:0): Retrying command
> ahcich4: Timeout on slot 19 port 0
> ahcich4: is 00000008 cs 00000000 ss 00000000 rs 00080000 tfd 40 serr 00000000 cmd 10009317
> (ada2:ahcich4:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 50 d4 4a 40 25 00 00 00 00 00
> (ada2:ahcich4:0:0:0): CAM status: Command timeout
> (ada2:ahcich4:0:0:0): Retrying command
> ahcich3: Timeout on slot 29 port 0
> ahcich3: is 00000008 cs 00000000 ss 00000000 rs 20000000 tfd 40 serr 00000000 cmd 10009d17
> (ada1:ahcich3:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 38 60 18 40 cf 00 00 00 00 00
> (ada1:ahcich3:0:0:0): CAM status: Command timeout
> (ada1:ahcich3:0:0:0): Retrying command
> ahcich5: Timeout on slot 17 port 0
> ahcich5: is 00000008 cs 00000000 ss 00000000 rs 00020000 tfd 40 serr 00000000 cmd 10009117
> (ada3:ahcich5:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 90 f1 4a 40 25 00 00 00 00 00
> (ada3:ahcich5:0:0:0): CAM status: Command timeout
> (ada3:ahcich5:0:0:0): Retrying command
> ahcich2: Timeout on slot 31 port 0
> ahcich2: is 00000008 cs 00000000 ss 00000000 rs 80000000 tfd 40 serr 00000000 cmd 10009f17
> (ada0:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 30 a1 59 40 c4 00 00 00 00 00
> (ada0:ahcich2:0:0:0): CAM status: Command timeout
> (ada0:ahcich2:0:0:0): Retrying command
> ahcich3: Timeout on slot 26 port 0
> ahcich3: is 00000008 cs 00000000 ss 00000000 rs 04000000 tfd 40 serr 00000000 cmd 10009a17
> (ada1:ahcich3:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 88 c0 59 40 c4 00 00 00 00 00
> (ada1:ahcich3:0:0:0): CAM status: Command timeout
> (ada1:ahcich3:0:0:0): Retrying command
> ahcich3: Timeout on slot 16 port 0
> ahcich3: is 00000008 cs 00000000 ss 00000000 rs 00010000 tfd 40 serr 00000000 cmd 10009017
> (ada1:ahcich3:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 90 cb 59 40 c4 00 00 00 00 00
> (ada1:ahcich3:0:0:0): CAM status: Command timeout
> (ada1:ahcich3:0:0:0): Retrying command
> ahcich5: Timeout on slot 28 port 0
> ahcich5: is 00000008 cs 00000000 ss 00000000 rs 10000000 tfd 40 serr 00000000 cmd 10009c17
> (ada3:ahcich5:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 78 6d 4b 40 25 00 00 00 00 00
> (ada3:ahcich5:0:0:0): CAM status: Command timeout
> (ada3:ahcich5:0:0:0): Retrying command
> ahcich2: Timeout on slot 20 port 0
> ahcich2: is 00000008 cs 00000000 ss 00000000 rs 00100000 tfd 40 serr 00000000 cmd 10009417
> (ada0:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 b0 bd 5a 40 c4 00 00 00 00 00
> (ada0:ahcich2:0:0:0): CAM status: Command timeout
> (ada0:ahcich2:0:0:0): Retrying command

 

[Chris Moore]

Did you look at the detailed output of your drive SMART status report?

command: smartctl -x /dev/da4

Should get results that look like this:

Code:

smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Desktop HDD.15
Device Model:     ST4000DM000-1F2168
Serial Number:    xxxxxxxx
LU WWN Device Id: 5 000c50 09221606a
Firmware Version: CC54
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5900 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Jun 17 16:38:32 2019 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Disabled
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Unavailable

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  107) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 506) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x1085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR--   115   099   006    -    99032192
  3 Spin_Up_Time            PO----   091   091   000    -    0
  4 Start_Stop_Count        -O--CK   100   100   020    -    102
  5 Reallocated_Sector_Ct   PO--CK   100   100   010    -    0
  7 Seek_Error_Rate         POSR--   077   060   030    -    54265527
  9 Power_On_Hours          -O--CK   078   078   000    -    19371
10 Spin_Retry_Count        PO--C-   100   100   097    -    0
12 Power_Cycle_Count       -O--CK   100   100   020    -    101
183 Runtime_Bad_Block       -O--CK   100   100   000    -    0
184 End-to-End_Error        -O--CK   100   100   099    -    0
187 Reported_Uncorrect      -O--CK   100   100   000    -    0
188 Command_Timeout         -O--CK   100   100   000    -    0 0 0
189 High_Fly_Writes         -O-RCK   100   100   000    -    0
190 Airflow_Temperature_Cel -O---K   064   052   045    -    36 (Min/Max 35/45)
191 G-Sense_Error_Rate      -O--CK   100   100   000    -    0
192 Power-Off_Retract_Count -O--CK   100   100   000    -    69
193 Load_Cycle_Count        -O--CK   083   083   000    -    35751
194 Temperature_Celsius     -O---K   036   048   000    -    36 (0 13 0 0 0)
197 Current_Pending_Sector  -O--C-   100   100   000    -    0
198 Offline_Uncorrectable   ----C-   100   100   000    -    0
199 UDMA_CRC_Error_Count    -OSRCK   200   200   000    -    0
240 Head_Flying_Hours       ------   100   253   000    -    16051h+52m+21.178s
241 Total_LBAs_Written      ------   100   253   000    -    53449642764
242 Total_LBAs_Read         ------   100   253   000    -    1417092037891
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      5  Ext. Comprehensive SMART error log
0x04       GPL,SL  R/O      8  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x24       GPL     R/O   1223  Current Device Internal Status Data log
0x25       GPL     R/O   1223  Saved Device Internal Status Data log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa1       GPL,SL  VS      20  Device vendor specific log
0xa2       GPL     VS    4496  Device vendor specific log
0xa8       GPL,SL  VS     129  Device vendor specific log
0xa9       GPL,SL  VS       1  Device vendor specific log
0xab       GPL     VS       1  Device vendor specific log
0xb0       GPL     VS    5176  Device vendor specific log
0xbe-0xbf  GPL     VS   65535  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL,SL  VS      10  Device vendor specific log
0xc3       GPL,SL  VS       8  Device vendor specific log
0xc4       GPL,SL  VS       5  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (5 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     19355         -
# 2  Short offline       Completed without error       00%     19331         -
# 3  Short offline       Completed without error       00%     19307         -
# 4  Extended offline    Completed without error       00%     19275         -
# 5  Short offline       Completed without error       00%     19259         -
# 6  Extended offline    Completed without error       00%     19227         -
# 7  Short offline       Completed without error       00%     19211         -
# 8  Short offline       Completed without error       00%     19187         -
# 9  Short offline       Completed without error       00%     19163         -
#10  Short offline       Completed without error       00%     19139         -
#11  Extended offline    Completed without error       00%     19107         -
#12  Short offline       Completed without error       00%     19091         -
#13  Extended offline    Completed without error       00%     19059         -
#14  Short offline       Completed without error       00%     19043         -
#15  Short offline       Completed without error       00%     19019         -
#16  Short offline       Completed without error       00%     18995         -
#17  Short offline       Completed without error       00%     18971         -
#18  Extended offline    Completed without error       00%     18939         -
#19  Short offline       Completed without error       00%     18923         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       522 (0x020a)
SCT Support Level:                   1
Device State:                        Active (0)
Current Temperature:                    36 Celsius
Power Cycle Min/Max Temperature:     35/45 Celsius
Lifetime    Min/Max Temperature:     13/48 Celsius
Under/Over Temperature Limit Count:   0/0

SCT Data Table command not supported

SCT Error Recovery Control command not supported

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 2) ==
0x01  0x008  4             101  ---  Lifetime Power-On Resets
0x01  0x010  4           19371  ---  Power-on Hours
0x01  0x018  6     53743102015  ---  Logical Sectors Written
0x01  0x020  6       515004590  ---  Number of Write Commands
0x01  0x028  6    168297044561  ---  Logical Sectors Read
0x01  0x030  6      1112770269  ---  Number of Read Commands
0x01  0x038  6               -  ---  Date and Time TimeStamp
0x03  =====  =               =  ===  == Rotating Media Statistics (rev 1) ==
0x03  0x008  4           19371  ---  Spindle Motor Power-on Hours
0x03  0x010  4           10874  ---  Head Flying Hours
0x03  0x018  4           35751  ---  Head Load Events
0x03  0x020  4               0  ---  Number of Reallocated Logical Sectors
0x03  0x028  4               0  ---  Read Recovery Attempts
0x03  0x030  4               0  ---  Number of Mechanical Start Failures
0x04  =====  =               =  ===  == General Errors Statistics (rev 1) ==
0x04  0x008  4               0  ---  Number of Reported Uncorrectable Errors
0x04  0x010  4               0  ---  Resets Between Cmd Acceptance and Completion
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

Pending Defects log (GP Log 0x0c) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x000a  2            1  Device-to-host register FISes sent due to a COMRESET
0x0001  2            0  Command failed due to ICRC error
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS

 

[Joe Fenton]

Thanks Chris,

I got the following results for one of my drives, the others seem similar. Seem okay to me on a quick look over? Should I be running occasional extended tests?

Code:

smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD40EFRX-68WT0N0
Serial Number:    WD-WCC4E1TJ5VJS
LU WWN Device Id: 5 0014ee 20df48bc4
Firmware Version: 82.00A82
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Jun 19 20:54:12 2019 BST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (51840) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off supp                            ort.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 518) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x703d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    0
  3 Spin_Up_Time            POS--K   183   172   021    -    7841
  4 Start_Stop_Count        -O--CK   100   100   000    -    37
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   200   200   000    -    0
  9 Power_On_Hours          -O--CK   074   074   000    -    19669
 10 Spin_Retry_Count        -O--CK   100   253   000    -    0
 11 Calibration_Retry_Count -O--CK   100   253   000    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    37
192 Power-Off_Retract_Count -O--CK   200   200   000    -    18
193 Load_Cycle_Count        -O--CK   200   200   000    -    541
194 Temperature_Celsius     -O---K   113   108   000    -    39
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   200   000    -    0
198 Offline_Uncorrectable   ----CK   100   253   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    0
200 Multi_Zone_Error_Rate   ---R--   100   253   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      6  Ext. Comprehensive SMART error log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa0-0xa7  GPL,SL  VS      16  Device vendor specific log
0xa8-0xb6  GPL,SL  VS       1  Device vendor specific log
0xb7       GPL,SL  VS      39  Device vendor specific log
0xbd       GPL,SL  VS       1  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL     VS      93  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA                            _of_first_error
# 1  Short offline       Completed without error       00%     19653         -
# 2  Short offline       Completed without error       00%     19624         -
# 3  Short offline       Completed without error       00%     19605         -
# 4  Short offline       Completed without error       00%     19599         -
# 5  Short offline       Completed without error       00%     19581         -
# 6  Short offline       Completed without error       00%     19551         -
# 7  Short offline       Completed without error       00%     19533         -
# 8  Short offline       Completed without error       00%     19503         -
# 9  Short offline       Completed without error       00%     19485         -
#10  Short offline       Completed without error       00%     19455         -
#11  Short offline       Completed without error       00%     19437         -
#12  Short offline       Completed without error       00%     19431         -
#13  Short offline       Completed without error       00%     19414         -
#14  Short offline       Completed without error       00%     19384         -
#15  Short offline       Completed without error       00%     19366         -
#16  Short offline       Completed without error       00%     19336         -
#17  Short offline       Completed without error       00%     19318         -
#18  Short offline       Completed without error       00%     19288         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       258 (0x0102)
SCT Support Level:                   1
Device State:                        Active (0)
Current Temperature:                    39 Celsius
Power Cycle Min/Max Temperature:     23/43 Celsius
Lifetime    Min/Max Temperature:     15/44 Celsius
Under/Over Temperature Limit Count:   0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (137)

Index    Estimated Time   Temperature Celsius
 138    2019-06-19 12:57    39  ********************
 ...    ..(121 skipped).    ..  ********************
 260    2019-06-19 14:59    39  ********************
 261    2019-06-19 15:00    40  *********************
 ...    ..(251 skipped).    ..  *********************
  35    2019-06-19 19:12    40  *********************
  36    2019-06-19 19:13    39  ********************
 ...    ..(100 skipped).    ..  ********************
 137    2019-06-19 20:54    39  ********************

SCT Error Recovery Control:
           Read:     70 (7.0 seconds)
          Write:     70 (7.0 seconds)

Device Statistics (GP/SMART Log 0x04) not supported

Pending Defects log (GP Log 0x0c) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2           70  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2           70  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000f  2            0  R_ERR response for host-to-device data FIS, CRC
0x0012  2            0  R_ERR response for host-to-device non-data FIS, CRC
0x8000  4      1409276  Vendor specific

 

[Chris Moore]

Should I be running occasional extended tests?

Yes. I do weekly long SMART tests and I get reports sent to me using the scripts here:

GitHub repository for FreeNAS scripts, including disk burnin
https://www.ixsystems.com/community...for-freenas-scripts-including-disk-burnin.28/
-
Original @Bidule0hm - Scripts to report SMART, ZPool and UPS status, HDD/CPU T°, HDD identification and backup the config
https://www.ixsystems.com/community/threads/scripts-to-report-smart-zpool-and-ups-status-hdd-cpu-t°-hdd-identification-and-backup-the-config.27365/

 

[Chris Moore]

Seem okay to me on a quick look over?

I would suggest manually running a long test on all your drives, then look at these results.

 

[Joe Fenton]

I will do that. My system is set up to mirror data, so the chance of data loss is quite low, is that right?

 

[Joe Fenton]

I finally got around to looking into the extended test results.

Code:

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD40EFRX-68N32N0
Serial Number:    WD-WCC7K7DYT49D
LU WWN Device Id: 5 0014ee 20f7a279c
Firmware Version: 82.00A82
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Jul  6 09:03:27 2019 BST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (43200) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off supp                                                                                                                                                             ort.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 458) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x303d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_                                                                                                                                                             FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -                                                                                                                                                                    0
  3 Spin_Up_Time            0x0027   183   177   021    Pre-fail  Always       -                                                                                                                                                                    5808
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -                                                                                                                                                                    15
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -                                                                                                                                                                    0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -                                                                                                                                                                    0
  9 Power_On_Hours          0x0032   085   085   000    Old_age   Always       -                                                                                                                                                                    11586
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -                                                                                                                                                                    0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -                                                                                                                                                                    0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -                                                                                                                                                                    15
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -                                                                                                                                                                    7
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -                                                                                                                                                                    28
194 Temperature_Celsius     0x0022   112   103   000    Old_age   Always       -                                                                                                                                                                    38
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -                                                                                                                                                                    0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -                                                                                                                                                                    0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -                                                                                                                                                                    0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -                                                                                                                                                                    0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -                                                                                                                                                                    0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA                                                                                                                                                             _of_first_error
# 1  Extended offline    Completed without error       00%     11213         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

From the original post:

Code:

> (ada0:ahcich2:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 30 5a 59 40 c4 00 00 00 00 00
> (ada0:ahcich2:0:0:0): CAM status: Command timeout
> (ada0:ahcich2:0:0:0): Retrying command

It's a similar result on the other drives. I still feel unsure about why I'm getting these security emails that seem to suggest timeouts on all my freenas hard disks, but the extended smart reports the drives to be okay, as does the OS?

 

[Joe Fenton]

I was wondering if anybody could shed any further light on the security emails that I'm getting daily, with these CAM status: Command timeouts appearing to come from all my drives. The SMART tests seem to always report no error. The daily emails I think started up when I upgraded from Freenas 11 to 11.1, or it could have been 11.1 to 11.2. It's been going on for a while now, I just don't see the point of them if a) there is no security problem, and b) there is no disk problem.

I'm running the script recommended above for emailing the SMART results, and I've also got questions around that

Code:

########## SMART status report summary for all drives on server FREENAS ##########

+------+------------------------+----+-----+-----+-----+-------+-------+--------+------+---------+------+-------+----+
|Device|Serial                  |Temp|Power|Start|Spin |ReAlloc|Current|Offline |Seek  |Total    |High  |Command|Last|
|      |Number                  |    |On   |Stop |Retry|Sectors|Pending|Uncorrec|Errors|Seeks    |Fly   |Timeout|Test|
|      |                        |    |Hours|Count|Count|       |Sectors|Sectors |      |         |Writes|Count  |Age |
+------+------------------------+----+-----+-----+-----+-------+-------+--------+------+---------+------+-------+----+
|ada0 ?|WD-WCC7K7DYT49D         | 36 |12584|   15|    0|      0|      0|       0|   N/A|      N/A|   N/A|    N/A| 27*|
|ada1  |WD-WCC4E1TJ5VJS         | 39 |21063|   37|    0|      0|      0|       0|   N/A|      N/A|   N/A|    N/A|   0|
|ada2 ?|WD-WCC4E5PYKA2J         | 41* |31245|   87|    0|      0|      0|       0|   N/A|      N/A|   N/A|    N/A| 27*|
|ada3 ?|WD-WCC4E7HHEJHH         | 39 |31246|   85|    0|      0|      0|       0|   N/A|      N/A|   N/A|    N/A| 27*|
+------+------------------------+----+-----+-----+-----+-------+-------+--------+------+---------+------+-------+----+

Why are 3 of the drives device number followed by a '?' and the same drives also have a * after 27 for last test. ada2 also has a * by the temp, being 41, is that because it's a little high?