~/ $ zpool status pool: freenas-boot state: ONLINE scan: scrub repaired 0 in 0 days 00:05:53 with 0 errors on Tue Apr 9 03:50:53 2019 config: NAME STATE READ WRITE CKSUM freenas-boot ONLINE 0 0 0 da0p2 ONLINE 0 0 0 errors: No known data errors pool: pool state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://illumos.org/msg/ZFS-8000-9P scan: scrub repaired 0 in 0 days 10:35:30 with 0 errors on Wed Apr 10 06:37:08 2019 config: NAME STATE READ WRITE CKSUM pool ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/235b119f-c819-11e7-a1cf-d05099785fe5 ONLINE 0 0 24 gptid/8757de96-c89c-11e7-af76-d05099785fe5 ONLINE 0 0 0 gptid/804f5c9f-c90b-11e7-a177-d05099785fe5 ONLINE 0 0 0 gptid/57173bc9-ca8d-11e7-83d8-d05099785fe5 ONLINE 0 0 0 gptid/74469597-ca8d-11e7-83d8-d05099785fe5 ONLINE 0 0 0 gptid/ba57886e-cb4f-11e7-9fab-d05099785fe5 ONLINE 0 0 0 errors: No known data errors
~/ $ zpool status pool: freenas-boot state: ONLINE scan: scrub repaired 0 in 0 days 00:05:53 with 0 errors on Tue Apr 9 03:50:53 2019 config: NAME STATE READ WRITE CKSUM freenas-boot ONLINE 0 0 0 da0p2 ONLINE 0 0 0 errors: No known data errors pool: pool state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://illumos.org/msg/ZFS-8000-9P scan: resilvered 60K in 0 days 00:00:01 with 0 errors on Sat Apr 13 00:09:27 2019 config: NAME STATE READ WRITE CKSUM pool ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/235b119f-c819-11e7-a1cf-d05099785fe5 ONLINE 4 449 0 gptid/8757de96-c89c-11e7-af76-d05099785fe5 ONLINE 0 0 0 gptid/804f5c9f-c90b-11e7-a177-d05099785fe5 ONLINE 0 0 0 gptid/57173bc9-ca8d-11e7-83d8-d05099785fe5 ONLINE 0 0 0 gptid/74469597-ca8d-11e7-83d8-d05099785fe5 ONLINE 0 0 0 gptid/ba57886e-cb4f-11e7-9fab-d05099785fe5 ONLINE 0 0 0
glabel status
to find out what gptid relates to what da#. For example, on my system:root@Emily-NAS:~ # glabel status Name Status Components gptid/ae487c50-bec3-11e8-b1c8-0cc47a9cd5a4 N/A nvd0p2 gptid/ae52d59d-bec3-11e8-b1c8-0cc47a9cd5a4 N/A nvd0p3 gptid/af7c42c6-bf05-11e8-b5f3-0cc47a9cd5a4 N/A da0p1 gptid/b07bc723-bf05-11e8-b5f3-0cc47a9cd5a4 N/A da1p1 gptid/b1893397-bf05-11e8-b5f3-0cc47a9cd5a4 N/A da2p1 gptid/b2bfc678-bf05-11e8-b5f3-0cc47a9cd5a4 N/A da3p1 gptid/b3c1849e-bf05-11e8-b5f3-0cc47a9cd5a4 N/A da4p1 gptid/b4d16ad2-bf05-11e8-b5f3-0cc47a9cd5a4 N/A da5p1 gptid/2e919d3d-2c1a-11e9-af8b-00074306773b N/A da6p1 gptid/2f292da6-2c1a-11e9-af8b-00074306773b N/A da7p1 gptid/2fb95d07-2c1a-11e9-af8b-00074306773b N/A da8p1 gptid/30514e6b-2c1a-11e9-af8b-00074306773b N/A da9p1 gptid/8710385b-becf-11e8-b1c8-0cc47a9cd5a4 N/A da10p1 gptid/87e94156-becf-11e8-b1c8-0cc47a9cd5a4 N/A da11p1 gptid/88db19ad-becf-11e8-b1c8-0cc47a9cd5a4 N/A da12p1 gptid/89addd3b-becf-11e8-b1c8-0cc47a9cd5a4 N/A da13p1 gptid/8a865453-becf-11e8-b1c8-0cc47a9cd5a4 N/A da14p1 gptid/8b66b1ef-becf-11e8-b1c8-0cc47a9cd5a4 N/A da15p1 gptid/8c69bc72-becf-11e8-b1c8-0cc47a9cd5a4 N/A da16p1 gptid/8d48655d-becf-11e8-b1c8-0cc47a9cd5a4 N/A da17p1 gptid/8e2b6d1f-becf-11e8-b1c8-0cc47a9cd5a4 N/A da18p1 gptid/8efea929-becf-11e8-b1c8-0cc47a9cd5a4 N/A da19p1 gptid/8fd4d25c-becf-11e8-b1c8-0cc47a9cd5a4 N/A da20p1 gptid/90c2759a-becf-11e8-b1c8-0cc47a9cd5a4 N/A da21p1 gptid/b9de3232-bf05-11e8-b5f3-0cc47a9cd5a4 N/A da22p1 gptid/baf4aba8-bf05-11e8-b5f3-0cc47a9cd5a4 N/A da23p1 gptid/bbf26621-bf05-11e8-b5f3-0cc47a9cd5a4 N/A da24p1 gptid/bc1e50e5-c1fa-11e8-87f0-0cc47a9cd5a4 N/A da25p1 gptid/a03dd690-c1fb-11e8-87f0-0cc47a9cd5a4 N/A da26p1 gptid/a6ed2ed5-c240-11e8-87f0-0cc47a9cd5a4 N/A da27p1 gptid/41d3312f-2c1a-11e9-af8b-00074306773b N/A da28p1 gptid/426b7b47-2c1a-11e9-af8b-00074306773b N/A da29p1 gptid/43029d18-2c1a-11e9-af8b-00074306773b N/A da30p1 gptid/af54c9c6-4277-11e9-af8b-00074306773b N/A da31p1 gptid/f69330dd-4b12-11e6-a97c-002590aecc79 N/A ada0p1 gptid/f6a61d33-4b12-11e6-a97c-002590aecc79 N/A ada0p2 gptid/f648c607-4b12-11e6-a97c-002590aecc79 N/A ada1p1 gptid/f659fd6d-4b12-11e6-a97c-002590aecc79 N/A ada1p2 root@Emily-NAS:~ #
gptid/235b119f-c819-11e7-a1cf-d05099785fe5
and after the resliver is done, keep an eye out for new errors.gptid/235b119f-c819-11e7-a1cf-d05099785fe5
. This morning I received the following two emails:Checking status of zfs pools: NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT freenas-boot 14.2G 8.36G 5.89G - - - 58% 1.00x ONLINE - pool 32.6T 18.7T 13.9T - - 13% 57% 1.00x DEGRADED /mnt pool: pool state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Thu Apr 18 22:06:18 2019 10.1T scanned at 602M/s, 9.24T issued at 548M/s, 18.7T total 1.54T resilvered, 49.41% done, 0 days 05:01:45 to go config: NAME STATE READ WRITE CKSUM pool DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 replacing-0 OFFLINE 0 0 0 10202630468201435458 OFFLINE 0 0 0 was /dev/gptid/235b119f-c819-11e7-a1cf-d05099785fe5 gptid/dc16ce6e-6260-11e9-9a47-d05099d0f832 ONLINE 0 0 0 gptid/8757de96-c89c-11e7-af76-d05099785fe5 ONLINE 0 0 0 gptid/804f5c9f-c90b-11e7-a177-d05099785fe5 ONLINE 0 0 0 gptid/57173bc9-ca8d-11e7-83d8-d05099785fe5 ONLINE 0 0 0 gptid/74469597-ca8d-11e7-83d8-d05099785fe5 ONLINE 0 0 0 gptid/ba57886e-cb4f-11e7-9fab-d05099785fe5 ONLINE 0 0 0 errors: No known data errors
freenas.local kernel log messages: > GEOM_MIRROR: Cannot open consumer ada1p1 (error=1). > GEOM_MIRROR: Device swap2 destroyed. > Limiting closed port RST response from 361 to 200 packets/sec
It is in the process of replacing the failed drive and estimating five hours to complete the process.I'm uncertain what this means.
GEOM_MIRROR
is about the swap space that was allocated on the storage drive. After your finished resilvering the storage pool, the middleware may be smart enough to fix that for you. If it doesn't, it is fine because you have swap space on all the other storage disks.~/ $ zpool status pool: freenas-boot state: ONLINE scan: scrub repaired 0 in 0 days 00:06:07 with 0 errors on Wed Apr 17 03:51:07 2019 config: NAME STATE READ WRITE CKSUM freenas-boot ONLINE 0 0 0 da0p2 ONLINE 0 0 0 errors: No known data errors pool: pool state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. scan: resilvered 3.11T in 0 days 11:49:25 with 0 errors on Fri Apr 19 09:55:43 2019 config: NAME STATE READ WRITE CKSUM pool ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/dc16ce6e-6260-11e9-9a47-d05099d0f832 ONLINE 0 0 0 gptid/8757de96-c89c-11e7-af76-d05099785fe5 ONLINE 0 0 0 gptid/804f5c9f-c90b-11e7-a177-d05099785fe5 ONLINE 0 0 0 gptid/57173bc9-ca8d-11e7-83d8-d05099785fe5 ONLINE 0 0 0 gptid/74469597-ca8d-11e7-83d8-d05099785fe5 ONLINE 0 0 0 gptid/ba57886e-cb4f-11e7-9fab-d05099785fe5 ONLINE 0 0 0
~/ $ sudo smartctl -x /dev/ada0 smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Red Device Model: WDC WD60EFRX-68MYMN1 Serial Number: WD-WX21D25R52XN LU WWN Device Id: 5 0014ee 2b757149c Firmware Version: 82.00A82 User Capacity: 6,001,175,126,016 bytes [6.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5700 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Mon Apr 22 22:05:32 2019 PDT SMART support is: Available - device has SMART capability. SMART support is: Enabled AAM feature is: Unavailable APM feature is: Unavailable Rd look-ahead is: Enabled Write cache is: Enabled DSN feature is: Unavailable ATA Security is: Disabled, frozen [SEC2] Wt Cache Reorder: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 3584) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 689) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x303d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE 1 Raw_Read_Error_Rate POSR-K 200 200 051 - 0 3 Spin_Up_Time POS--K 100 253 021 - 0 4 Start_Stop_Count -O--CK 100 100 000 - 1 5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0 7 Seek_Error_Rate -OSR-K 100 253 000 - 0 9 Power_On_Hours -O--CK 100 100 000 - 96 10 Spin_Retry_Count -O--CK 100 253 000 - 0 11 Calibration_Retry_Count -O--CK 100 253 000 - 0 12 Power_Cycle_Count -O--CK 100 100 000 - 1 192 Power-Off_Retract_Count -O--CK 200 200 000 - 0 193 Load_Cycle_Count -O--CK 200 200 000 - 4 194 Temperature_Celsius -O---K 105 099 000 - 47 196 Reallocated_Event_Count -O--CK 200 200 000 - 0 197 Current_Pending_Sector -O--CK 200 200 000 - 0 198 Offline_Uncorrectable ----CK 100 253 000 - 0 199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0 200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0 ||||||_ K auto-keep |||||__ C event count ||||___ R error rate |||____ S speed/performance ||_____ O updated online |______ P prefailure warning General Purpose Log Directory Version 1 SMART Log Directory Version 1 [multi-sector log support] Address Access R/W Size Description 0x00 GPL,SL R/O 1 Log Directory 0x01 SL R/O 1 Summary SMART error log 0x02 SL R/O 5 Comprehensive SMART error log 0x03 GPL R/O 6 Ext. Comprehensive SMART error log 0x06 SL R/O 1 SMART self-test log 0x07 GPL R/O 1 Extended self-test log 0x09 SL R/W 1 Selective self-test log 0x10 GPL R/O 1 NCQ Command Error log 0x11 GPL R/O 1 SATA Phy Event Counters log 0x21 GPL R/O 1 Write stream error log 0x22 GPL R/O 1 Read stream error log 0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log 0x80-0x9f GPL,SL R/W 16 Host vendor specific log 0xa0-0xa7 GPL,SL VS 16 Device vendor specific log 0xa8-0xb6 GPL,SL VS 1 Device vendor specific log 0xb7 GPL,SL VS 54 Device vendor specific log 0xbd GPL,SL VS 1 Device vendor specific log 0xc0 GPL,SL VS 1 Device vendor specific log 0xc1 GPL VS 93 Device vendor specific log 0xe0 GPL,SL R/W 1 SCT Command/Status 0xe1 GPL,SL R/W 1 SCT Data Transfer SMART Extended Comprehensive Error Log Version: 1 (6 sectors) No Errors Logged SMART Extended Self-test Log Version: 1 (1 sectors) Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 33 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. SCT Status Version: 3 SCT Version (vendor specific): 258 (0x0102) SCT Support Level: 1 Device State: Active (0) Current Temperature: 47 Celsius Power Cycle Min/Max Temperature: 26/53 Celsius Lifetime Min/Max Temperature: 26/53 Celsius Under/Over Temperature Limit Count: 0/0 Vendor specific: 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 SCT Temperature History Version: 2 Temperature Sampling Period: 1 minute Temperature Logging Interval: 1 minute Min/Max recommended Temperature: 0/60 Celsius Min/Max Temperature Limit: -41/85 Celsius Temperature History Size (Index): 478 (30) Index Estimated Time Temperature Celsius 31 2019-04-22 14:08 48 ***************************** ... ..( 30 skipped). .. ***************************** 62 2019-04-22 14:39 48 ***************************** 63 2019-04-22 14:40 49 ****************************** ... ..(125 skipped). .. ****************************** 189 2019-04-22 16:46 49 ****************************** 190 2019-04-22 16:47 50 ******************************* ... ..(127 skipped). .. ******************************* 318 2019-04-22 18:55 50 ******************************* 319 2019-04-22 18:56 49 ****************************** ... ..( 82 skipped). .. ****************************** 402 2019-04-22 20:19 49 ****************************** 403 2019-04-22 20:20 48 ***************************** ... ..( 88 skipped). .. ***************************** 14 2019-04-22 21:49 48 ***************************** 15 2019-04-22 21:50 47 **************************** ... ..( 14 skipped). .. **************************** 30 2019-04-22 22:05 47 **************************** SCT Error Recovery Control: Read: 70 (7.0 seconds) Write: 70 (7.0 seconds) Device Statistics (GP/SMART Log 0x04) not supported Pending Defects log (GP Log 0x0c) not supported SATA Phy Event Counters (GP Log 0x11) ID Size Value Description 0x0001 2 0 Command failed due to ICRC error 0x0002 2 0 R_ERR response for data FIS 0x0003 2 0 R_ERR response for device-to-host data FIS 0x0004 2 0 R_ERR response for host-to-device data FIS 0x0005 2 0 R_ERR response for non-data FIS 0x0006 2 0 R_ERR response for device-to-host non-data FIS 0x0007 2 0 R_ERR response for host-to-device non-data FIS 0x0008 2 0 Device-to-host non-data FIS retries 0x0009 2 5 Transition from drive PhyRdy to drive PhyNRdy 0x000a 2 8 Device-to-host register FISes sent due to a COMRESET 0x000b 2 0 CRC errors within host-to-device FIS 0x000d 2 0 Non-CRC errors within host-to-device FIS 0x000f 2 0 R_ERR response for host-to-device data FIS, CRC 0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC 0x8000 4 346073 Vendor specific