CAM Status Error during SCRUB

T

tealcomp

Explorer
Joined
Sep 5, 2016
Messages
59
Hi All:

As some of you know, we recently built a new FreeNAS box about a month or so ago. During that time and since, we have been going through the paces, running memory, CPU, and lots of HDD tests, in preparation for the device to take over as the master NAS server in our home. This week, we finally started to RSYNC the data over from our QNAP device over to this new server. After copying over the data to this server, we decided it would be a good idea to run the server's first scrub. We started the scrub up and verified all was progressing before going to bed. Awoke this morning to an email from the server indicating that 6 of the 10 drives had encountered errors.

Code:
jupiter.local kernel log messages:
(da4:mps0:0:4:0): READ(10). CDB: 28 00 8c e6 48 b8 00 00 e8 00 length 118784 SMID 468 terminated ioc 804b scsi 0 state 0 xfer 0
(da4:mps0:0:4:0): READ(10). CDB: 28 00 8c e6 48 b8 00 00 e8 00
(da4:mps0:0:4:0): CAM status: CCB request completed with an error
(da4:mps0:0:4:0): Retrying command
(da4:mps0:0:4:0): READ(10). CDB: 28 00 8c e6 47 d0 00 00 e8 00
(da4:mps0:0:4:0): CAM status: SCSI Status Error
(da4:mps0:0:4:0): SCSI status: Check Condition
(da4:mps0:0:4:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da4:mps0:0:4:0): Info: 0x8ce647d0
(da4:mps0:0:4:0): Error 5, Unretryable error
GEOM_ELI: g_eli_read_done() failed (error=5) gptid/13ef3990-fb93-11e6-abcc-0cc47a8668da.eli[READ(offset=1208170225664, length=118784)]
	(da4:mps0:0:4:0): READ(10). CDB: 28 00 8c e9 1e 28 00 00 e8 00 length 118784 SMID 361 terminated ioc 804b scsi 0 state 0 xfer 0
(da4:mps0:0:4:0): READ(10). CDB: 28 00 8c e9 1e 28 00 00 e8 00
(da4:mps0:0:4:0): CAM status: CCB request completed with an error
(da4:mps0:0:4:0): Retrying command
(da4:mps0:0:4:0): READ(10). CDB: 28 00 8c e9 1d 48 00 00 e0 00
(da4:mps0:0:4:0): CAM status: SCSI Status Error
(da4:mps0:0:4:0): SCSI status: Check Condition
(da4:mps0:0:4:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da4:mps0:0:4:0): Info: 0x8ce91d48
(da4:mps0:0:4:0): Error 5, Unretryable error
GEOM_ELI: g_eli_read_done() failed (error=5) gptid/13ef3990-fb93-11e6-abcc-0cc47a8668da.eli[READ(offset=1208265314304, length=114688)]
	(da0:mps0:0:0:0): READ(10). CDB: 28 00 8c e7 86 b8 00 00 e0 00 length 114688 SMID 719 terminated ioc 804b scsi 0 state 0 xfer 0
(da0:mps0:0:0:0): READ(10). CDB: 28 00 8c e7 86 b8 00 00 e0 00
(da0:mps0:0:0:0): CAM status: CCB request completed with an error
(da0:mps0:0:0:0): Retrying command
(da0:mps0:0:0:0): READ(10). CDB: 28 00 8c e7 85 d0 00 00 e8 00
(da0:mps0:0:0:0): CAM status: SCSI Status Error
(da0:mps0:0:0:0): SCSI status: Check Condition
(da0:mps0:0:0:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da0:mps0:0:0:0): Info: 0x8ce785d0
(da0:mps0:0:0:0): Error 5, Unretryable error
GEOM_ELI: g_eli_read_done() failed (error=5) gptid/10cd817e-fb93-11e6-abcc-0cc47a8668da.eli[READ(offset=1208211906560, length=118784)]
	(da6:mps0:0:6:0): READ(10). CDB: 28 00 8c e8 cf b0 00 00 28 00 length 20480 SMID 531 terminated ioc 804b scsi 0 state 0 xfer 0
(da6:mps0:0:6:0): READ(10). CDB: 28 00 8c e8 cf b0 00 00 28 00
(da6:mps0:0:6:0): CAM status: CCB request completed with an error
(da6:mps0:0:6:0): Retrying command
(da6:mps0:0:6:0): READ(10). CDB: 28 00 8c e8 ce c8 00 00 e8 00
(da6:mps0:0:6:0): CAM status: SCSI Status Error
(da6:mps0:0:6:0): SCSI status: Check Condition
(da6:mps0:0:6:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da6:mps0:0:6:0): Info: 0x8ce8cec8
(da6:mps0:0:6:0): Error 5, Unretryable error
GEOM_ELI: g_eli_read_done() failed (error=5) gptid/156d3a72-fb93-11e6-abcc-0cc47a8668da.eli[READ(offset=1208255025152, length=118784)]
	(da2:mps0:0:2:0): READ(10). CDB: 28 00 8c e8 d0 50 00 00 20 00 length 16384 SMID 838 terminated ioc 804b scsi 0 state 0 xfer 0
(da2:mps0:0:2:0): READ(10). CDB: 28 00 8c e8 d0 50 00 00 20 00
(da2:mps0:0:2:0): CAM status: CCB request completed with an error
(da2:mps0:0:2:0): Retrying command
(da2:mps0:0:2:0): READ(10). CDB: 28 00 8c e8 cf 68 00 00 e8 00
(da2:mps0:0:2:0): CAM status: SCSI Status Error
(da2:mps0:0:2:0): SCSI status: Check Condition
(da2:mps0:0:2:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da2:mps0:0:2:0): Info: 0x8ce8cf68
(da2:mps0:0:2:0): Error 5, Unretryable error
GEOM_ELI: g_eli_read_done() failed (error=5) gptid/1252f2b2-fb93-11e6-abcc-0cc47a8668da.eli[READ(offset=1208255107072, length=118784)]
	(da6:mps0:0:6:0): READ(10). CDB: 28 00 8c e9 bb 20 00 00 e8 00 length 118784 SMID 505 terminated ioc 804b scsi 0 state 0 xfer 0
(da6:mps0:0:6:0): READ(10). CDB: 28 00 8c e9 bb 20 00 00 e8 00
(da6:mps0:0:6:0): CAM status: CCB request completed with an error
(da6:mps0:0:6:0): Retrying command
(da6:mps0:0:6:0): READ(10). CDB: 28 00 8c e9 ba 38 00 00 e8 00
(da6:mps0:0:6:0): CAM status: SCSI Status Error
(da6:mps0:0:6:0): SCSI status: Check Condition
(da6:mps0:0:6:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da6:mps0:0:6:0): Info: 0x8ce9ba38
(da6:mps0:0:6:0): Error 5, Unretryable error
GEOM_ELI: g_eli_read_done() failed (error=5) gptid/156d3a72-fb93-11e6-abcc-0cc47a8668da.eli[READ(offset=1208285884416, length=118784)]
(ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 e8 18 90 e7 40 8c 00 00 00 00 00
(ada0:ahcich0:0:0:0): CAM status: ATA Status Error
(ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada0:ahcich0:0:0:0): RES: 41 40 18 90 e7 00 8c 00 00 00 00
(ada0:ahcich0:0:0:0): Retrying command
	(da2:mps0:0:2:0): READ(10). CDB: 28 00 8c e7 e8 70 00 00 e8 00 length 118784 SMID 578 terminated ioc 804b scsi 0 state 0 xfer 0
(da2:mps0:0:2:0): READ(10). CDB: 28 00 8c e7 e8 70 00 00 e8 00
(da2:mps0:0:2:0): CAM status: CCB request completed with an error
(da2:mps0:0:2:0): Retrying command
(da2:mps0:0:2:0): READ(10). CDB: 28 00 8c e7 e7 90 00 00 e0 00
(da2:mps0:0:2:0): CAM status: SCSI Status Error
(da2:mps0:0:2:0): SCSI status: Check Condition
(da2:mps0:0:2:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da2:mps0:0:2:0): Info: 0x8ce7e790
(da2:mps0:0:2:0): Error 5, Unretryable error
GEOM_ELI: g_eli_read_done() failed (error=5) gptid/1252f2b2-fb93-11e6-abcc-0cc47a8668da.eli[READ(offset=1208224718848, length=114688)]
	(da1:mps0:0:1:0): READ(10). CDB: 28 00 8c ea 67 50 00 00 e0 00 length 114688 SMID 768 terminated ioc 804b scsi 0 state 0 xfer 0
(da1:mps0:0:1:0): READ(10). CDB: 28 00 8c ea 67 50 00 00 e0 00
(da1:mps0:0:1:0): CAM status: CCB request completed with an error
(da1:mps0:0:1:0): Retrying command
(da1:mps0:0:1:0): READ(10). CDB: 28 00 8c ea 66 68 00 00 e8 00
(da1:mps0:0:1:0): CAM status: SCSI Status Error
(da1:mps0:0:1:0): SCSI status: Check Condition
(da1:mps0:0:1:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da1:mps0:0:1:0): Info: 0x8cea6668
(da1:mps0:0:1:0): Error 5, Unretryable error
GEOM_ELI: g_eli_read_done() failed (error=5) gptid/118f126a-fb93-11e6-abcc-0cc47a8668da.eli[READ(offset=1208308453376, length=118784)]
	(da4:mps0:0:4:0): READ(10). CDB: 28 00 8c eb 0f 70 00 00 e8 00 length 118784 SMID 621 terminated ioc 804b scsi 0 state 0 xfer 0
(da4:mps0:0:4:0): READ(10). CDB: 28 00 8c eb 0f 70 00 00 e8 00
(da4:mps0:0:4:0): CAM status: CCB request completed with an error
(da4:mps0:0:4:0): Retrying command
(da4:mps0:0:4:0): READ(10). CDB: 28 00 8c eb 0e 88 00 00 e8 00
(da4:mps0:0:4:0): CAM status: SCSI Status Error
(da4:mps0:0:4:0): SCSI status: Check Condition
(da4:mps0:0:4:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da4:mps0:0:4:0): Info: 0x8ceb0e88
(da4:mps0:0:4:0): Error 5, Unretryable error
GEOM_ELI: g_eli_read_done() failed (error=5) gptid/13ef3990-fb93-11e6-abcc-0cc47a8668da.eli[READ(offset=1208330489856, length=118784)]
(ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 e0 50 0f eb 40 8c 00 00 00 00 00
(ada0:ahcich0:0:0:0): CAM status: ATA Status Error
(ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada0:ahcich0:0:0:0): RES: 41 40 50 0f eb 00 8c 00 00 00 00
(ada0:ahcich0:0:0:0): Retrying command
	(da1:mps0:0:1:0): READ(10). CDB: 28 00 8c ec 3f a0 00 00 c0 00 length 98304 SMID 924 terminated ioc 804b scsi 0 state 0 xfer 0
(da1:mps0:0:1:0): READ(10). CDB: 28 00 8c ec 3f a0 00 00 c0 00
(da1:mps0:0:1:0): CAM status: CCB request completed with an error
(da1:mps0:0:1:0): Retrying command
(da1:mps0:0:1:0): READ(10). CDB: 28 00 8c ec 3e b8 00 00 e8 00
(da1:mps0:0:1:0): CAM status: SCSI Status Error
(da1:mps0:0:1:0): SCSI status: Check Condition
(da1:mps0:0:1:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
	(da2:mps0:0:2:0): READ(10). CDB: 28 00 8c ec 3f 50 00 00 c0 00 length 98304 SMID 964 terminated ioc 804b scsi 0 state 0 xfer (da1:mps0:0:1:0): Info: 0x8cec3eb8
0
(da1:(da2:mps0:0:2:0): READ(10). CDB: 28 00 8c ec 3f 50 00 00 c0 00
mps0:0:(da2:mps0:0:2:0): CAM status: CCB request completed with an error
1:(da2:0): mps0:0:Error 5, Unretryable error
2:0): GEOM_ELIRetrying command
: g_eli_read_done() failed (error=5)(da2:mps0:0:2:0): READ(10). CDB: 28 00 8c ec 3e 68 00 00 e8 00
(da2:mps0:0:2:0): CAM status: SCSI Status Error
gptid/118f126a-fb93-11e6-abcc-0cc47a8668da.eli[READ(offset=1208370360320, length=118784)](da2:mps0:0:2:0): SCSI status: Check Condition

(da2:mps0:0:2:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da2:mps0:0:2:0): Info: 0x8cec3e68
(da2:mps0:0:2:0): Error 5, Unretryable error
GEOM_ELI: g_eli_read_done() failed (error=5) gptid/1252f2b2-fb93-11e6-abcc-0cc47a8668da.eli[READ(offset=1208370319360, length=118784)]
(ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 e8 78 13 a0 40 99 01 00 00 00 00
(ada0:ahcich0:0:0:0): CAM status: ATA Status Error
(ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada0:ahcich0:0:0:0): RES: 41 40 78 13 a0 00 99 01 00 00 00
(ada0:ahcich0:0:0:0): Retrying command
	(da4:mps0:0:4:0): READ(16). CDB: 88 00 00 00 00 01 99 a0 a9 90 00 00 00 e8 00 00 length 118784 SMID 403 terminated ioc 804b scsi 0 state 0 xfer 0
(da4:mps0:0:4:0): READ(16). CDB: 88 00 00 00 00 01 99 a0 a9 90 00 00 00 e8 00 00
(da4:mps0:0:4:0): CAM status: CCB request completed with an error
(da4:mps0:0:4:0): Retrying command
(da4:mps0:0:4:0): READ(16). CDB: 88 00 00 00 00 01 99 a0 a8 a8 00 00 00 e8 00 00
(da4:mps0:0:4:0): CAM status: SCSI Status Error
(da4:mps0:0:4:0): SCSI status: Check Condition
(da4:mps0:0:4:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da4:mps0:0:4:0): Info: 0x199a0a8a8
(da4:mps0:0:4:0): Error 5, Unretryable error
GEOM_ELI: g_eli_read_done() failed (error=5) gptid/13ef3990-fb93-11e6-abcc-0cc47a8668da.eli[READ(offset=3516526514176, length=118784)]
	(da4:mps0:0:4:0): READ(16). CDB: 88 00 00 00 00 01 99 a7 44 88 00 00 00 e8 00 00 length 118784 SMID 849 terminated ioc 804b scsi 0 state 0 xfer 0
(da4:mps0:0:4:0): READ(16). CDB: 88 00 00 00 00 01 99 a7 44 88 00 00 00 e8 00 00
(da4:mps0:0:4:0): CAM status: CCB request completed with an error
(da4:mps0:0:4:0): Retrying command
(da4:mps0:0:4:0): READ(16). CDB: 88 00 00 00 00 01 99 a7 43 a8 00 00 00 e0 00 00
(da4:mps0:0:4:0): CAM status: SCSI Status Error
(da4:mps0:0:4:0): SCSI status: Check Condition
(da4:mps0:0:4:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da4:mps0:0:4:0): Info: 0x199a743a8
(da4:mps0:0:4:0): Error 5, Unretryable error
GEOM_ELI: g_eli_read_done() failed (error=5) gptid/13ef3990-fb93-11e6-abcc-0cc47a8668da.eli[READ(offset=3516748156928, length=114688)]


-- End of security output --


ada0
da0, da1, da2, da4, da6

So an error came up like this before, and it seemed to be caused by a fault fan switch on the back of my case. To Fractal's credit, they replaced the switch and I did some testing to validate that resolved the problem. Since we could not reproduce the problem with the new switch, we figured the issue was resolved. That issue is in a previous post. It seemed moving the switch back and forth during a badblocks test caused the cam errors to throw.
Spring forward to now, and we are again seeing these errors. First thought was, maybe it was the breakout cables or perhaps the M1015 card ; but then we noticed that ADA0 (drive connected directly to the SATA port) also encountered a SATA equivalent error. Given that drive is connected via a different signaling cable independent of the da* series of drives, it seems unlikely (though not impossible) that cabling is the problem.

The bigger oddity is if a smartctl -x is issued, we see these FPMA errors on the aforementioned drives, but if a smartctl -a is run you do not see the errors. In fact, looking for the key indicators of drive issues, none are immediately present.

Checked for the following areas for all (10) drives:

Raw_Read_Error_Rate
Reallocated_Event_Ct
Current_Pending_Sector
Offline_Uncorrectable
Multi_Zone_Error_Rate

This is starting to look more and more like some kind of power oddity. The investigation is still ongoing, but look forward to hearing input from the community.

We are in the process of running a new extended test on all (10) drives, which will be posted shortly after completion for your consideration. The status of the pool will also be included. Everything looks in order except for those CAM errors and we don't like to see errors, unless they can be effectively explained and if at all possible solved :)

Thanks,
-Dan
 
Last edited:
T

tealcomp

Explorer
Joined
Sep 5, 2016
Messages
59
Anything is possible; but the drives have never been over 39'C and the other components seem to be well within the thermal design limits. There are 6 fans (not including the CPU fan) cooling this case, because of initial cooling concerns. The extended is still running; will post all results once completed.
 
R

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,778
tealcomp said:
the drives have never been over 39'C and the other components seem to be well within the thermal design limits
Click to expand...
Just something to consider, temperature changes can affect all kinds of things, including connectors, already-marginal components, power consumption etc.
 
T

tealcomp

Explorer
Joined
Sep 5, 2016
Messages
59
Definitely won't discount it; but, there are automated sensor alerts set up from IPMI, as well as SMART monitoring of the drives enabled with email notification. The one thing that cannot be checked, at least with what's on board is the M1015 controller temperature. Originally thought the M1015 was suspect until a similar error on one of the drives that is attached directly to the motherboard showed up. Aside from a direct temperature probe, any thoughts about how to go about ruling out heat issues beyond what has been already done?
 
R

rs225

Guru
Joined
Jun 28, 2014
Messages
878
It does sound like power. Are the drives powered by many direct lines to the PSU, or are many drives sharing a single line to the PSU? Are there any fans on these lines?

Is the building wiring relatively modern, or are there any two-prong adapters or loads that might have come on overnight?
 
T

tealcomp

Explorer
Joined
Sep 5, 2016
Messages
59
There are (3) separate power lines to the drives; each rack of (4) drives has an independent power cable to the PSU, additionally the (2) drives in the motherboard side of the chassis are also on their own cable. The fans are not on the same cables as the HDD's, (3) run from a sata power cable to the 3-way switch (the Fractals that came with the case) and the other (3) are powered by the motherboard. The house is modern with proper amperage to the outlets; this server is also plugged into a sinewave generating UPS, so the power coming in should be clean. Note I have multiple other servers in the house and they are not experiencing any of these issues to my knowledge.
 
T

tealcomp

Explorer
Joined
Sep 5, 2016
Messages
59
As promised, here is the output for smartctl -a from all drives, ada0-ada1, da0-da3

Code:
==========
da0
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Western Digital Red
Device Model:	 WDC WD60EFRX-68L0BN1
Serial Number:	WD-WX31D96KN1T1
LU WWN Device Id: 5 0014ee 2635b8899
Firmware Version: 82.00A82
User Capacity:	6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	5700 rpm
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Sat Mar  4 23:00:14 2017 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection:		 ( 5744) seconds.
Offline data collection
capabilities:			  (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time:	  (   2) minutes.
Extended self-test routine
recommended polling time:	  ( 711) minutes.
Conveyance self-test routine
recommended polling time:	  (   5) minutes.
SCT capabilities:			(0x303d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x002f   200   200   051	Pre-fail  Always	   -	   0
  3 Spin_Up_Time			0x0027   202   196   021	Pre-fail  Always	   -	   8900
  4 Start_Stop_Count		0x0032   100   100   000	Old_age   Always	   -	   32
  5 Reallocated_Sector_Ct   0x0033   200   200   140	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x002e   200   200   000	Old_age   Always	   -	   0
  9 Power_On_Hours		  0x0032   099   099   000	Old_age   Always	   -	   734
10 Spin_Retry_Count		0x0032   100   253   000	Old_age   Always	   -	   0
11 Calibration_Retry_Count 0x0032   100   253   000	Old_age   Always	   -	   0
12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   32
192 Power-Off_Retract_Count 0x0032   200   200   000	Old_age   Always	   -	   30
193 Load_Cycle_Count		0x0032   200   200   000	Old_age   Always	   -	   11
194 Temperature_Celsius	 0x0022   117   114   000	Old_age   Always	   -	   35
196 Reallocated_Event_Count 0x0032   200   200   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0032   200   200   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0030   100   253   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x0032   200   200   000	Old_age   Always	   -	   6
200 Multi_Zone_Error_Rate   0x0008   200   200   000	Old_age   Offline	  -	   0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline	Completed without error	   00%	   733		 -
# 2  Short offline	   Completed without error	   00%	   720		 -
# 3  Extended offline	Completed without error	   00%	   513		 -
# 4  Extended offline	Completed without error	   00%	   389		 -
# 5  Short offline	   Completed without error	   00%	   297		 -
# 6  Short offline	   Completed without error	   00%	   241		 -
# 7  Short offline	   Completed without error	   00%	   240		 -
# 8  Short offline	   Completed without error	   00%	   228		 -
# 9  Conveyance offline  Completed without error	   00%		50		 -
#10  Conveyance offline  Completed without error	   00%		49		 -
#11  Extended offline	Completed without error	   00%		45		 -
#12  Short offline	   Completed without error	   00%		32		 -
#13  Short offline	   Completed without error	   00%		31		 -
#14  Short offline	   Completed without error	   00%		31		 -
#15  Short offline	   Completed without error	   00%		31		 -
#16  Short offline	   Completed without error	   00%		31		 -
#17  Short offline	   Completed without error	   00%		31		 -
#18  Short offline	   Completed without error	   00%		30		 -
#19  Short offline	   Completed without error	   00%		30		 -
#20  Short offline	   Completed without error	   00%		30		 -
#21  Short offline	   Completed without error	   00%		30		 -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

==========
da1
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Western Digital Red
Device Model:	 WDC WD60EFRX-68L0BN1
Serial Number:	WD-WXJ1H26HV3M3
LU WWN Device Id: 5 0014ee 2b8aecae1
Firmware Version: 82.00A82
User Capacity:	6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	5700 rpm
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Sat Mar  4 23:00:14 2017 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection:		 ( 1364) seconds.
Offline data collection
capabilities:			  (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time:	  (   2) minutes.
Extended self-test routine
recommended polling time:	  ( 667) minutes.
Conveyance self-test routine
recommended polling time:	  (   5) minutes.
SCT capabilities:			(0x303d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x002f   200   200   051	Pre-fail  Always	   -	   0
  3 Spin_Up_Time			0x0027   201   197   021	Pre-fail  Always	   -	   8908
  4 Start_Stop_Count		0x0032   100   100   000	Old_age   Always	   -	   32
  5 Reallocated_Sector_Ct   0x0033   200   200   140	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x002e   200   200   000	Old_age   Always	   -	   0
  9 Power_On_Hours		  0x0032   100   100   000	Old_age   Always	   -	   727
10 Spin_Retry_Count		0x0032   100   253   000	Old_age   Always	   -	   0
11 Calibration_Retry_Count 0x0032   100   253   000	Old_age   Always	   -	   0
12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   32
192 Power-Off_Retract_Count 0x0032   200   200   000	Old_age   Always	   -	   30
193 Load_Cycle_Count		0x0032   200   200   000	Old_age   Always	   -	   11
194 Temperature_Celsius	 0x0022   117   114   000	Old_age   Always	   -	   35
196 Reallocated_Event_Count 0x0032   200   200   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0032   200   200   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0030   100   253   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x0032   200   200   000	Old_age   Always	   -	   8
200 Multi_Zone_Error_Rate   0x0008   200   200   000	Old_age   Offline	  -	   0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline	Completed without error	   00%	   726		 -
# 2  Short offline	   Completed without error	   00%	   713		 -
# 3  Extended offline	Completed without error	   00%	   506		 -
# 4  Extended offline	Completed without error	   00%	   382		 -
# 5  Short offline	   Completed without error	   00%	   289		 -
# 6  Short offline	   Completed without error	   00%	   234		 -
# 7  Short offline	   Completed without error	   00%	   233		 -
# 8  Short offline	   Completed without error	   00%	   221		 -
# 9  Conveyance offline  Completed without error	   00%		50		 -
#10  Extended offline	Completed without error	   00%		44		 -
#11  Short offline	   Completed without error	   00%		32		 -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

==========
da2
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Western Digital Red
Device Model:	 WDC WD60EFRX-68L0BN1
Serial Number:	WD-WX31D96KNFNL
LU WWN Device Id: 5 0014ee 20e0628b9
Firmware Version: 82.00A82
User Capacity:	6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	5700 rpm
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Sat Mar  4 23:00:15 2017 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection:		 ( 5024) seconds.
Offline data collection
capabilities:			  (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time:	  (   2) minutes.
Extended self-test routine
recommended polling time:	  ( 703) minutes.
Conveyance self-test routine
recommended polling time:	  (   5) minutes.
SCT capabilities:			(0x303d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x002f   200   200   051	Pre-fail  Always	   -	   0
  3 Spin_Up_Time			0x0027   201   197   021	Pre-fail  Always	   -	   8908
  4 Start_Stop_Count		0x0032   100   100   000	Old_age   Always	   -	   32
  5 Reallocated_Sector_Ct   0x0033   200   200   140	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x002e   200   200   000	Old_age   Always	   -	   0
  9 Power_On_Hours		  0x0032   099   099   000	Old_age   Always	   -	   733
10 Spin_Retry_Count		0x0032   100   253   000	Old_age   Always	   -	   0
11 Calibration_Retry_Count 0x0032   100   253   000	Old_age   Always	   -	   0
12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   32
192 Power-Off_Retract_Count 0x0032   200   200   000	Old_age   Always	   -	   30
193 Load_Cycle_Count		0x0032   200   200   000	Old_age   Always	   -	   11
194 Temperature_Celsius	 0x0022   119   115   000	Old_age   Always	   -	   33
196 Reallocated_Event_Count 0x0032   200   200   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0032   200   200   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0030   100   253   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x0032   200   200   000	Old_age   Always	   -	   1
200 Multi_Zone_Error_Rate   0x0008   200   200   000	Old_age   Offline	  -	   0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline	Completed without error	   00%	   733		 -
# 2  Short offline	   Completed without error	   00%	   719		 -
# 3  Extended offline	Completed without error	   00%	   512		 -
# 4  Extended offline	Completed without error	   00%	   389		 -
# 5  Short offline	   Completed without error	   00%	   296		 -
# 6  Short offline	   Completed without error	   00%	   241		 -
# 7  Short offline	   Completed without error	   00%	   240		 -
# 8  Short offline	   Completed without error	   00%	   228		 -
# 9  Conveyance offline  Completed without error	   00%		50		 -
#10  Extended offline	Completed without error	   00%		45		 -
#11  Short offline	   Completed without error	   00%		32		 -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

==========
da3
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Western Digital Red
Device Model:	 WDC WD60EFRX-68L0BN1
Serial Number:	WD-WX31D96KNL35
LU WWN Device Id: 5 0014ee 2b8b0fcc5
Firmware Version: 82.00A82
User Capacity:	6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	5700 rpm
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Sat Mar  4 23:00:15 2017 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection:		 ( 6164) seconds.
Offline data collection
capabilities:			  (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time:	  (   2) minutes.
Extended self-test routine
recommended polling time:	  ( 715) minutes.
Conveyance self-test routine
recommended polling time:	  (   5) minutes.
SCT capabilities:			(0x303d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x002f   200   200   051	Pre-fail  Always	   -	   0
  3 Spin_Up_Time			0x0027   201   197   021	Pre-fail  Always	   -	   8925
  4 Start_Stop_Count		0x0032   100   100   000	Old_age   Always	   -	   32
  5 Reallocated_Sector_Ct   0x0033   200   200   140	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x002e   200   200   000	Old_age   Always	   -	   0
  9 Power_On_Hours		  0x0032   099   099   000	Old_age   Always	   -	   734
10 Spin_Retry_Count		0x0032   100   253   000	Old_age   Always	   -	   0
11 Calibration_Retry_Count 0x0032   100   253   000	Old_age   Always	   -	   0
12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   32
192 Power-Off_Retract_Count 0x0032   200   200   000	Old_age   Always	   -	   30
193 Load_Cycle_Count		0x0032   200   200   000	Old_age   Always	   -	   11
194 Temperature_Celsius	 0x0022   121   116   000	Old_age   Always	   -	   31
196 Reallocated_Event_Count 0x0032   200   200   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0032   200   200   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0030   100   253   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x0032   200   200   000	Old_age   Always	   -	   4
200 Multi_Zone_Error_Rate   0x0008   200   200   000	Old_age   Offline	  -	   0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline	Completed without error	   00%	   734		 -
# 2  Short offline	   Completed without error	   00%	   720		 -
# 3  Extended offline	Completed without error	   00%	   513		 -
# 4  Extended offline	Completed without error	   00%	   389		 -
# 5  Short offline	   Completed without error	   00%	   297		 -
# 6  Short offline	   Completed without error	   00%	   241		 -
# 7  Short offline	   Completed without error	   00%	   240		 -
# 8  Short offline	   Completed without error	   00%	   228		 -
# 9  Conveyance offline  Completed without error	   00%		50		 -
#10  Extended offline	Completed without error	   00%		45		 -
#11  Short offline	   Completed without error	   00%		32		 -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
 
T

tealcomp

Explorer
Joined
Sep 5, 2016
Messages
59
And da4-da7

Code:
==========
da4
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Western Digital Red
Device Model:	 WDC WD60EFRX-68L0BN1
Serial Number:	WD-WXN1H26SNFXJ
LU WWN Device Id: 5 0014ee 2b8b10daa
Firmware Version: 82.00A82
User Capacity:	6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	5700 rpm
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Sat Mar  4 23:00:15 2017 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:	  ( 241)	Self-test routine in progress...
					10% of test remaining.
Total time to complete Offline
data collection:		 ( 7964) seconds.
Offline data collection
capabilities:			  (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time:	  (   2) minutes.
Extended self-test routine
recommended polling time:	  ( 733) minutes.
Conveyance self-test routine
recommended polling time:	  (   5) minutes.
SCT capabilities:			(0x303d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x002f   200   200   051	Pre-fail  Always	   -	   0
  3 Spin_Up_Time			0x0027   218   198   021	Pre-fail  Always	   -	   8091
  4 Start_Stop_Count		0x0032   100   100   000	Old_age   Always	   -	   30
  5 Reallocated_Sector_Ct   0x0033   200   200   140	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x002e   200   200   000	Old_age   Always	   -	   0
  9 Power_On_Hours		  0x0032   099   099   000	Old_age   Always	   -	   737
10 Spin_Retry_Count		0x0032   100   253   000	Old_age   Always	   -	   0
11 Calibration_Retry_Count 0x0032   100   253   000	Old_age   Always	   -	   0
12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   30
192 Power-Off_Retract_Count 0x0032   200   200   000	Old_age   Always	   -	   28
193 Load_Cycle_Count		0x0032   200   200   000	Old_age   Always	   -	   11
194 Temperature_Celsius	 0x0022   118   115   000	Old_age   Always	   -	   34
196 Reallocated_Event_Count 0x0032   200   200   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0032   200   200   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0030   100   253   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x0032   200   200   000	Old_age   Always	   -	   4
200 Multi_Zone_Error_Rate   0x0008   200   200   000	Old_age   Offline	  -	   0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline	   Completed without error	   00%	   723		 -
# 2  Extended offline	Completed without error	   00%	   516		 -
# 3  Extended offline	Completed without error	   00%	   392		 -
# 4  Short offline	   Completed without error	   00%	   361		 -
# 5  Short offline	   Completed without error	   00%	   244		 -
# 6  Short offline	   Completed without error	   00%	   243		 -
# 7  Short offline	   Completed without error	   00%	   231		 -
# 8  Conveyance offline  Completed without error	   00%		51		 -
# 9  Extended offline	Completed without error	   00%		45		 -
#10  Short offline	   Completed without error	   00%		32		 -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

==========
da5
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Western Digital Red
Device Model:	 WDC WD60EFRX-68L0BN1
Serial Number:	WD-WXJ1H26RTE0H
LU WWN Device Id: 5 0014ee 263592502
Firmware Version: 82.00A82
User Capacity:	6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	5700 rpm
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Sat Mar  4 23:00:15 2017 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection:		 ( 6824) seconds.
Offline data collection
capabilities:			  (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time:	  (   2) minutes.
Extended self-test routine
recommended polling time:	  ( 722) minutes.
Conveyance self-test routine
recommended polling time:	  (   5) minutes.
SCT capabilities:			(0x303d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x002f   200   200   051	Pre-fail  Always	   -	   0
  3 Spin_Up_Time			0x0027   218   198   021	Pre-fail  Always	   -	   8075
  4 Start_Stop_Count		0x0032   100   100   000	Old_age   Always	   -	   30
  5 Reallocated_Sector_Ct   0x0033   200   200   140	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x002e   200   200   000	Old_age   Always	   -	   0
  9 Power_On_Hours		  0x0032   099   099   000	Old_age   Always	   -	   735
10 Spin_Retry_Count		0x0032   100   253   000	Old_age   Always	   -	   0
11 Calibration_Retry_Count 0x0032   100   253   000	Old_age   Always	   -	   0
12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   30
192 Power-Off_Retract_Count 0x0032   200   200   000	Old_age   Always	   -	   28
193 Load_Cycle_Count		0x0032   200   200   000	Old_age   Always	   -	   11
194 Temperature_Celsius	 0x0022   118   115   000	Old_age   Always	   -	   34
196 Reallocated_Event_Count 0x0032   200   200   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0032   200   200   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0030   100   253   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x0032   200   200   000	Old_age   Always	   -	   5
200 Multi_Zone_Error_Rate   0x0008   200   200   000	Old_age   Offline	  -	   0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline	Completed without error	   00%	   735		 -
# 2  Short offline	   Completed without error	   00%	   721		 -
# 3  Extended offline	Completed without error	   00%	   515		 -
# 4  Extended offline	Completed without error	   00%	   391		 -
# 5  Short offline	   Completed without error	   00%	   299		 -
# 6  Short offline	   Completed without error	   00%	   243		 -
# 7  Short offline	   Completed without error	   00%	   242		 -
# 8  Short offline	   Completed without error	   00%	   230		 -
# 9  Conveyance offline  Completed without error	   00%		51		 -
#10  Extended offline	Completed without error	   00%		45		 -
#11  Short offline	   Completed without error	   00%		32		 -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

==========
da6
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Western Digital Red
Device Model:	 WDC WD60EFRX-68L0BN1
Serial Number:	WD-WXN1H26PKWHH
LU WWN Device Id: 5 0014ee 2635b7faf
Firmware Version: 82.00A82
User Capacity:	6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	5700 rpm
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Sat Mar  4 23:00:15 2017 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection:		 ( 2624) seconds.
Offline data collection
capabilities:			  (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time:	  (   2) minutes.
Extended self-test routine
recommended polling time:	  ( 680) minutes.
Conveyance self-test routine
recommended polling time:	  (   5) minutes.
SCT capabilities:			(0x303d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x002f   200   200   051	Pre-fail  Always	   -	   0
  3 Spin_Up_Time			0x0027   219   197   021	Pre-fail  Always	   -	   8016
  4 Start_Stop_Count		0x0032   100   100   000	Old_age   Always	   -	   30
  5 Reallocated_Sector_Ct   0x0033   200   200   140	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x002e   200   200   000	Old_age   Always	   -	   0
  9 Power_On_Hours		  0x0032   100   100   000	Old_age   Always	   -	   729
10 Spin_Retry_Count		0x0032   100   253   000	Old_age   Always	   -	   0
11 Calibration_Retry_Count 0x0032   100   253   000	Old_age   Always	   -	   0
12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   30
192 Power-Off_Retract_Count 0x0032   200   200   000	Old_age   Always	   -	   28
193 Load_Cycle_Count		0x0032   200   200   000	Old_age   Always	   -	   11
194 Temperature_Celsius	 0x0022   119   115   000	Old_age   Always	   -	   33
196 Reallocated_Event_Count 0x0032   200   200   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0032   200   200   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0030   100   253   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x0032   200   200   000	Old_age   Always	   -	   2
200 Multi_Zone_Error_Rate   0x0008   200   200   000	Old_age   Offline	  -	   0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline	Completed without error	   00%	   728		 -
# 2  Short offline	   Completed without error	   00%	   715		 -
# 3  Extended offline	Completed without error	   00%	   508		 -
# 4  Extended offline	Completed without error	   00%	   384		 -
# 5  Short offline	   Completed without error	   00%	   292		 -
# 6  Short offline	   Completed without error	   00%	   236		 -
# 7  Short offline	   Completed without error	   00%	   235		 -
# 8  Short offline	   Completed without error	   00%	   223		 -
# 9  Conveyance offline  Completed without error	   00%		51		 -
#10  Extended offline	Completed without error	   00%		44		 -
#11  Short offline	   Completed without error	   00%		32		 -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

==========
da7
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Western Digital Red
Device Model:	 WDC WD60EFRX-68L0BN1
Serial Number:	WD-WX31D96KN7P5
LU WWN Device Id: 5 0014ee 2b8aee02b
Firmware Version: 82.00A82
User Capacity:	6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	5700 rpm
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Sat Mar  4 23:00:15 2017 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection:		 ( 6344) seconds.
Offline data collection
capabilities:			  (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time:	  (   2) minutes.
Extended self-test routine
recommended polling time:	  ( 717) minutes.
Conveyance self-test routine
recommended polling time:	  (   5) minutes.
SCT capabilities:			(0x303d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x002f   200   200   051	Pre-fail  Always	   -	   0
  3 Spin_Up_Time			0x0027   217   197   021	Pre-fail  Always	   -	   8150
  4 Start_Stop_Count		0x0032   100   100   000	Old_age   Always	   -	   30
  5 Reallocated_Sector_Ct   0x0033   200   200   140	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x002e   200   200   000	Old_age   Always	   -	   0
  9 Power_On_Hours		  0x0032   099   099   000	Old_age   Always	   -	   734
10 Spin_Retry_Count		0x0032   100   253   000	Old_age   Always	   -	   0
11 Calibration_Retry_Count 0x0032   100   253   000	Old_age   Always	   -	   0
12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   30
192 Power-Off_Retract_Count 0x0032   200   200   000	Old_age   Always	   -	   28
193 Load_Cycle_Count		0x0032   200   200   000	Old_age   Always	   -	   11
194 Temperature_Celsius	 0x0022   119   117   000	Old_age   Always	   -	   33
196 Reallocated_Event_Count 0x0032   200   200   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0032   200   200   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0030   100   253   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x0032   200   200   000	Old_age   Always	   -	   6
200 Multi_Zone_Error_Rate   0x0008   200   200   000	Old_age   Offline	  -	   0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline	Completed without error	   00%	   734		 -
# 2  Short offline	   Completed without error	   00%	   720		 -
# 3  Extended offline	Completed without error	   00%	   514		 -
# 4  Extended offline	Completed without error	   00%	   390		 -
# 5  Short offline	   Completed without error	   00%	   298		 -
# 6  Short offline	   Completed without error	   00%	   242		 -
# 7  Short offline	   Completed without error	   00%	   241		 -
# 8  Short offline	   Completed without error	   00%	   229		 -
# 9  Conveyance offline  Completed without error	   00%		51		 -
#10  Extended offline	Completed without error	   00%		45		 -
#11  Short offline	   Completed without error	   00%		32		 -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

==========
ada0
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Western Digital Red
Device Model:	 WDC WD60EFRX-68L0BN1
Serial Number:	WD-WX31D96KN4SP
LU WWN Device Id: 5 0014ee 20e062b3a
Firmware Version: 82.00A82
User Capacity:	6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	5700 rpm
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Sat Mar  4 23:00:15 2017 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection:		 ( 6824) seconds.
Offline data collection
capabilities:			  (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time:	  (   2) minutes.
Extended self-test routine
recommended polling time:	  ( 722) minutes.
Conveyance self-test routine
recommended polling time:	  (   5) minutes.
SCT capabilities:			(0x303d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x002f   200   200   051	Pre-fail  Always	   -	   0
  3 Spin_Up_Time			0x0027   206   198   021	Pre-fail  Always	   -	   8666
  4 Start_Stop_Count		0x0032   100   100   000	Old_age   Always	   -	   29
  5 Reallocated_Sector_Ct   0x0033   200   200   140	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x002e   200   200   000	Old_age   Always	   -	   0
  9 Power_On_Hours		  0x0032   099   099   000	Old_age   Always	   -	   776
10 Spin_Retry_Count		0x0032   100   253   000	Old_age   Always	   -	   0
11 Calibration_Retry_Count 0x0032   100   253   000	Old_age   Always	   -	   0
12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   29
192 Power-Off_Retract_Count 0x0032   200   200   000	Old_age   Always	   -	   12
193 Load_Cycle_Count		0x0032   200   200   000	Old_age   Always	   -	   27
194 Temperature_Celsius	 0x0022   122   117   000	Old_age   Always	   -	   30
196 Reallocated_Event_Count 0x0032   200   200   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0032   200   200   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0030   100   253   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x0032   200   200   000	Old_age   Always	   -	   0
200 Multi_Zone_Error_Rate   0x0008   200   200   000	Old_age   Offline	  -	   0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline	Completed without error	   00%	   776		 -
# 2  Short offline	   Completed without error	   00%	   762		 -
# 3  Extended offline	Completed without error	   00%	   555		 -
# 4  Extended offline	Completed without error	   00%	   430		 -
# 5  Short offline	   Completed without error	   00%	   338		 -
# 6  Short offline	   Completed without error	   00%	   282		 -
# 7  Short offline	   Completed without error	   00%	   281		 -
# 8  Short offline	   Completed without error	   00%	   269		 -
# 9  Conveyance offline  Completed without error	   00%		52		 -
#10  Extended offline	Completed without error	   00%		46		 -
#11  Short offline	   Completed without error	   00%		33		 -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

==========
ada1
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Western Digital Red
Device Model:	 WDC WD60EFRX-68L0BN1
Serial Number:	WD-WXJ1H26M2MNF
LU WWN Device Id: 5 0014ee 2b8aef626
Firmware Version: 82.00A82
User Capacity:	6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	5700 rpm
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Sat Mar  4 23:00:15 2017 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection:		 ( 2624) seconds.
Offline data collection
capabilities:			  (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time:	  (   2) minutes.
Extended self-test routine
recommended polling time:	  ( 680) minutes.
Conveyance self-test routine
recommended polling time:	  (   5) minutes.
SCT capabilities:			(0x303d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x002f   200   200   051	Pre-fail  Always	   -	   0
  3 Spin_Up_Time			0x0027   206   198   021	Pre-fail  Always	   -	   8691
  4 Start_Stop_Count		0x0032   100   100   000	Old_age   Always	   -	   31
  5 Reallocated_Sector_Ct   0x0033   200   200   140	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x002e   200   200   000	Old_age   Always	   -	   0
  9 Power_On_Hours		  0x0032   099   099   000	Old_age   Always	   -	   776
10 Spin_Retry_Count		0x0032   100   253   000	Old_age   Always	   -	   0
11 Calibration_Retry_Count 0x0032   100   253   000	Old_age   Always	   -	   0
12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   31
192 Power-Off_Retract_Count 0x0032   200   200   000	Old_age   Always	   -	   13
193 Load_Cycle_Count		0x0032   200   200   000	Old_age   Always	   -	   28
194 Temperature_Celsius	 0x0022   119   114   000	Old_age   Always	   -	   33
196 Reallocated_Event_Count 0x0032   200   200   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0032   200   200   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0030   100   253   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x0032   200   200   000	Old_age   Always	   -	   0
200 Multi_Zone_Error_Rate   0x0008   200   200   000	Old_age   Offline	  -	   0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline	Completed without error	   00%	   775		 -
# 2  Short offline	   Completed without error	   00%	   762		 -
# 3  Extended offline	Completed without error	   00%	   555		 -
# 4  Extended offline	Completed without error	   00%	   429		 -
# 5  Short offline	   Completed without error	   00%	   337		 -
# 6  Short offline	   Completed without error	   00%	   282		 -
# 7  Short offline	   Completed without error	   00%	   281		 -
# 8  Short offline	   Completed without error	   00%	   269		 -
# 9  Conveyance offline  Completed without error	   00%		52		 -
#10  Extended offline	Completed without error	   00%		45		 -
#11  Short offline	   Completed without error	   00%		32		 -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
 
T

tealcomp

Explorer
Joined
Sep 5, 2016
Messages
59
And zpool -v output

Code:
jupiter# zpool status -v
  pool: freenas-boot
state: ONLINE
  scan: none requested
config:

	NAME		STATE	 READ WRITE CKSUM
	freenas-boot  ONLINE	   0	 0	 0
	  mirror-0  ONLINE	   0	 0	 0
		ada2p2  ONLINE	   0	 0	 0
		ada3p2  ONLINE	   0	 0	 0

errors: No known data errors

  pool: jovian
state: ONLINE
  scan: scrub repaired 2.79M in 5h41m with 0 errors on Sat Mar  4 05:30:43 2017
config:

	NAME												STATE	 READ WRITE CKSUM
	jovian											  ONLINE	   0	 0	 0
	  raidz3-0										  ONLINE	   0	 0	 0
		gptid/0f5c2d8b-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/100edac5-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/10cd817e-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/118f126a-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/1252f2b2-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/131ab491-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/13ef3990-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/14ae12fc-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/156d3a72-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/163621cb-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0

errors: No known data errors
jupiter#
 
JohnDigital

JohnDigital

Guru
Joined
Jan 7, 2015
Messages
1,155
I too vote power on this one. Id lean toward cables and such like usual, but the fact its split between onboard and M1015 ports to me is the giveaway. Is there anything in the IPMI SEL logs? I had a very similar issue once on an Antec 1000w power supply and in the IPMI system logs, there were errors dealing with power corresponding to the timestamps on the camstatus errors. Have a look in there and see what you see.
 
C

CraigD

Patron
Joined
Mar 8, 2016
Messages
343
tealcomp said:
the drives have never been over 39'C
Click to expand...

I used to think this about my drives, Imagine my surprise
Code:
root@freenas] ~# smartctl -l scttemp /dev/ada0

Current Temperature:  28 Celsius
Power Cycle Min/Max Temperature:  25/32 Celsius
Lifetime  Min/Max Temperature:  14/51 Celsius
Under/Over Temperature Limit Count:  0/0


Since been corrected

Have Fun
 
T

tealcomp

Explorer
Joined
Sep 5, 2016
Messages
59
And zpool -v output

Code:
jupiter# zpool status -v
  pool: freenas-boot
state: ONLINE
  scan: none requested
config:

	NAME		STATE	 READ WRITE CKSUM
	freenas-boot  ONLINE	   0	 0	 0
	  mirror-0  ONLINE	   0	 0	 0
		ada2p2  ONLINE	   0	 0	 0
		ada3p2  ONLINE	   0	 0	 0

errors: No known data errors

  pool: jovian
state: ONLINE
  scan: scrub repaired 2.79M in 5h41m with 0 errors on Sat Mar  4 05:30:43 2017
config:

	NAME												STATE	 READ WRITE CKSUM
	jovian											  ONLINE	   0	 0	 0
	  raidz3-0										  ONLINE	   0	 0	 0
		gptid/0f5c2d8b-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/100edac5-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/10cd817e-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/118f126a-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/1252f2b2-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/131ab491-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/13ef3990-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/14ae12fc-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/156d3a72-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/163621cb-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0

errors: No known data errors
jupiter#

CraigD said:
I used to think this about my drives, Imagine my surprise
Code:
root@freenas] ~# smartctl -l scttemp /dev/ada0

Current Temperature:  28 Celsius
Power Cycle Min/Max Temperature:  25/32 Celsius
Lifetime  Min/Max Temperature:  14/51 Celsius
Under/Over Temperature Limit Count:  0/0


Since been corrected

Have Fun
Click to expand...
CraigD said:
I used to think this about my drives, Imagine my surprise
Code:
root@freenas] ~# smartctl -l scttemp /dev/ada0

Current Temperature:  28 Celsius
Power Cycle Min/Max Temperature:  25/32 Celsius
Lifetime  Min/Max Temperature:  14/51 Celsius
Under/Over Temperature Limit Count:  0/0


Since been corrected

Have Fun
Click to expand...

Craig:

I base that information on the smartctl -x output I have kept track of. I will run the -scttemp switch to see if the values correspond. At any rate these drives are a little over a month old so I have been keeping an eagle eye on all aspects of the system.

-Dan
 
T

tealcomp

Explorer
Joined
Sep 5, 2016
Messages
59
John Digital said:
I too vote power on this one. Id lean toward cables and such like usual, but the fact its split between onboard and M1015 ports to me is the giveaway. Is there anything in the IPMI SEL logs? I had a very similar issue once on an Antec 1000w power supply and in the IPMI system logs, there were errors dealing with power corresponding to the timestamps on the camstatus errors. Have a look in there and see what you see.
Click to expand...

John!

My FreeNAS guardian :) I have been monitoring the IPMI gui logs and I was coming up empty. Are those the logs you are referring to, or was there something from terminal I can take a look at? If it is power, it would likely be the PSU itself on the 12v rail is to blame, since I am not feeding all (10) drives from the same cables, there are (3) separate cables feeding them.

I am about ready to start another scrub to see if I can replicate this behavior. The good news is we follow very stringent backup policies so the data is not in any jeopardy

-Dan
 
T

tealcomp

Explorer
Joined
Sep 5, 2016
Messages
59
tealcomp said:
And zpool -v output

Code:
jupiter# zpool status -v
  pool: freenas-boot
state: ONLINE
  scan: none requested
config:

	NAME		STATE	 READ WRITE CKSUM
	freenas-boot  ONLINE	   0	 0	 0
	  mirror-0  ONLINE	   0	 0	 0
		ada2p2  ONLINE	   0	 0	 0
		ada3p2  ONLINE	   0	 0	 0

errors: No known data errors

  pool: jovian
state: ONLINE
  scan: scrub repaired 2.79M in 5h41m with 0 errors on Sat Mar  4 05:30:43 2017
config:

	NAME												STATE	 READ WRITE CKSUM
	jovian											  ONLINE	   0	 0	 0
	  raidz3-0										  ONLINE	   0	 0	 0
		gptid/0f5c2d8b-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/100edac5-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/10cd817e-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/118f126a-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/1252f2b2-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/131ab491-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/13ef3990-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/14ae12fc-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/156d3a72-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0
		gptid/163621cb-fb93-11e6-abcc-0cc47a8668da.eli  ONLINE	   0	 0	 0

errors: No known data errors
jupiter#




Craig:

I base that information on the smartctl -x output I have kept track of. I will run the -scttemp switch to see if the values correspond. At any rate these drives are a little over a month old so I have been keeping an eagle eye on all aspects of the system.

-Dan
Click to expand...

Craig:

Thanks for helping me rule out the temperature another way!

Code:
for x in da{0..7} ada{0,1}; do echo ${x}; smartctl -l scttemp /dev/${x} | grep Lifetime; done


Code:
da0
Lifetime  Min/Max Temperature:  20/38 Celsius
da1
Lifetime  Min/Max Temperature:  21/38 Celsius
da2
Lifetime  Min/Max Temperature:  21/37 Celsius
da3
Lifetime  Min/Max Temperature:  21/36 Celsius
da4
Lifetime  Min/Max Temperature:  21/37 Celsius
da5
Lifetime  Min/Max Temperature:  20/37 Celsius
da6
Lifetime  Min/Max Temperature:  20/37 Celsius
da7
Lifetime  Min/Max Temperature:  21/35 Celsius
ada0
Lifetime  Min/Max Temperature:  22/35 Celsius
ada1
Lifetime  Min/Max Temperature:  21/38 Celsius
 
Jailer

Jailer

Not strong, but bad
Joined
Sep 12, 2014
Messages
4,977
Power may be the issue but I would start with cabling. Your errors are all UDMA CRC errors and that's usually, but not always, indicative of bad cabling.
 
T

tealcomp

Explorer
Joined
Sep 5, 2016
Messages
59
Jailer said:
Power may be the issue but I would start with cabling. Your errors are all UDMA CRC errors and that's usually, but not always, indicative of bad cabling.
Click to expand...

That was the first thought here as well; BUT, one of the (2) other drives connected to the motherboard was also showing an error. The error is slightly different, but given it is attached to a SATA port and not the M1015, ultimately that is what led us down the power side of things. Still, it's not going to be fun to diagnose. Data cables are easy LOL, so will most likely start there. If that fails, guess reach out to Seasonic and see what they have to offer; though this would be the first time there was a Seasonic issue.

-Dan
 
JohnDigital

JohnDigital

Guru
Joined
Jan 7, 2015
Messages
1,155
I think if you log in to ipmi from a web browser then go into "server health" and have a look around in there you can view sensor readings and every error the BMC has encountered and such. This is where im talking about. If all are green and there are no sorts of errors in the SEL logs, certainly this means the server isnt aware of these problems, then you might look at cabling/HBA. Is it possible that errors from the drives connected via the HBA cause ada0 to throw a related error? Try a different pcie slot too while you fiddle. Physically touch the HBA heatsink during a scrub is it merely warm or do you have blisters now? Reseat SFF cables. Check, replace, RMA things common to the 6 drives in question. The highest common denominator is power, then HBA, cabling, drives, so on down the line.
 
T

tealcomp

Explorer
Joined
Sep 5, 2016
Messages
59
Hey John:

So ran another scrub and sure enough, boom, error city; well at least it is not one of those spurious things that leaves me questioning whether I have it fixed or not.
Checked IPMI logs and nothing useful in IPMI, or should I say outside of the norm.
Ventured to touch the HBA during the scrub and frankly it felt warm, but certainly not hot..fingers still intact!
Since this debacle originally came about because of that #*^$#&^ switch in the Fractal case, I have disconnected those fans and for now plugged them directly into the motherboard. So, that switch is now out of the running as a component of my problems :) I am going to order new breakout cables just because I am not thrilled with these current ones. This will be the third set, so hoping third time is the charm. I will post results as they happen.

-Dan
 
T

tealcomp

Explorer
Joined
Sep 5, 2016
Messages
59
OK, so update...can at least now say, it's NOT the fan switch :) However, this time both of the drives attached to the motherboard received the same error, whereas before it was only one of them. So, looking at these cables to replace the current ones connecting the (8) drives to the HBA. Their big selling point is the 90 degree turn on the data connector which should reduce any unnecessary cable bends.

http://store.supermicro.com/cable/sas-sata/90cm-minisas-sata-cbl-0388l.html

Supermicro was the only place that had these cables, but unfortunately they also have the sideband cable which isn't needed. Based on reading though, these should be fine. Any thoughts or other suggestions?

-Dan
 
You must log in or register to reply here.

Similar threads

A
  • Locked
SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
Replies
13
Views
8K
antsrealm
A
patrick sullivan
  • Locked
nastest.local.nastest daily security run output error with log
Replies
3
Views
933
patrick sullivan
patrick sullivan
R
Help with the following error message:
Replies
12
Views
3K
Jailer
Jailer
J
  • Locked
SOLVED CAM status: SCSI Status Error ABORTED COMMAND NAK received
Replies
4
Views
5K
kdragon75
kdragon75
M
  • Locked
CAM status: SCSI Status Error in dmesg
Replies
1
Views
1K
Chris Moore
Chris Moore
Top