SMART on 3ware controller.

cyberjock · Aug 20, 2012

I'm trying to setup a FreeNAS server with a 9650SE-24M8 controller. Currently the controller is setup with JBOD for all drives. FreeNAS is setup with 3 drives temporarily using a RAIDZ while I experiment. However i'm having trouble getting SMART to work. Currently if I go to Storage -> Volumes -> View Disks I can see the disks. I do not get serial numbers in the GUI, not sure if this is expected or an indication of the problem. I did verify the box is checked for "Enable SMART". I've been trying to do SMART commands from the prompt to verify the hard drive supports SMART.

So I did smartctl -a -d 3ware,0 /dev/twa0

Code:

smartctl 5.42 2011-10-20 r3458 [FreeBSD 8.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Green
Device Model:     WDC WD15EADS-00S2B0
Serial Number:    WD-WCAVY0394928
LU WWN Device Id: 5 0014ee 258494bd0
Firmware Version: 01.00A01
User Capacity:    1,500,301,910,016 bytes [1.50 TB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Tue Aug 21 00:17:10 2012 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84)	Offline data collection activity
					was suspended by an interrupting command from host.
					Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(31560) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 255) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x303f)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       1
  3 Spin_Up_Time            0x0027   152   149   021    Pre-fail  Always       -       9400
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       24
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       35
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       23
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       14
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       115
194 Temperature_Celsius     0x0022   104   104   000    Old_age   Always       -       48
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

So now that I've proved the hard drive is supported it's time to give SMART a try! I click to start the service and I get the error:

Code:

Configuration file /usr/local/etc/smartd.conf parsed but has no entries (like /dev/hda)

Then the service turns back to off in the GUI.

So I checked out /usr/local/etc/smartd.conf and sure enough there's no entries.

I then tried adding -d 3ware,0 /dev/twa0 to Storage -> Volumes -> View Disks -> edit da0. Still the same error.
Then I tried adding a SMART Test to run by going to System -> SMART Tests -> Add test and created a conveyance test. I selected the 1 disk I'm experimenting with and the service still gives the same error on startup.

I've spent the last 3 hours with google and forum searches and I see alot of people have problems like mine but no definitive answer on how to fix it. A few people said they created startup scripts and whatnot but I couldn't find an entry that provided enough information that I could reproduce the result.

I tried creating entries manually, but FreeNAS seems to regenerate the smartd.conf file erasing my changes before the service starts. This was expected because of my searching but I figured I had nothing to lose.

Any ideas how I can troubleshoot this problem or how to resolve it? I'm half expecting someone is going to paste a link to some thread on how to fix it in 2 easy steps. There's alot of threads dealing with SMART issues but I'm pretty sure I've read them all at least once between all the different search terms I've used.

jgreco · Aug 21, 2012

Getting SMART running on the 3Ware cards is a matter of getting the configuration right. As you noted, FreeNAS regenerates the configuration prior to starting it. My recollection is that you can manually create a smartd.conf and then manually start smartd, and if you do and that works, you might want to drop a bug report about it to see if the developers would put in a conf file fix.

William Grzybowski · Aug 21, 2012

Can you post the output of # camcontrol devlist ?

cyberjock · Aug 21, 2012

Code:

<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 0 lun 0 (pass0,da0)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 1 lun 0 (pass1,da1)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 2 lun 0 (pass2,da2)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 3 lun 0 (pass3,da3)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 4 lun 0 (pass4,da4)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 5 lun 0 (pass5,da5)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 6 lun 0 (pass6,da6)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 7 lun 0 (pass7,da7)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 8 lun 0 (pass8,da8)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 9 lun 0 (pass9,da9)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 10 lun 0 (pass10,da10)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 11 lun 0 (pass11,da11)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 12 lun 0 (pass12,da12)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 13 lun 0 (pass13,da13)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 14 lun 0 (pass14,da14)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 15 lun 0 (pass15,da15)
<Corsair Voyager 1100>             at scbus11 target 0 lun 0 (pass16,da16)

Edit: At the time I ran camcontrol I had 16 disks installed. Disks 0 through 7 are each in their own RAID0(total of 8 RAID0s... don't ask LOL) and disks 8-15 are JBOD. It appears that whether the disk is JBOD or part of an array they are all hiding behind the controller. If I do smartctl -a -d 3ware,XX /dev/twa0 I can actually get data for all 16 drives and it is accurate. /shrug

@jgreco- From the forum posts it sounds like I just make the conf file and have a script that overwrites the blank conf with the one I make and then start smartd? What command starts smartd? I tried # smartd but it just goes back to a prompt and I'm not sure if it's actually running and I don't know how to check for it.

jgreco · Aug 21, 2012

noobsauce80 said:
Edit: At the time I ran camcontrol I had 16 disks installed. Disks 0 through 7 are each in their own RAID0(total of 8 RAID0s... don't ask LOL) and disks 8-15 are JBOD. It appears that whether the disk is JBOD or part of an array they are all hiding behind the controller. If I do smartctl -a -d 3ware,XX /dev/twa0 I can actually get data for all 16 drives and it is accurate. /shrug

@jgreco- From the forum posts it sounds like I just make the conf file and have a script that overwrites the blank conf with the one I make and then start smartd? What command starts smartd? I tried # smartd but it just goes back to a prompt and I'm not sure if it's actually running and I don't know how to check for it.

If you happen to have a FreeNAS system *without* the 3Ware controller but *with* SMART set up, go out to the CLI, do something like "ps agxlww|grep smartd" to see the system's chosen invocation for smartd, and you will have an idea what a running setup should look like. Also grab that smartd.conf to use as a starting point.

Since smartd can be invoked with an explicit config file (like "smartd -c /usr/local/etc/smartd.conf") we have the option of not bothering with the system-generated one; put one over on your datastore, like /mnt/datastore/etc/smartd.conf and play with it there. Then, what I am guessing is most likely to work is to tell FreeNAS to start smartd, which will try to launch it but fail. If there are any unexpected dependencies, this is likely to make sure that the box will "take" your manual invocation of smartd. This may be completely unnecessary, but it's what I'd try as part of my first round. Anyways, once you do that, then try a manual invocation of smartd pointing to your local smartd.conf, and I'm guessing that's extremely likely to work - if your smartd.conf is correct, at least.

You can check that smartd is running with the ps command provided above.

While I do have a 3Ware-based FreeNAS box, I'm not going to mess with something handling production traffic to actually check this out, sorry, so I'm just giving you the general outline so that you can experiment for yourself.

Let me know how it goes. If you can't make it work, I think I have another 3Ware chassis sitting around here somewhere that could be made good-to-go in short order, but things are a bit busy right now.

cyberjock · Aug 21, 2012

jgreco said:
If you happen to have a FreeNAS system *without* the 3Ware controller but *with* SMART set up, go out to the CLI, do something like "ps agxlww|grep smartd" to see the system's chosen invocation for smartd, and you will have an idea what a running setup should look like. Also grab that smartd.conf to use as a starting point.

Since smartd can be invoked with an explicit config file (like "smartd -c /usr/local/etc/smartd.conf") we have the option of not bothering with the system-generated one; put one over on your datastore, like /mnt/datastore/etc/smartd.conf and play with it there. Then, what I am guessing is most likely to work is to tell FreeNAS to start smartd, which will try to launch it but fail. If there are any unexpected dependencies, this is likely to make sure that the box will "take" your manual invocation of smartd. This may be completely unnecessary, but it's what I'd try as part of my first round. Anyways, once you do that, then try a manual invocation of smartd pointing to your local smartd.conf, and I'm guessing that's extremely likely to work - if your smartd.conf is correct, at least.

You can check that smartd is running with the ps command provided above.

While I do have a 3Ware-based FreeNAS box, I'm not going to mess with something handling production traffic to actually check this out, sorry, so I'm just giving you the general outline so that you can experiment for yourself.

Let me know how it goes. If you can't make it work, I think I have another 3Ware chassis sitting around here somewhere that could be made good-to-go in short order, but things are a bit busy right now.

I prefer it this way. Then I have to learn to feed myself :)

cyberjock · Aug 21, 2012

So I did some searching. Using my laptop with a spare USB key I was able to get some info. Smartd runs as /usr/local/sbin/smartd -p /var/run/smartd.pid -i 1800

Viewing the /usr/local/etc/smartd.conf I found:

Code:

/dev/ada0 -n never -W 0,0,0 -m myemail@gmail.com

So I think I can figure all of this out ;) I'm not where the server is right now but I've got a good idea of how to do it now. My test config file will be:

Code:

/dev/twa0 -d 3ware,0 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,1 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,2 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,3 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,4 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,5 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,6 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,7 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,8 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,9 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,10 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,11 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,12 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,13 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,14 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,15 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,16 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,17 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,18 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,19 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,20 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,21 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,22 -n never -W 0,0,0 -m myemail@gmail.com
/dev/twa0 -d 3ware,23 -n never -W 0,0,0 -m myemail@gmail.com

Then I'll have smartd invoked by using the command /usr/local/sbin/smartd -p /var/run/smartd.pid -i 1800 -c /mnt/tank/smartd.conf

Now that I think I have a handle on this.. how do I make this run at startup?

Also, based on my laptop I think it is reasonable to assume that the serial numbers won't magically fill in under Storage -> Volumes -> View Disks when I get smartd to manually load. My laptop serial number appeared the first time FreeNAS booted up using defaults. Is there any way to get that to work? The things I am sure of are that da0 matches to port 0, da1 matches to part 1, etc unless you are using the 3ware hardware RAID arrays(which I won't be). However, if you skip a port, then they'll all be off by one after that. I was really hoping to get serial numbers to work.

I know that any SMART extra options I add under "Edit" for the view disks screen does move with the drive. If I swap 2 disks the extra options follow it so somehow the system is tracking the hardware specifically despite them all being the same exact model.

Back when i was experimenting with the highpoint controllers in non-raid mode I don't believe the serial numbers were appearing their either, but I'd have to double check. I was more concerned with fixing the driver issue because my controller wasn't being acknowledged, which it is now in the nightly builds :)

cyberjock · Aug 21, 2012

Just for anyones knowledge, the latest nightly build doesn't work "out of the box" either. I wasn't expecting it to, but since I had a USB I figured I'd try it.

jgreco · Aug 21, 2012

noobsauce80 said:

Well, at least on earlier FreeNAS's, what's generating this is /conf/base/etc/rc.d/ix-smartd, and you have this

Code:

disk=$(/usr/local/bin/python /usr/local/www/freenasUI/middleware/notifier.py identifier_to_device "${ident}")

and without studying how notifier.py:identifier_to_device works (it's ugly but then again I don't do much Python so my opinion is biased) it isn't clear what the practical options are. I'd say easiest path? File a bug report/feature request. In the meantime you can probably comment out generate_smartd_config and replace it with something like "cat /mnt/pool/foo/etc/yoursmartd.conf" or something like that, ugly but ought to work to force a static smartd.conf

cyberjock · Aug 21, 2012

So I did all the things above hoping it would work. I setup my own config file and attempted to start smartd. It runs for a few seconds, then goes back to CLI. When I do ps agxlww | grep smartd I get nothing. So apparently I'm doing something wrong or it won't work the way I had hoped. More experimenting and reading is in order!

jgreco · Aug 22, 2012

Try adding "-d -n" to your smartd options (--debug, --no-fork). Guessing it'll tell you what's making it unhappy.

cyberjock · Aug 22, 2012

I ran smartd -p /var/run/smartd.pid -i 1800 -c /mnt/harddisk/smartd.conf -d -n and I got:

Code:

smartd 5.42 2011-10-20 r3458 [FreeBSD 8.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=======> INVALID CHOICE OF OPTIONS: -d and -p <=======

Error: pid file /var/run/smartd.pid not written in debug (-d) mode

So I got rid of the -p and it starts and outputs a bunch of stuff..

Code:

smartd 5.42 2011-10-20 r3458 [FreeBSD 8.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

Opened configuration file /mnt/harddisk/smartd.conf
Drive: /dev/twa0, implied '-a' Directive on line 1 of file /mnt/harddisk/smartd.conf
Drive: /dev/twa0, implied '-a' Directive on line 2 of file /mnt/harddisk/smartd.conf
Drive: /dev/twa0, implied '-a' Directive on line 3 of file /mnt/harddisk/smartd.conf
Drive: /dev/twa0, implied '-a' Directive on line 4 of file /mnt/harddisk/smartd.conf
Drive: /dev/twa0, implied '-a' Directive on line 5 of file /mnt/harddisk/smartd.conf
Drive: /dev/twa0, implied '-a' Directive on line 6 of file /mnt/harddisk/smartd.conf
Drive: /dev/twa0, implied '-a' Directive on line 7 of file /mnt/harddisk/smartd.conf
Drive: /dev/twa0, implied '-a' Directive on line 8 of file /mnt/harddisk/smartd.conf
Drive: /dev/twa0, implied '-a' Directive on line 9 of file /mnt/harddisk/smartd.conf
Configuration file /mnt/harddisk/smartd.conf parsed.
Device: /dev/twa0 [3ware_disk_00], opened
Device: /dev/twa0 [3ware_disk_00], WDC WD15EADS-00S2B0, S/N:WD-WCAVY0394928, WWN:5-0014ee-258494bd0, FW:01.00A01, 1.50 TB
Device: /dev/twa0 [3ware_disk_00], found in smartd database: Western Digital Caviar Green
Device: /dev/twa0 [3ware_disk_00], is SMART capable. Adding to "monitor" list.
Device: /dev/twa0 [3ware_disk_01], opened
Device: /dev/twa0 [3ware_disk_01], ST31000340AS, S/N:9QJ0ZSNZ, WWN:5-000c50-00d45874d, FW:SD1A, 1.00 TB
Device: /dev/twa0 [3ware_disk_01], found in smartd database: Seagate Barracuda 7200.11
Device: /dev/twa0 [3ware_disk_01], is SMART capable. Adding to "monitor" list.
Device: /dev/twa0 [3ware_disk_02], opened
Device: /dev/twa0 [3ware_disk_02], ST31000528AS, S/N:6VP4QAHW, WWN:5-000c50-0280321ea, FW:CC38, 1.00 TB
Device: /dev/twa0 [3ware_disk_02], found in smartd database: Seagate Barracuda 7200.12
Device: /dev/twa0 [3ware_disk_02], is SMART capable. Adding to "monitor" list.
Device: /dev/twa0 [3ware_disk_03], opened
Device: /dev/twa0 [3ware_disk_03], ST31500341AS, S/N:9VS0NXRE, WWN:5-000c50-010a1145e, FW:CC1H, 1.50 TB
Device: /dev/twa0 [3ware_disk_03], found in smartd database: Seagate Barracuda 7200.11
Device: /dev/twa0 [3ware_disk_03], is SMART capable. Adding to "monitor" list.
Device: /dev/twa0 [3ware_disk_04], opened
Device: /dev/twa0 [3ware_disk_04], ST31000340AS, S/N:3QJ01YT1, WWN:5-000c50-00265a4e0, FW:SD04, 1.00 TB
Device: /dev/twa0 [3ware_disk_04], found in smartd database: Seagate Barracuda 7200.11
Device: /dev/twa0 [3ware_disk_04], WARNING: There are known problems with these drives,
see the following Seagate web pages:
http://seagate.custkb.com/seagate/crm/selfservice/search.jsp?DocId=207931
http://seagate.custkb.com/seagate/crm/selfservice/search.jsp?DocId=207951
http://seagate.custkb.com/seagate/crm/selfservice/search.jsp?DocId=207957
Device: /dev/twa0 [3ware_disk_04], is SMART capable. Adding to "monitor" list.
Device: /dev/twa0 [3ware_disk_05], opened
Device: /dev/twa0 [3ware_disk_05], ST31500341AS, S/N:9VS0NJVB, WWN:5-000c50-010b45f89, FW:CC1H, 1.50 TB
Device: /dev/twa0 [3ware_disk_05], found in smartd database: Seagate Barracuda 7200.11
Device: /dev/twa0 [3ware_disk_05], is SMART capable. Adding to "monitor" list.
Device: /dev/twa0 [3ware_disk_06], opened
Device: /dev/twa0 [3ware_disk_06], ST31500341AS, S/N:9VS0NKPE, WWN:5-000c50-010b45fa1, FW:CC1H, 1.50 TB
Device: /dev/twa0 [3ware_disk_06], found in smartd database: Seagate Barracuda 7200.11
Device: /dev/twa0 [3ware_disk_06], is SMART capable. Adding to "monitor" list.
Device: /dev/twa0 [3ware_disk_07], opened
Device: /dev/twa0 [3ware_disk_07], ST31000340AS, S/N:9QJ02WYC, WWN:5-000c50-0027a7224, FW:SD1A, 1.00 TB
Device: /dev/twa0 [3ware_disk_07], found in smartd database: Seagate Barracuda 7200.11
Device: /dev/twa0 [3ware_disk_07], is SMART capable. Adding to "monitor" list.
Device: /dev/twa0 [3ware_disk_08], opened
Device: /dev/twa0 [3ware_disk_08], ST31000340AS, S/N:5QJ03CZG, WWN:5-000c50-0028bc66b, FW:AD14, 1.00 TB
Device: /dev/twa0 [3ware_disk_08], found in smartd database: Seagate Barracuda 7200.11
Device: /dev/twa0 [3ware_disk_08], WARNING: There are known problems with these drives,
THIS DRIVE MAY OR MAY NOT BE AFFECTED,
see the following web pages for details:
http://seagate.custkb.com/seagate/crm/selfservice/search.jsp?DocId=207931
http://seagate.custkb.com/seagate/crm/selfservice/search.jsp?DocId=207951
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=632758
Device: /dev/twa0 [3ware_disk_08], is SMART capable. Adding to "monitor" list.
Monitoring 9 ATA and 0 SCSI devices
Device: /dev/twa0 [3ware_disk_00], opened ATA device
Device: /dev/twa0 [3ware_disk_00], previous self-test completed without error
Device: /dev/twa0 [3ware_disk_01], opened ATA device
Device: /dev/twa0 [3ware_disk_01], 3 Currently unreadable (pending) sectors
Sending warning via mail to myemail@email.com ...
Warning via mail to myemail@email.com: successful
Device: /dev/twa0 [3ware_disk_01], 3 Offline uncorrectable sectors
Sending warning via mail to myemail@email.com ...
Warning via mail to myemail@email.com: successful
Device: /dev/twa0 [3ware_disk_01], previous self-test completed without error
Device: /dev/twa0 [3ware_disk_02], opened ATA device
Device: /dev/twa0 [3ware_disk_02], previous self-test completed without error
Device: /dev/twa0 [3ware_disk_03], opened ATA device
Device: /dev/twa0 [3ware_disk_03], previous self-test completed without error
Device: /dev/twa0 [3ware_disk_04], opened ATA device
Device: /dev/twa0 [3ware_disk_04], previous self-test completed without error
Device: /dev/twa0 [3ware_disk_05], opened ATA device
Device: /dev/twa0 [3ware_disk_05], previous self-test completed without error
Device: /dev/twa0 [3ware_disk_06], opened ATA device
Device: /dev/twa0 [3ware_disk_06], previous self-test completed without error
Device: /dev/twa0 [3ware_disk_07], opened ATA device
Device: /dev/twa0 [3ware_disk_07], previous self-test completed without error
Device: /dev/twa0 [3ware_disk_08], opened ATA device
Device: /dev/twa0 [3ware_disk_08], previous self-test completed without error

There was more but I stopped there. It never exited to a prompt so I closed the terminal window. I did do a ps agxlww|grep smartd and it does appear to still be running. I'm guessing I got it working. Got an email from the bad drive and everything! I don't plan to use these drives in the final system. They were just spare drives. I'm going to do some more data gathering and see if I need to open a ticket for this issue or if there is an existing ticket that could benefit from my knowledge. I will have access to this server in the future so it does make a good platform for testing any fixes for 3ware controllers(or at least the 9000 series). I did try changing the conf to include all 23 ports hoping that if I just include them all it would ignore the empty ports. Unfortunately SMARTD exits with an error if no disk is attached. It is necessary to specify exactly which ports have hard disks attached.

Now to figure out how to get it to load this way at startup. ;) I need to find a simple way so that in the future when I do a FreeNAS update I can have someone run a script to "reinstall" the SMART bootup line or find a way to install it permanently. :D

I do not appear to get serial numbers for the hard disks when I "view disks" in FreeNAS GUI. :( I thought it was a far stretch to get serials when SMART loaded because my laptop worked perfectly without any configuration settings. Any way to fix that?

paleoN · Aug 23, 2012

What's the output of:

Code:

camcontrol devlist -v

It's what they parse in ix-smartd rev. 12158 in trunk. They recently added some Highpoint RR code to it. Perhaps you can use that to work out how to check only the ports with drives attached.

cyberjock · Aug 23, 2012

output of camcontrol devlist -v

Code:

scbus0 on ahcich0 bus 0:
<>                                 at scbus0 target -1 lun -1 ()
scbus1 on ahcich1 bus 0:
<>                                 at scbus1 target -1 lun -1 ()
scbus2 on ata2 bus 0:
<>                                 at scbus2 target -1 lun -1 ()
scbus3 on twa0 bus 0:
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 0 lun 0 (pass0,da0)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 1 lun 0 (pass1,da1)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 2 lun 0 (pass2,da2)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 3 lun 0 (pass3,da3)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 4 lun 0 (pass4,da4)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 5 lun 0 (pass5,da5)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 6 lun 0 (pass6,da6)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 7 lun 0 (pass7,da7)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 8 lun 0 (pass8,da8)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 9 lun 0 (pass9,da9)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 10 lun 0 (pass10,da10)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 11 lun 0 (pass11,da11)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 12 lun 0 (pass12,da12)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 13 lun 0 (pass13,da13)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 14 lun 0 (pass14,da14)
<AMCC 9650SE-24M DISK 4.10>        at scbus3 target 15 lun 0 (pass15,da15)
<>                                 at scbus3 target -1 lun -1 ()
scbus4 on sbp0 bus 0:
<>                                 at scbus4 target -1 lun -1 ()
scbus5 on ahcich2 bus 0:
<>                                 at scbus5 target -1 lun -1 ()
scbus6 on ahcich3 bus 0:
<>                                 at scbus6 target -1 lun -1 ()
scbus7 on ahcich4 bus 0:
<>                                 at scbus7 target -1 lun -1 ()
scbus8 on ahcich5 bus 0:
<>                                 at scbus8 target -1 lun -1 ()
scbus9 on ahcich6 bus 0:
<>                                 at scbus9 target -1 lun -1 ()
scbus10 on ahcich7 bus 0:
<>                                 at scbus10 target -1 lun -1 ()
scbus11 on umass-sim0 bus 0:
<Corsair Voyager 1100>             at scbus11 target 0 lun 0 (pass16,da16)
scbus-1 on xpt0 bus 0:
<>                                 at scbus-1 target -1 lun -1 (xpt0)

Note: The physical ports actually in use are 0-13, 19 and 23 at this time. That information does not appear to be able to be extracted from the information above.

cyberjock · Aug 23, 2012

paleoN said:
What's the output of:
Code:
camcontrol devlist -v

It's what they parse in ix-smartd rev. 12158 in trunk. They recently added some Highpoint RR code to it. Perhaps you can use that to work out how to check only the ports with drives attached.

I did go looking and found ticket https://support.freenas.org/ticket/1074. I'm not fluent on all the code, but I did piece out the part that I could figure out. If I run smartctl -o /dev/twa0 -d 3ware,0 | grep "Serial Number:" I get:

Code:

Serial Number:    WD***************

Unfortunately I'm too new to bash to really know my way around

.

I'm definitely open to learning if given a direction. I'm having a blast learning all of this stuff. Never used awk before so I'm going to go read up on that. ;) It looks like we may be able to add another line below the highpoint entry to say something like "if 3ware then do this..."

paleoN · Aug 23, 2012

noobsauce80 said:
Note: The physical ports actually in use are 0-13, 19 and 23 at this time. That information does not appear to be able to be extracted from the information above.

Yes, according to the smart docs, "use the 3ware CLI or 3dm tool to determine which ports (physical disks) correspond to particular 3ware units."

Code:

tw_cli /cx show drivestatus

Where the x is the controller number, i.e. 0.

cyberjock · Aug 23, 2012

Damn, this just.. sucks.

To be honest I wish there was some kind of a config file where you could specify which disk was which device. Assuming you have no hard drives plugged into any other controllers and you have only 1 3ware controller(I'm not sure how the system handles 2x 3ware controllers simultaneously) but the devices are assigned from da0 on up, and the ports follow that logic if you are running JBOD.

For instance, if you have an 8 port 3ware controller with hard drives in ports 0,1,3,7,8 the conversion would be:

port 0 = da0
port 1 = da1
port 3 = da2
port 7 = da3
port 8 = da4

If you have any hardware RAIDs setup, those hard disks are listed as the first devices by port, then all JBOD after. Why you'd do a hardware RAID and JBOD at the same time is beyond me.

But if there was a way to manually inform the OS which was which, that could help for 3ware controllers as well as other controllers where you can't detect what port is what device.

Obviously if you have 10 hard disks and 11 devices(USB counts as the last one) and you were to just scan all 3ware ports from 0 to 24 with smartctl -o /dev/twa0 -d 3ware,0 | grep "Serial Number:" you could easily just match the serial numbers chronologically to the daXX devices. If no disk is attached you get no output so you know that physical port on the 3ware controller can be ignored. That seems to be the "best" way I can think of to "detect" what ports are being used on the controller.

Edit: Im impressed. The tw_cli is built into FreeNAS. I really figured it wouldn't be and that would make everything messy. Here's an output of # tw_cli /c0 show drivestatus

Code:

VPort Status         Unit Size      Type  Phy Encl-Slot    Model
------------------------------------------------------------------------------
p0    OK             u0   1.36 TB   SATA  0   -            WDC WD15EADS-00S2B0 
p1    OK             u1   931.51 GB SATA  1   -            ST31000340AS        
p2    OK             u2   931.51 GB SATA  2   -            ST31000528AS        
p3    OK             u3   1.36 TB   SATA  3   -            ST31500341AS        
p4    OK             u4   931.51 GB SATA  4   -            ST31000340AS        
p5    OK             u5   1.36 TB   SATA  5   -            ST31500341AS        
p6    OK             u6   1.36 TB   SATA  6   -            ST31500341AS        
p7    OK             u7   931.51 GB SATA  7   -            ST31000340AS        
p8    OK             u8   931.51 GB SATA  8   -            ST31000340AS        
p9    OK             u9   1.36 TB   SATA  9   -            ST31500341AS        
p10   OK             u10  931.51 GB SATA  10  -            ST31000528AS        
p11   OK             u11  1.36 TB   SATA  11  -            ST31500341AS        
p12   OK             u12  931.51 GB SATA  12  -            ST31000340AS        
p13   OK             u13  1.36 TB   SATA  13  -            ST31500341AS        
p19   OK             u14  931.51 GB SATA  19  -            ST31000340AS        
p23   OK             u15  1.36 TB   SATA  23  -            ST31500341AS

In my case u0 = da0, u1=da1, etc. up to da15. da16 = USB key.

paleoN · Aug 23, 2012

noobsauce80 said:
Edit: Im impressed. The tw_cli is built into FreeNAS. I really figured it wouldn't be and that would make everything messy.

That's why I suggested the command. ;) What's more it has it's own page in the documentation. I had forgotten about it myself, but I remembered seeing it when I viewed the cli utils page.

See if this will give you the serials as well.

Code:

tw_cli /c0 show all

If it does than you can just parse that instead of running tw_cli /c0 show drivestatus -> parsing the output -> running smartctl to pick up the serials.

My bash is a bit rusty and ix-smartd is a Bourne shell script, but with your outputs I may take a look at anyway. Probably won't get to it until tomorrow though.

cyberjock · Aug 23, 2012

tw_cli /co show all does not show serial numbers :(

Code:

/c0 Driver Version = 3.80.06.003
/c0 Model = 9650SE-24M8
/c0 Available Memory = 448MB
/c0 Firmware Version = FE9X 4.10.00.024
/c0 Bios Version = BE9X 4.08.00.004
/c0 Boot Loader Version = BL9X 3.08.00.001
/c0 Serial Number = ***************XX
/c0 PCB Version = Rev 032
/c0 PCHIP Version = 2.00
/c0 ACHIP Version = 1.90
/c0 Number of Ports = 24
/c0 Number of Drives = 16
/c0 Number of Units = 16
/c0 Total Optimal Units = 16
/c0 Not Optimal Units = 0 
/c0 JBOD Export Policy = on
/c0 Disk Spinup Policy = 1
/c0 Spinup Stagger Time Policy (sec) = 1
/c0 Auto-Carving Policy = off
/c0 Auto-Carving Size = 2048 GB
/c0 Auto-Rebuild Policy = off
/c0 Rebuild Mode = Adaptive
/c0 Rebuild Rate = 1
/c0 Verify Mode = Adaptive
/c0 Verify Rate = 1
/c0 Controller Bus Type = PCIe
/c0 Controller Bus Width = 4 lanes
/c0 Controller Bus Speed = 2.5 Gbps/lane

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    JBOD      OK             -       -       -       1396.97   Ri     OFF    
u1    JBOD      OK             -       -       -       931.312   Ri     OFF    
u2    JBOD      OK             -       -       -       1397.27   Ri     OFF    
u3    JBOD      OK             -       -       -       931.513   Ri     OFF    
u4    JBOD      OK             -       -       -       931.513   Ri     OFF    
u5    JBOD      OK             -       -       -       1397.27   Ri     OFF    
u6    JBOD      OK             -       -       -       931.513   Ri     OFF    
u7    JBOD      OK             -       -       -       1397.27   Ri     OFF    
u8    JBOD      OK             -       -       -       1397.27   Ri     OFF    
u9    JBOD      OK             -       -       -       931.513   Ri     OFF    
u10   JBOD      OK             -       -       -       931.513   Ri     OFF    
u11   JBOD      OK             -       -       -       1397.26   Ri     OFF    
u12   JBOD      OK             -       -       -       931.512   Ri     OFF    
u13   JBOD      OK             -       -       -       1397.27   Ri     OFF    
u14   JBOD      OK             -       -       -       931.513   Ri     OFF    
u15   JBOD      OK             -       -       -       1397.27   Ri     OFF    

VPort Status         Unit Size      Type  Phy Encl-Slot    Model
------------------------------------------------------------------------------
p0    OK             u0   1.36 TB   SATA  0   -            WDC WD15EADS-00S2B0 
p1    OK             u1   931.51 GB SATA  1   -            ST31000340AS        
p2    OK             u2   931.51 GB SATA  2   -            ST31000528AS        
p3    OK             u3   1.36 TB   SATA  3   -            ST31500341AS        
p4    OK             u4   931.51 GB SATA  4   -            ST31000340AS        
p5    OK             u5   1.36 TB   SATA  5   -            ST31500341AS        
p6    OK             u6   1.36 TB   SATA  6   -            ST31500341AS        
p7    OK             u7   931.51 GB SATA  7   -            ST31000340AS        
p8    OK             u8   931.51 GB SATA  8   -            ST31000340AS        
p9    OK             u9   1.36 TB   SATA  9   -            ST31500341AS        
p10   OK             u10  931.51 GB SATA  10  -            ST31000528AS        
p11   OK             u11  1.36 TB   SATA  11  -            ST31500341AS        
p12   OK             u12  931.51 GB SATA  12  -            ST31000340AS        
p13   OK             u13  1.36 TB   SATA  13  -            ST31500341AS        
p19   OK             u14  931.51 GB SATA  19  -            ST31000340AS        
p23   OK             u15  1.36 TB   SATA  23  -            ST31500341AS

cyberjock · Aug 23, 2012

I got something from google though. If i do tw_cli /c0/u0 show serial then I get:

Code:

/c0/u0 serial number = ******************

Obviously if we simply do a tw_cli /c0/u0 show serial all the way to /c0/u24 that would cover all ports. For ports that have no drives connected the error you get is:

Code:

Error: (CLI:006) Specified unit does not exist.

I'm still not sure how you could definitively prove u0 = da0 aside from simply assuming it :P

Edit: As a "safe" bet we could always do /c0/u0 to /c0/u126 since some 3ware SAS controllers support up to 127 devices with a SAS expander. Of course, I still haven't figured out how to easily convert the /uXX to /daXX. I'm still searching though!

Important Announcement for the TrueNAS Community.

SMART on 3ware controller.

Inactive Account

Resident Grinch

Wizard

Inactive Account

Resident Grinch

Inactive Account

Inactive Account

Inactive Account

Resident Grinch

Inactive Account

Resident Grinch

Inactive Account

Wizard

Inactive Account

Inactive Account

Wizard

Inactive Account

Wizard

Inactive Account

Inactive Account

Similar threads