Cannot access Files

[Stal00se]

· WARNING: The volume HOBFiles (ZFS) status is UNKNOWN


System Information

Hostname

freenas.local

Build

FreeNAS-8.3.1-RELEASE-p2-x64 (r12686+b770da6_dirty)

Platform

AMD Phenom(tm) 9450e Quad-Core Processor

Memory

8176MB

System Time

Thu Sep 17 12:11:35 PDT 2015

Uptime

12:11PM up 11 days, 23:34, 0 users

Load Average

0.10, 0.09, 0.04




Have ran this system with 4 640Gb Hard Drives for approximately 3+years. Recently I could not access my files. Unfortunately, I am not Linux skilled.


I can access the GUI for Freenas, but not the files themselves.


Question is, can I go with a new Freenas USB with the newest release and somehow have it recognize my array and files, or will it be necessary to go though several Linux command prompts to fix my system?

 

[dlavigne]

Before doing that, paste the output of zpool status (use Insert -> Code so it's readable).

 

[Stal00se]

I opened shell and typed zpool status. The reply was "no pools available".

[root@freenas ~]# zpool status
no pools available

 

[danb35]

OK, what about 'zpool import'?

BTW, FreeNAS has nothing to do with Linux--it's FreeBSD. Both are Unix-like operating systems, but they're definitely not the same thing.

 

[Stal00se]

Sorry, my lack of knowledge is showing. I simply put this box together a few years ago and never had the time to learn it. here is the output from zpool import.

[root@freenas ~]# zpool import
pool: HOBFiles
id: 14636708750071156672
state: FAULTED
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
devices and try again.
The pool may be active on another system, but can be imported using
the '-f' flag.
see: http://www.sun.com/msg/ZFS-8000-3C
config:

HOBFiles FAULTED corrupted data
raidz1-0 FAULTED corrupted data
803319005229190279 UNAVAIL cannot open
gptid/ab282c38-b102-11e2-b438-00306771456b ONLINE
gptid/abf0d07f-b102-11e2-b438-00306771456b ONLINE
gptid/acc4705b-b102-11e2-b438-00306771456b ONLINE
[root@freenas ~]#

 

[danb35]

It looks like one of your hard drives has died, and there's some problem with the pool data on the other drives. What's the output of 'camcontrol devlist'?

 

[Stal00se]

[root@freenas ~]# camcontrol devlist
<WDC WD6400AARS-00Y5B1 80.00A80> at scbus1 target 0 lun 0 (pass0,ada0)
<WDC WD6400AARS-00Y5B1 80.00A80> at scbus2 target 0 lun 0 (pass1,ada1)
<WDC WD6400AARS-00Y5B1 80.00A80> at scbus3 target 0 lun 0 (pass2,ada2)
<TSSTcorp CDDVDW SH-222BB SB00> at scbus4 target 0 lun 0 (pass3,cd0)
<Memorex Mini TD 001B PMAP> at scbus6 target 0 lun 0 (pass4,da0)

 

[SweetAndLow]

Looks like multiple disks have died. Can you post the output of smartctl -a /dev/ada[0-2].

Edit: actually I'm not sure the disks died. They seem to still be part of the pool but just corrupt.

 

[danb35]

One of the disks is definitely deceased (it's certainly not pining for the fjords)--or at a minimum, it's somehow gotten disconnected. You'll notice that zpool import shows that the pool has four disks (in RAIDZ1), but camcontrol devlist shows only three. But, yes, it looks like data from the remaining disks has gotten corrupted in some way, and the SMART output would be relevant in diagnosing that.

@Stal00se, when you provide the smartctl output, please follow Dru's request above to put it in code tags--click the More Options... button below the reply box, and from that editor, you'd be able to do Insert -> Code as she suggests.

 

[Glorious1]

I would try shutting down and detaching and reattaching the SATA cables. You never know, might be just a bad connection. Do you do regular scrubs and SMART tests?

 

[Stal00se]

Like I said, I never had the time to play with it. I am having a problem figuring out how to do smartctl -a /dev/ada[0-2]
I get an error back that says the command cannot do multiple devices
I can do one at a time ie. smartctl -a /dev/ada0
However, the text is too long and cannot be copied
is (use Insert -> Code so it's readable) part of the cli?

 

[Glorious1]

Right, you have to do smartctl one drive at a time. I don't know why you wouldn't be able to copy the output. Maybe you're doing this in the GUI shell instead of SSH? The Insert > Code is part of the forum website. Read danb35's message carefully. Also the 4th icon from the right in the large group above the reply box gives you the insert option.

 

[Stal00se]

My apologies, I cannot get into the freenas array using Putty. I only have the GUI. I assume the correct port to use in Putty is 22?

 

[Bidule0hm]

Yes, port 22.

You can also use smartctl -a /dev/ada0 | less and use up and down arrows to navigate in the output (and Q to quit).

Or if you have a working pool shared on the network you can also put the output in a file by doing this smartctl -a /dev/ada0 > /mnt/your_pool_name/your_dataset_name/a_file_name.txt ;)

 

[danb35]

To connect using ssh (i.e., putty or some other ssh client), the service needs to be enabled in the web GUI, and you probably also want to enable root logins. SSH really is a much better way of accessing the CLI than the shell from the web GUI--the web GUI's shell doesn't have scrollback (which you really need for what you're working on right now), it traps a lot of control characters, and just generally doesn't work all that well. It's fine for running a command or two, and it's definitely better than nothing, but a proper ssh client is much better.

 

[Stal00se]

Sorry I have not posted in the last 24 hours. Been down with the flu.

I did the following instructions: To connect using ssh (i.e., putty or some other ssh client), the service needs to be enabled in the web GUI, and you probably also want to enable root logins

When i log on to Freenas with either Putty or Bitvise SSH client I get the same
error message

login as: pbstaley
pbstaley@192.168.0.2's password:
Last login: Sun Sep 20 05:30:58 2015 from 192.168.0.3
Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
The Regents of the University of California. All rights reserved.

FreeBSD 8.3-RELEASE-p7 (FREENAS.amd64) #1 r249203M: Sat Apr 6 09:28:27 PDT 2013

FreeNAS (c) 2009-2013, The FreeNAS Development Team
All rights reserved.
FreeNAS is released under the modified BSD license.

For more information, documentation, help or support, go here:
http://freenas.org
Welcome to FreeNAS
Could not chdir to home directory /nonexistent: No such file or directory
[pbstaley@freenas] /> smartctl -a /dev/ada0
smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

/dev/xpt0 control device couldn't opened: Permission denied
Unable to get CAM device list
/dev/ada0: Unable to detect device type
Smartctl: please specify device type with the -d option.

Use smartctl -h to get a usage summary

[pbstaley@freenas] />


This is the tail end of the output when I run smartctl -a /dev/ada0 on to the GUI

199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always -
0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline -
9

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[root@freenas ~]#

 

[Glorious1]

You have a couple of issues. One is that user pbstaley has no home directory on the FreeNAS. You can fix that at Account > Users > pbstaley, there will be a place to enter or browse to a home directory to set it.

Since you are logged in as pbstaley rather than root, you don't have permission for the smartctl command. Just put "sudo " in front of it (without quotes). It will ask for your password. This elevates you to root permissions for that command. To work "Permit sudo" must be checked in your user settings.

And please remember to put all your output in Insert > Code tags as described before.

 

[Stal00se]

Code:

login as: root
root@192.168.0.2's password:
Last login: Sun Sep 20 07:37:34 2015 from 192.168.0.3
Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
        The Regents of the University of California.  All rights reserved.

FreeBSD 8.3-RELEASE-p7 (FREENAS.amd64) #1 r249203M: Sat Apr  6 09:28:27 PDT 2013

        FreeNAS (c) 2009-2013, The FreeNAS Development Team
        All rights reserved.
        FreeNAS is released under the modified BSD license.

        For more information, documentation, help or support, go here:
        http://freenas.org
Welcome to FreeNAS
[root@freenas] ~# smartctl -a /dev/ada0
smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Green (Adv. Format)
Device Model:     WDC WD6400AARS-00Y5B1
Serial Number:    WD-WCAV56806636
LU WWN Device Id: 5 0014ee 2593a897d
Firmware Version: 80.00A80
User Capacity:    640,135,028,736 bytes [640 GB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Sun Sep 20 07:39:56 2015 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84) Offline data collection activity
                                        was suspended by an interrupting command from host.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (12180) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 142) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x3031) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   224   120   021    Pre-fail  Always       -       1775
  4 Start_Stop_Count        0x0032   073   073   000    Old_age   Always       -       27631
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   060   060   000    Old_age   Always       -       29313
10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   073   073   000    Old_age   Always       -       27271
192 Power-Off_Retract_Count 0x0032   164   164   000    Old_age   Always       -       27155
193 Load_Cycle_Count        0x0032   137   137   000    Old_age   Always       -       189784
194 Temperature_Celsius     0x0022   111   097   000    Old_age   Always       -       36
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       9
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       9
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       9

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[root@freenas] ~# smartctl -a /dev/ada1
smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Green (Adv. Format)
Device Model:     WDC WD6400AARS-00Y5B1
Serial Number:    WD-WCAV56432499
LU WWN Device Id: 5 0014ee 2ae909bda
Firmware Version: 80.00A80
User Capacity:    640,135,028,736 bytes [640 GB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Sun Sep 20 07:40:23 2015 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (12660) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 148) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x3031) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   142   116   021    Pre-fail  Always       -       5866
  4 Start_Stop_Count        0x0032   098   098   000    Old_age   Always       -       2673
  5 Reallocated_Sector_Ct   0x0033   173   173   140    Pre-fail  Always       -       215
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   060   060   000    Old_age   Always       -       29616
10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       584
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       370
193 Load_Cycle_Count        0x0032   180   180   000    Old_age   Always       -       60267
194 Temperature_Celsius     0x0022   116   097   000    Old_age   Always       -       31
196 Reallocated_Event_Count 0x0032   001   001   000    Old_age   Always       -       213
197 Current_Pending_Sector  0x0032   187   187   000    Old_age   Always       -       1449
198 Offline_Uncorrectable   0x0030   200   187   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       6
200 Multi_Zone_Error_Rate   0x0008   200   001   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      6760         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[root@freenas] ~# smartctl -a /dev/ada2
smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Green (Adv. Format)
Device Model:     WDC WD6400AARS-00Y5B1
Serial Number:    WD-WCAV56805278
LU WWN Device Id: 5 0014ee 2ae90742b
Firmware Version: 80.00A80
User Capacity:    640,135,028,736 bytes [640 GB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Sun Sep 20 07:40:40 2015 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (12960) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 151) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x3031) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   193   051    Pre-fail  Always       -       8279
  3 Spin_Up_Time            0x0027   190   120   021    Pre-fail  Always       -       3475
  4 Start_Stop_Count        0x0032   099   099   000    Old_age   Always       -       1409
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   069   069   000    Old_age   Always       -       22665
10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       1177
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       445
193 Load_Cycle_Count        0x0032   162   162   000    Old_age   Always       -       114194
194 Temperature_Celsius     0x0022   115   096   000    Old_age   Always       -       32
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   198   000    Old_age   Always       -       1
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       1
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       585
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       1

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[root@freenas] ~# smartctl -a /dev/ada3
smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Green (Adv. Format)
Device Model:     WDC WD6400AARS-00Y5B1
Serial Number:    WD-WCAV56570171
LU WWN Device Id: 5 0014ee 2593ad75e
Firmware Version: 80.00A80
User Capacity:    640,135,028,736 bytes [640 GB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Sun Sep 20 07:40:55 2015 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (13200) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 154) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x3031) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   127   119   021    Pre-fail  Always       -       6616
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       665
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   056   056   000    Old_age   Always       -       32786
10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       305
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       209
193 Load_Cycle_Count        0x0032   146   146   000    Old_age   Always       -       162707
194 Temperature_Celsius     0x0022   117   096   000    Old_age   Always       -       30
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[root@freenas] ~#

 

[danb35]

Well, the SMART data shows a few problems. First, you've never run long SMART self-tests on any of the disks, and you've only run one short SMART self-test on ada1, and that was 23,000 hours ago. I'd recommend running long SMART tests on all the disks (smartctl -t long /dev/adaN for each disk), and checking the SMART output after they finish (expect them to take several hours). Then set up a regular testing schedule--short tests every 1-7 days, long tests every 1-4 weeks.

Second, ada0 and ada2 are both showing bad sectors. Neither one is showing a lot of them at this time, but it's enough to be concerned about.

Third, the load cycle count on all the disks is way too high. The WDIDLE3.EXE utility should be able to stop that from growing, but that isn't good for the life of your disks.

But with all of that said, it looks like all four disks are back online now--previously you only had three of them. That's a good sign. What's the result of 'zpool status' and 'zpool import' now?

 

[Stal00se]

Code:

login as: root
root@192.168.0.2's password:
Last login: Sun Sep 20 07:39:37 2015 from 192.168.0.3
Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
        The Regents of the University of California.  All rights reserved.

FreeBSD 8.3-RELEASE-p7 (FREENAS.amd64) #1 r249203M: Sat Apr  6 09:28:27 PDT 2013

        FreeNAS (c) 2009-2013, The FreeNAS Development Team
        All rights reserved.
        FreeNAS is released under the modified BSD license.

        For more information, documentation, help or support, go here:
        http://freenas.org
Welcome to FreeNAS
[root@freenas] ~# zpool status
no pools available
[root@freenas] ~# zpool import
   pool: HOBFiles
     id: 14636708750071156672
  state: FAULTED
status: The pool metadata is corrupted.
action: The pool cannot be imported due to damaged devices or data.
        The pool may be active on another system, but can be imported using
        the '-f' flag.
   see: http://www.sun.com/msg/ZFS-8000-72
config:

        HOBFiles                                        FAULTED  corrupted data
          raidz1-0                                      FAULTED  corrupted data
            gptid/aa4f3dd5-b102-11e2-b438-00306771456b  ONLINE
            gptid/ab282c38-b102-11e2-b438-00306771456b  ONLINE
            gptid/abf0d07f-b102-11e2-b438-00306771456b  ONLINE
            gptid/acc4705b-b102-11e2-b438-00306771456b  ONLINE
[root@freenas] ~#