SOLVED RAIDZ2 - hung after zvol delete

joshg678 · Oct 20, 2013

I need some Help.

recently upgraded to freenas 9.1 after having 8.3 for about a year. everything was working fine after the upgrade.

8gm ram
4x 3TB z2 config
zpool named Data

had a zvol (Veeam 1TB) went to delete, clicked delete from the gui and waited. after 4 hours the system was completely hung (would not ping.)

after a reboot the system hangs trying to load the zfs pool and won't boot.

Error message: Solaris: WARNING: can't open objset for Data/Veeam

Any suggestions on where i should begin?

cyberjock · Oct 20, 2013

Post the output of:

camcontrol devlist
gpart list
zpool import
zpool status

joshg678 · Oct 21, 2013

ok,
So i had to remove the drives, wait for it to boot up and fail loading the pool. remove the pool from the gui and then reboot with the drives connected.

Code:

[root@fileserver] ~# camcontrol devlist
<ST31000340AS SD15>                at scbus0 target 0 lun 0 (ada0,pass0)
<ST31000524AS JC4B>                at scbus1 target 0 lun 0 (ada1,pass1)
<ST31000528AS CC38>                at scbus2 target 0 lun 0 (ada2,pass2)
<ST31000521AS CC14>                at scbus3 target 0 lun 0 (ada3,pass3)
<WDC WD30EFRX-68AX9N0 80.00A80>    at scbus4 target 0 lun 0 (ada4,pass4)
<WDC WD30EFRX-68AX9N0 80.00A80>    at scbus5 target 0 lun 0 (ada5,pass5)
<WDC WD30EFRX-68AX9N0 80.00A80>    at scbus6 target 0 lun 0 (ada6,pass6)
<WDC WD30EFRX-68AX9N0 80.00A80>    at scbus7 target 0 lun 0 (ada7,pass7)
<  >                              at scbus10 target 0 lun 0 (da0,pass8)
<  >                              at scbus11 target 0 lun 0 (da1,pass9)
<  >                              at scbus12 target 0 lun 0 (da2,pass10)

Code:

Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 1953525134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 65536
  Mode: r1w1e1
  rawuuid: bc977642-38f8-11e3-a5e5-00095b0719ed
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: ada0p2
  Mediasize: 998057319936 (929G)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 2147549184
  Mode: r1w1e2
  rawuuid: bca9bf0e-38f8-11e3-a5e5-00095b0719ed
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 998057319936
  offset: 2147549184
  type: freebsd-zfs
  index: 2
  end: 1953525134
  start: 4194432
Consumers:
1. Name: ada0
  Mediasize: 1000204886016 (931G)
  Sectorsize: 512
  Mode: r2w2e5
 
Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 1953525134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada1p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 65536
  Mode: r1w1e1
  rawuuid: bd0970b2-38f8-11e3-a5e5-00095b0719ed
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: ada1p2
  Mediasize: 998057319936 (929G)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 2147549184
  Mode: r1w1e2
  rawuuid: bd148dab-38f8-11e3-a5e5-00095b0719ed
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 998057319936
  offset: 2147549184
  type: freebsd-zfs
  index: 2
  end: 1953525134
  start: 4194432
Consumers:
1. Name: ada1
  Mediasize: 1000204886016 (931G)
  Sectorsize: 512
  Mode: r2w2e5
 
Geom name: ada2
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 1953525134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada2p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 65536
  Mode: r1w1e1
  rawuuid: bd6e2cdb-38f8-11e3-a5e5-00095b0719ed
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: ada2p2
  Mediasize: 998057319936 (929G)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 2147549184
  Mode: r1w1e2
  rawuuid: bd79a0c3-38f8-11e3-a5e5-00095b0719ed
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 998057319936
  offset: 2147549184
  type: freebsd-zfs
  index: 2
  end: 1953525134
  start: 4194432
Consumers:
1. Name: ada2
  Mediasize: 1000204886016 (931G)
  Sectorsize: 512
  Mode: r2w2e5
 
Geom name: ada3
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 1953525134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada3p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 65536
  Mode: r1w1e1
  rawuuid: bdcf4d38-38f8-11e3-a5e5-00095b0719ed
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: ada3p2
  Mediasize: 998057319936 (929G)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 2147549184
  Mode: r1w1e2
  rawuuid: bddd8127-38f8-11e3-a5e5-00095b0719ed
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 998057319936
  offset: 2147549184
  type: freebsd-zfs
  index: 2
  end: 1953525134
  start: 4194432
Consumers:
1. Name: ada3
  Mediasize: 1000204886016 (931G)
  Sectorsize: 512
  Mode: r2w2e5
 
Geom name: ada4
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada4p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e1
  rawuuid: 7e7b4978-5760-11e2-bd2e-b8975a2f0abf
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: ada4p2
  Mediasize: 2998445412352 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r0w0e0
  rawuuid: 7e8e4e3a-5760-11e2-bd2e-b8975a2f0abf
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2998445412352
  offset: 2147549184
  type: freebsd-zfs
  index: 2
  end: 5860533127
  start: 4194432
Consumers:
1. Name: ada4
  Mediasize: 3000592982016 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e2
 
Geom name: ada5
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada5p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e1
  rawuuid: 7ed54849-5760-11e2-bd2e-b8975a2f0abf
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: ada5p2
  Mediasize: 2998445412352 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r0w0e0
  rawuuid: 7ee98421-5760-11e2-bd2e-b8975a2f0abf
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2998445412352
  offset: 2147549184
  type: freebsd-zfs
  index: 2
  end: 5860533127
  start: 4194432
Consumers:
1. Name: ada5
  Mediasize: 3000592982016 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e2
 
Geom name: ada6
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada6p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e1
  rawuuid: 7f2fd889-5760-11e2-bd2e-b8975a2f0abf
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: ada6p2
  Mediasize: 2998445412352 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r0w0e0
  rawuuid: 7f433762-5760-11e2-bd2e-b8975a2f0abf
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2998445412352
  offset: 2147549184
  type: freebsd-zfs
  index: 2
  end: 5860533127
  start: 4194432
Consumers:
1. Name: ada6
  Mediasize: 3000592982016 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e2
 
Geom name: ada7
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada7p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e1
  rawuuid: 7f8c1f74-5760-11e2-bd2e-b8975a2f0abf
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: ada7p2
  Mediasize: 2998445412352 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r0w0e0
  rawuuid: 7f9f857e-5760-11e2-bd2e-b8975a2f0abf
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2998445412352
  offset: 2147549184
  type: freebsd-zfs
  index: 2
  end: 5860533127
  start: 4194432
Consumers:
1. Name: ada7
  Mediasize: 3000592982016 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e2
 
Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 15433727
first: 63
entries: 4
scheme: MBR
Providers:
1. Name: da0s1
  Mediasize: 988291584 (942M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 32256
  Mode: r0w0e0
  rawtype: 165
  length: 988291584
  offset: 32256
  type: freebsd
  index: 1
  end: 1930319
  start: 63
2. Name: da0s2
  Mediasize: 988291584 (942M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 988356096
  Mode: r1w0e1
  attrib: active
  rawtype: 165
  length: 988291584
  offset: 988356096
  type: freebsd
  index: 2
  end: 3860639
  start: 1930383
3. Name: da0s3
  Mediasize: 1548288 (1.5M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 1976647680
  Mode: r0w0e0
  rawtype: 165
  length: 1548288
  offset: 1976647680
  type: freebsd
  index: 3
  end: 3863663
  start: 3860640
4. Name: da0s4
  Mediasize: 21159936 (20M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 1978195968
  Mode: r1w1e2
  rawtype: 165
  length: 21159936
  offset: 1978195968
  type: freebsd
  index: 4
  end: 3904991
  start: 3863664
Consumers:
1. Name: da0
  Mediasize: 7902068736 (7.4G)
  Sectorsize: 512
  Mode: r2w1e4
 
Geom name: da1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da1p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 65536
  Mode: r1w1e1
  rawuuid: 9c269263-3901-11e3-a5e5-00095b0719ed
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: da1p2
  Mediasize: 1998251367936 (1.8T)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 2147549184
  Mode: r1w1e2
  rawuuid: 9c3ece71-3901-11e3-a5e5-00095b0719ed
  rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 1998251367936
  offset: 2147549184
  type: freebsd-ufs
  index: 2
  end: 3907029134
  start: 4194432
Consumers:
1. Name: da1
  Mediasize: 2000398934016 (1.8T)
  Sectorsize: 512
  Mode: r2w2e5
 
Geom name: da2
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da2p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 65536
  Mode: r1w1e1
  rawuuid: 383e62c5-3901-11e3-a5e5-00095b0719ed
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: da2p2
  Mediasize: 1998251367936 (1.8T)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 2147549184
  Mode: r1w1e2
  rawuuid: 3851803e-3901-11e3-a5e5-00095b0719ed
  rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 1998251367936
  offset: 2147549184
  type: freebsd-ufs
  index: 2
  end: 3907029134
  start: 4194432
Consumers:
1. Name: da2
  Mediasize: 2000398934016 (1.8T)
  Sectorsize: 512
  Mode: r2w2e5
 
Geom name: da0s1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 1930256
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: da0s1a
  Mediasize: 988283392 (942M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 40448
  Mode: r0w0e0
  rawtype: 0
  length: 988283392
  offset: 8192
  type: !0
  index: 1
  end: 1930256
  start: 16
Consumers:
1. Name: da0s1
  Mediasize: 988291584 (942M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 32256
  Mode: r0w0e0
 
Geom name: da0s2
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 1930256
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: da0s2a
  Mediasize: 988283392 (942M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 988364288
  Mode: r1w0e1
  rawtype: 0
  length: 988283392
  offset: 8192
  type: !0
  index: 1
  end: 1930256
  start: 16
Consumers:
1. Name: da0s2
  Mediasize: 988291584 (942M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 988356096
  Mode: r1w0e1

Code:

[root@fileserver] ~# zpool import
  pool: Data
    id: 1509996785937744451
  state: ONLINE
status: The pool is formatted using a legacy on-disk version.
action: The pool can be imported using its name or numeric identifier, though
        some features will not be available without an explicit 'zpool upgrade'.
config:
 
        Data                                            ONLINE
          raidz2-0                                      ONLINE
            gptid/7e8e4e3a-5760-11e2-bd2e-b8975a2f0abf  ONLINE
            gptid/7ee98421-5760-11e2-bd2e-b8975a2f0abf  ONLINE
            gptid/7f433762-5760-11e2-bd2e-b8975a2f0abf  ONLINE
            gptid/7f9f857e-5760-11e2-bd2e-b8975a2f0abf  ONLINE

Code:

[root@fileserver] ~# zpool status
  pool: iSCSI
state: ONLINE
  scan: none requested
config:
 
        NAME                                            STATE    READ WRITE CKSUM
        iSCSI                                          ONLINE      0    0    0
          raidz2-0                                      ONLINE      0    0    0
            gptid/bca9bf0e-38f8-11e3-a5e5-00095b0719ed  ONLINE      0    0    0
            gptid/bd148dab-38f8-11e3-a5e5-00095b0719ed  ONLINE      0    0    0
            gptid/bd79a0c3-38f8-11e3-a5e5-00095b0719ed  ONLINE      0    0    0
            gptid/bddd8127-38f8-11e3-a5e5-00095b0719ed  ONLINE      0    0    0
 
errors: No known data errors

cyberjock · Oct 21, 2013

Everything looks okay. However, I believe that "data" is a name that is not compatible with FreeNAS. It's one of those "forbidden" names. You might want to try renaming your pool, then mounting it.

You can rename a pool with the command zpool import Data SOMEOTHERNAME. Note that this will make your current FreeNAS configuration not reflect reality because you changed things from the command line. You will need to "export" the old Data pool, then use the auto-import function. I wouldn't do this just yet though as there's some other things to look at below.

Other than that, nothing really looks bad. Have you checked out the SMART data for disks ada0 through ada3? Also, your really short on RAM for the amount of disk space. It's possible that your RAM being as low as it is for the amount of disk space is partially responsible. You really should try to have about 16GB of RAM for that system.

And what's with the extra 3 entries on the camcontrol devlist? Are you virtualizing FreeNAS? Post your hardware specs. That shouldn't really be showing up like that.

joshg678 · Oct 21, 2013

Thanks for taking a look.

This is all a physical system. Two of those devices are my external usb drives, the other is the flash drive freenas is installed to. Not sure why they are not showing correctly.

Biostar Hi-Fi A85W
AMD A10-5700
Memory G.SKILL Ripjaws Series Model F3-8500CL7D-8GBRL 2x 4GB
4x 3TB Western Digital Red
4x 1TB Seagate somethings.

I recently added the iSCSI pool, i could possibly export that and try re adding my data pool to see if that helps any. I'm looking to add 16gb ram, or replace current with 16gb.

The "data" pool is on ada 4-7, here is the smart info

Code:

[root@fileserver] ~# smartctl -q noserial -a /dev/ada4
smartctl 6.1 2013-03-16 r3800 [FreeBSD 9.1-STABLE amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
 
=== START OF INFORMATION SECTION ===
Model Family:    Western Digital Red (AF)
Device Model:    WDC WD30EFRX-68AX9N0
Firmware Version: 80.00A80
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:    512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:  ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Oct 21 10:12:42 2013 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
 
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
 
General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (  0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (41640) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (  2) minutes.
Extended self-test routine
recommended polling time:        ( 418) minutes.
Conveyance self-test routine
recommended polling time:        (  5) minutes.
SCT capabilities:              (0x70bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.
 
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG    VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate    0x002f  200  200  051    Pre-fail  Always      -      0
  3 Spin_Up_Time            0x0027  174  174  021    Pre-fail  Always      -      6258
  4 Start_Stop_Count        0x0032  100  100  000    Old_age  Always      -      39
  5 Reallocated_Sector_Ct  0x0033  200  200  140    Pre-fail  Always      -      0
  7 Seek_Error_Rate        0x002e  100  253  000    Old_age  Always      -      0
  9 Power_On_Hours          0x0032  091  091  000    Old_age  Always      -      6902
10 Spin_Retry_Count        0x0032  100  253  000    Old_age  Always      -      0
11 Calibration_Retry_Count 0x0032  100  253  000    Old_age  Always      -      0
12 Power_Cycle_Count      0x0032  100  100  000    Old_age  Always      -      38
192 Power-Off_Retract_Count 0x0032  200  200  000    Old_age  Always      -      28
193 Load_Cycle_Count        0x0032  200  200  000    Old_age  Always      -      10
194 Temperature_Celsius    0x0022  120  111  000    Old_age  Always      -      30
196 Reallocated_Event_Count 0x0032  200  200  000    Old_age  Always      -      0
197 Current_Pending_Sector  0x0032  200  200  000    Old_age  Always      -      0
198 Offline_Uncorrectable  0x0030  100  253  000    Old_age  Offline      -      0
199 UDMA_CRC_Error_Count    0x0032  200  200  000    Old_age  Always      -      0
200 Multi_Zone_Error_Rate  0x0008  200  200  000    Old_age  Offline      -      0
 
SMART Error Log Version: 1
No Errors Logged
 
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error      00%      6427        -
# 2  Extended offline    Completed without error      00%      5726        -
# 3  Extended offline    Completed without error      00%      4965        -
# 4  Extended offline    Completed without error      00%      4221        -
# 5  Extended offline    Completed without error      00%      3524        -
# 6  Extended offline    Completed without error      00%      2781        -
# 7  Extended offline    Completed without error      00%      2062        -
# 8  Extended offline    Completed without error      00%      1319        -
# 9  Extended offline    Completed without error      00%      648        -
 
SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

[root@fileserver] ~# smartctl -q noserial -a /dev/ada5
smartctl 6.1 2013-03-16 r3800 [FreeBSD 9.1-STABLE amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
 
=== START OF INFORMATION SECTION ===
Model Family:    Western Digital Red (AF)
Device Model:    WDC WD30EFRX-68AX9N0
Firmware Version: 80.00A80
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:    512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:  ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Oct 21 10:13:09 2013 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
 
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
 
General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (  0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (41640) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (  2) minutes.
Extended self-test routine
recommended polling time:        ( 418) minutes.
Conveyance self-test routine
recommended polling time:        (  5) minutes.
SCT capabilities:              (0x70bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.
 
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG    VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate    0x002f  200  200  051    Pre-fail  Always      -      0
  3 Spin_Up_Time            0x0027  170  169  021    Pre-fail  Always      -      6483
  4 Start_Stop_Count        0x0032  100  100  000    Old_age  Always      -      39
  5 Reallocated_Sector_Ct  0x0033  200  200  140    Pre-fail  Always      -      0
  7 Seek_Error_Rate        0x002e  100  253  000    Old_age  Always      -      0
  9 Power_On_Hours          0x0032  091  091  000    Old_age  Always      -      6902
10 Spin_Retry_Count        0x0032  100  253  000    Old_age  Always      -      0
11 Calibration_Retry_Count 0x0032  100  253  000    Old_age  Always      -      0
12 Power_Cycle_Count      0x0032  100  100  000    Old_age  Always      -      38
192 Power-Off_Retract_Count 0x0032  200  200  000    Old_age  Always      -      28
193 Load_Cycle_Count        0x0032  200  200  000    Old_age  Always      -      10
194 Temperature_Celsius    0x0022  122  113  000    Old_age  Always      -      28
196 Reallocated_Event_Count 0x0032  200  200  000    Old_age  Always      -      0
197 Current_Pending_Sector  0x0032  200  200  000    Old_age  Always      -      0
198 Offline_Uncorrectable  0x0030  100  253  000    Old_age  Offline      -      0
199 UDMA_CRC_Error_Count    0x0032  200  200  000    Old_age  Always      -      0
200 Multi_Zone_Error_Rate  0x0008  200  200  000    Old_age  Offline      -      0
 
SMART Error Log Version: 1
No Errors Logged
 
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error      00%      6427        -
# 2  Extended offline    Completed without error      00%      5726        -
# 3  Extended offline    Completed without error      00%      4965        -
# 4  Extended offline    Completed without error      00%      4221        -
# 5  Extended offline    Completed without error      00%      3524        -
# 6  Extended offline    Completed without error      00%      2781        -
# 7  Extended offline    Completed without error      00%      2062        -
# 8  Extended offline    Completed without error      00%      1319        -
# 9  Extended offline    Completed without error      00%      648        -
 
SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

[root@fileserver] ~# smartctl -q noserial -a /dev/ada6
smartctl 6.1 2013-03-16 r3800 [FreeBSD 9.1-STABLE amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
 
=== START OF INFORMATION SECTION ===
Model Family:    Western Digital Red (AF)
Device Model:    WDC WD30EFRX-68AX9N0
Firmware Version: 80.00A80
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:    512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:  ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Oct 21 10:13:29 2013 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
 
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
 
General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (  0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (39540) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (  2) minutes.
Extended self-test routine
recommended polling time:        ( 397) minutes.
Conveyance self-test routine
recommended polling time:        (  5) minutes.
SCT capabilities:              (0x70bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.
 
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG    VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate    0x002f  200  200  051    Pre-fail  Always      -      0
  3 Spin_Up_Time            0x0027  179  178  021    Pre-fail  Always      -      6041
  4 Start_Stop_Count        0x0032  100  100  000    Old_age  Always      -      39
  5 Reallocated_Sector_Ct  0x0033  200  200  140    Pre-fail  Always      -      0
  7 Seek_Error_Rate        0x002e  100  253  000    Old_age  Always      -      0
  9 Power_On_Hours          0x0032  091  091  000    Old_age  Always      -      6902
10 Spin_Retry_Count        0x0032  100  253  000    Old_age  Always      -      0
11 Calibration_Retry_Count 0x0032  100  253  000    Old_age  Always      -      0
12 Power_Cycle_Count      0x0032  100  100  000    Old_age  Always      -      38
192 Power-Off_Retract_Count 0x0032  200  200  000    Old_age  Always      -      28
193 Load_Cycle_Count        0x0032  200  200  000    Old_age  Always      -      10
194 Temperature_Celsius    0x0022  122  114  000    Old_age  Always      -      28
196 Reallocated_Event_Count 0x0032  200  200  000    Old_age  Always      -      0
197 Current_Pending_Sector  0x0032  200  200  000    Old_age  Always      -      0
198 Offline_Uncorrectable  0x0030  100  253  000    Old_age  Offline      -      0
199 UDMA_CRC_Error_Count    0x0032  200  200  000    Old_age  Always      -      0
200 Multi_Zone_Error_Rate  0x0008  200  200  000    Old_age  Offline      -      0
 
SMART Error Log Version: 1
No Errors Logged
 
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error      00%      6427        -
# 2  Extended offline    Completed without error      00%      5725        -
# 3  Extended offline    Completed without error      00%      4964        -
# 4  Extended offline    Completed without error      00%      4221        -
# 5  Extended offline    Completed without error      00%      3524        -
# 6  Extended offline    Completed without error      00%      2780        -
# 7  Extended offline    Completed without error      00%      2061        -
# 8  Extended offline    Completed without error      00%      1319        -
# 9  Extended offline    Completed without error      00%      648        -
 
SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

[root@fileserver] ~# smartctl -q noserial -a /dev/ada7
smartctl 6.1 2013-03-16 r3800 [FreeBSD 9.1-STABLE amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
 
=== START OF INFORMATION SECTION ===
Model Family:    Western Digital Red (AF)
Device Model:    WDC WD30EFRX-68AX9N0
Firmware Version: 80.00A80
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:    512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:  ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Oct 21 10:13:44 2013 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
 
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
 
General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (  0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (39900) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (  2) minutes.
Extended self-test routine
recommended polling time:        ( 400) minutes.
Conveyance self-test routine
recommended polling time:        (  5) minutes.
SCT capabilities:              (0x70bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.
 
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG    VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate    0x002f  200  200  051    Pre-fail  Always      -      0
  3 Spin_Up_Time            0x0027  179  178  021    Pre-fail  Always      -      6033
  4 Start_Stop_Count        0x0032  100  100  000    Old_age  Always      -      38
  5 Reallocated_Sector_Ct  0x0033  200  200  140    Pre-fail  Always      -      0
  7 Seek_Error_Rate        0x002e  100  253  000    Old_age  Always      -      0
  9 Power_On_Hours          0x0032  091  091  000    Old_age  Always      -      6902
10 Spin_Retry_Count        0x0032  100  253  000    Old_age  Always      -      0
11 Calibration_Retry_Count 0x0032  100  253  000    Old_age  Always      -      0
12 Power_Cycle_Count      0x0032  100  100  000    Old_age  Always      -      37
192 Power-Off_Retract_Count 0x0032  200  200  000    Old_age  Always      -      27
193 Load_Cycle_Count        0x0032  200  200  000    Old_age  Always      -      10
194 Temperature_Celsius    0x0022  120  111  000    Old_age  Always      -      30
196 Reallocated_Event_Count 0x0032  200  200  000    Old_age  Always      -      0
197 Current_Pending_Sector  0x0032  200  200  000    Old_age  Always      -      0
198 Offline_Uncorrectable  0x0030  100  253  000    Old_age  Offline      -      0
199 UDMA_CRC_Error_Count    0x0032  200  200  000    Old_age  Always      -      0
200 Multi_Zone_Error_Rate  0x0008  200  200  000    Old_age  Offline      -      0
 
SMART Error Log Version: 1
No Errors Logged
 
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error      00%      6427        -
# 2  Extended offline    Completed without error      00%      5726        -
# 3  Extended offline    Completed without error      00%      4964        -
# 4  Extended offline    Completed without error      00%      4221        -
# 5  Extended offline    Completed without error      00%      3524        -
# 6  Extended offline    Completed without error      00%      2781        -
# 7  Extended offline    Completed without error      00%      2061        -
# 8  Extended offline    Completed without error      00%      1319        -
# 9  Extended offline    Completed without error      00%      648        -
 
SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

The only thing I've attempted from command line thus far was "zpool import Data", and the system hung there, after about 1 hour became unresponsive and was still the same 8 hours later. After a power cycle it was as if i never ran that command.

if i did "zpool import Data Tank", that imports it and then rename to Tank which would require it to successfully import right?

cyberjock · Oct 21, 2013

joshg678 said:
The only thing I've attempted from command line thus far was "zpool import Data", and the system hung there, after about 1 hour became unresponsive and was still the same 8 hours later. After a power cycle it was as if i never ran that command.

Usually, if it won't mount that's the sign of a trashed pool. I say usually as there are exceptions. Before we go farther I'd run a RAM test with www.memtest.org. Just boot it up and let it run. You should give it 3 passes which will probably take a few hours. Your RAM is non-ECC. If its bad it probably trashed your pool. That's at least what experience on this forum will show. Normally the command you ran shouldn't take more than a few seconds. The fact that it takes longer leads me to think that the pool is damaged somehow.

joshg678 said:
if i did "zpool import Data Tank", that imports it and then rename to Tank which would require it to successfully import right?

I'm not 100% sure on the pool renaming and what you have to do. Normally you import and mount it at the same time. FreeNAS does it with the command zpool import (poolname) -m /mnt. In this case we'd be importing it but not mounting it. I'm not sure if you can do an import, rename, and mount at the same time. Unfortunately I can't test this for you right now as I don't have access to my VM. You might have to run the command you just wrote above, then export it(zpool export Tank), then reimport it from the FreeNAS GUI via the auto-import feature. You could also do a reboot after the rename and then try the auto-import feature.

None of these steps should be damaging to the pool, but there's no guarantees either. I'd wait and see if someone else can provide some advice(maybe different advice). Be careful what you do with the pool and the import commands. Many websites recommend commands that are one-way. If they make things worse there is no undoing them.

I'd definitely run the RAM test before you start trying to rename/import/mount it though. If your RAM is determined to be bad I would not bootup FreeNAS anymore until you replace the RAM.

titan_rw · Oct 21, 2013

Just out of curiosity, I assume deduplication has never been used on the zpool?

joshg678 · Oct 21, 2013

i'll give memtest a shot when i get home.

Never used deduplication, only had compression on one of the datasets (the recommended, gzip or something)

joshg678 · Oct 21, 2013

Ok, so i ran mem test and got 3 successful passes with no errors.
I'll export my second pool and see if i can import my original pool.

joshg678 · Oct 21, 2013

So i think my issue is insufficient RAM.

I did "zpool import Data Files" and it ran and ran for about 30 minuets. While it was running i was watching the memory usage, went from 7gb free all the way down to 200mb and started using the swap. Once it got about 164MB in the swap, it dropped the free memory down to 188k and thats when the whole system stopped responding. During this whole process the hard drives are busy with activity, and once the system crashes the activity stops.

If i took a guess, i would say since the system crashed when removing a zvol it must be trying to finish that task when being imported and it fails because i don't have enough ram. Does this sound plausible?

I'm going to get some more ram (and do another mem test) as i'll need it anyway even if have to rebuild from scratch

Any other ideas?

cyberjock · Oct 21, 2013

joshg678 said:
So i think my issue is insufficient RAM.

I did "zpool import Data Files" and it ran and ran for about 30 minuets. While it was running i was watching the memory usage, went from 7gb free all the way down to 200mb and started using the swap. Once it got about 164MB in the swap, it dropped the free memory down to 188k and thats when the whole system stopped responding. During this whole process the hard drives are busy with activity, and once the system crashes the activity stops.

If i took a guess, i would say since the system crashed when removing a zvol it must be trying to finish that task when being imported and it fails because i don't have enough ram. Does this sound plausible?

I'm going to get some more ram (and do another mem test) as i'll need it anyway even if have to rebuild from scratch

Any other ideas?

To be completely honest, I really doubt that's the problem unless you used dedup. But if I'm completely honest I also have to say that I don't have any better ideas. I might not understand what is going on with your RAM thing(200MB of ram free isn't uncommon). So I say go ahead and get more RAM and we'll see where that goes. :)

titan_rw · Oct 22, 2013

The fact it's eating swap during import is weird. I would have also assumed dedupe was used, as 8 GB should be enough to import the pool otherwise.

I wonder if importing it read only from the command line would help? As in fresh boot media, then importing?

cyberjock · Oct 22, 2013

One thing that people don't realize about the read_only setting during mount... it only prevents the user from writing to the pool. If the ZFS daemon finds an error(or otherwise feels it necessary to write to the pool, for example to undo an incomplete transaction or complete a transaction from the ZIL) then it will still write to the disks. I haven't been able to find a way to avoid this, and it may be that its not possible because the pool could be inconsistent and therefor not be mountable.

I'm wondering if something was wrong with his pool before he tried to delete the zvol and now ZFS is stuck between a rock and a hard place somewhere. That's kind of where I think this is going right now.

joshg678 · Oct 25, 2013

New ram came in yesterday.
Ran Memtest overnight, got 3 perfect passes. Now i have 16GB of ram.

I ran zpool import Data, it used 9GB of memory and then finished. (failed doing the mount points.)

Of course i forgot to rename it from data.

Exported it, and then did "zpool import Data Files"
Took like 5 seconds and now shows as being imported and is named Files

So i exported it from command line and did the Auto Import from the GUI.

It imported Perfectly, and that zvol i had attempted to delete is gone.

Holey sh**, what a relief i don't have to recover from backup.

I'm guessing my issue was i didn't have enough memory to delete that 1TB zvol.

Thanks so much for all the help!

cyberjock · Oct 25, 2013

Glad you got it figured out!

titan_rw · Oct 25, 2013

I thought long pool imports were usually due to unclean shutdowns forcing a longer scan through the pool's transactions. I wouldn't have thought that would require all that ram though. I thought large ram usage during import was a dedupe import thing.

Can you try this command on the zpool?

Code:

zdb -D Files

That will tell you if dedupe has ever been enabled. It should say "All DDTs are empty".

joshg678 · Oct 25, 2013

Code:

[root@fileserver] ~# zdb -D Files
All DDTs are empty

titan_rw · Oct 25, 2013

Strange. That's the first I've heard of a non-deduped pool being unmountable due to needing more ram.

joshg678 · Oct 25, 2013

I think it might only be because the system crashed while it was deleting a zvol? And i guess it crashed because it didn't have enough memory?

cyberjock · Oct 25, 2013

It's very possible. It is also possible that the deletion was going at such a slow pace that if you had let it sit long enough it would have eventually finished and mounted. When you are really short on RAM it is possible to have double digit kb/sec transfer rates. I had ultra slow speeds with my system with 12GB of RAM that previously had done 1GB/sec! Try converting that "deleting 1TB of data" to kb/sec and imagine how long it could take. The universe might experience a heat death before your pool finishes. :P

Important Announcement for the TrueNAS Community.

SOLVED RAIDZ2 - hung after zvol delete

Explorer

Inactive Account

Explorer

Inactive Account

Explorer

Inactive Account

Guru

Explorer

Explorer

Explorer

Inactive Account

Guru

Inactive Account

Explorer

Inactive Account

Guru

Explorer

Guru

Explorer

Inactive Account

Similar threads