Can't replace disk because pool is unavailable, even though its imported

[Aaron Lee Kafton]

Hoping someone can lead me in the right direction here... I had a raidz1 pool and had a bit of a misunderstanding and tried to add a drive to the raidz pool and instead it got added as a separate drive from the raidz. Well I was in the process of trying to move the data off when that 4th non-raid drive failed. I rebooted and zpool status didn't return any pools, but zpool import showed my pool with a status of Unavailable with a missing drive. The 3 raid drives online but the 4th drive wasn't even displayed. I ran zpool import -F pool which finished with a

Code:

freenas root: ZFS: vdev failure, zpool=Archive type=vdev.open.failed
freenas root: ZFS: failed to load zpool Archive

But now my pool is showing up in zpool status.
So I tried running every Online/Replace/Detach/Burrito command and they all return 'Pool': pool is unavailable. Is there a switch or something I'm missing here to get the 4th drive replaced?

Also, if someone's got an idea on bypassing the pool altogether and know of some data recovery software I could use to get to the files.

 

[cyberjock]

You are likely screwed my friend. That 4th disk is why your pool is unavailable. Because you added the drive by itself if that disk fails you lose all data. If you review the presentation in my sig I visually demonstrate exactly what you did. There is likely no recovery.

There are no commercial or other tools for recovering data from your pool. If your data is worth enough to you, you could try contacting some data recovery experts(expect to pay no less than about $1500 from personal experience) and see if they can help. ZFS isn't a file system that is commonly used, so you may have trouble even finding a data recovery company that can help.

 

[Aaron Lee Kafton]

Your pdf made my heart hurt. But thanks regardless.

 

[paleoN]

The data is likely gone. That being said what version of FreeNAS are you importing on?

Well I was in the process of trying to move the data off when that 4th non-raid drive failed.

Exactly how failed is this drive? Does is spin up at all?

Did you try importing the pool read only? I think you will likely need that 4th drive or at least one of the ZFS labels on it.

 

[Aaron Lee Kafton]

not really sure, just became unavailable, tried reconnecting it and got nothing. hooked it up to my laptop and the system recognized the drive (not the format of course.) after several attempts to put it back in, i gave up and formatted in ntfs so I could move other files around. Bought an identical drive and put it in, in its place. How do you import the pool as read only?

 

[Aaron Lee Kafton]

tried zpool import -o readonly=on Pool
and it came back as readonly is not a valid pool property

 

[paleoN]

hooked it up to my laptop and the system recognized the drive (not the format of course.) after several attempts to put it back in, i gave up and formatted in ntfs so I could move other files around.

This was rather dumb if you wanted the data that's in the pool. Done is done I suppose.


Are you going to bother to answer my first question? There is a reason I asked it.

That being said what version of FreeNAS are you importing on?

Looks like I will actually need to pick this up tomorrow.

 

[Aaron Lee Kafton]

Sorry I didn't see the first part of your post. I'm on 8.2

And yes, it was dumb in retrospect, and I definitely regret a lot of decisions I've made in this process. I was ill-informed and now I'm paying for it. Unfortunately my thought was that the drive could just be replaced and that it didn't hold any data anyway since I never added any data to the pool after I added the 4th drive and it wasn't a part of the raid. Obviously I thought wrong.

 

[paleoN]

Sorry I didn't see the first part of your post. I'm on 8.2

Which is what I thought. You need to be on 8.3.0-BETA or a recovery disk that has the ZFS V28 code. Go burn 8.3.0-BETA2 to a USB drive and boot with that.

Then try the following commands:

Code:

zpool import -o altroot=/mnt -o failmode=continue -o rdonly=on Archive

zpool status -v

mount

 

[Aaron Lee Kafton]

alright, here's where I'm at. Tried running the first command after installing the beta on a flash drive and I get a Fatal Trap 12 error. I've moved memory around, pulled out all but one stick and even switched it. (currently at 62% of Memtest86 with 0 errors) moved my Rosewill RC-218 from the pci-e 4x port to the 16x port, updated my bios, disabled every port that wasn't being used, and changed my underwear. If I run any command short of actually importing, no fatal trap

Code:

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x38
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff810786d1
stack pointer = 0x28:0xffffff81b277f040
frame pointer = 0x28:0xffffff81b277f070
code segment = base 0x0, limit 0xfffff, type 0x1b
                    = DLP 0, press 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 2877 (initial thread)
trap number = 12
panic: page fault
cpuid = 1

Oh, and just for good measure I moved two of the 3 raidz drives to the onboard 1.5 sata, took the other drive offline and got the same error

 

[paleoN]

If I run any command short of actually importing, no fatal trap

Fatal trap 12 can be a hardware error or a driver/kernel issue. Post the out of dmesg, which will include your RAM, x86 vs x64, show what hardware is detected, etc.

What ZFS tunings, if any, do you have. You may need some.

You could always try out a 9.0-RELEASE iso instead.

 

[Aaron Lee Kafton]

Code:

Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.3-RELEASE-p4 #2 r240088M: Tue Sep  4 06:56:00 PDT 2012
    root@build.ixsystems.com:/build/home/jpaetzel/8.3.0/os-base/amd64/build/home/jpaetzel/8.3.0/FreeBSD/src/sys/FREENAS.amd64 amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 3.00GHz (2992.71-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0xf41  Family = f  Model = 4  Stepping = 1
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x641d<SSE3,DTES64,MON,DS_CPL,CNXT-ID,CX16,xTPR>
  AMD Features=0x20000800<SYSCALL,LM>
  TSC: P-state invariant
real memory  = 6442450944 (6144 MB)
avail memory = 6109884416 (5826 MB)
ACPI APIC Table: <DELL   WS 470 >
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0: Changing APIC ID to 8
ioapic1: Changing APIC ID to 9
ioapic2: Changing APIC ID to 10
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
lapic0: Forcing LINT1 to edge trigger
kbd1 at kbdmux0
hpt27xx: RocketRAID 27xx controller driver v1.0 (Sep  4 2012 06:55:49)
cryptosoft0: <software crypto> on motherboard
aesni0: No AESNI support.
acpi0: <DELL WS 470 > on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <unknown> at device 0.1 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 0.2 on pci1
pci3: <ACPI PCI bus> on pcib3
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.4> port 0xdcc0-0xdcff mem 0xdfee0000-0xdfefffff irq 48 at device 14.0 on pci3
em0: [FILTER]
em0: Ethernet address: 00:12:3f:3a:21:1e
pcib4: <ACPI PCI-PCI bridge> irq 16 at device 3.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 4.0 on pci0
pci5: <ACPI PCI bus> on pcib5
mvs0: <Marvell 88SX7042 SATA controller> port 0xcc00-0xccff mem 0xdfb00000-0xdfbfffff irq 16 at device 0.0 on pci5
mvs0: Gen-IIe, 4 3Gbps ports, Port Multiplier supported with FBS
mvs0: [ITHREAD]
mvsch0: <Marvell SATA channel> at channel 0 on mvs0
mvsch0: [ITHREAD]
mvsch1: <Marvell SATA channel> at channel 1 on mvs0
mvsch1: [ITHREAD]
mvsch2: <Marvell SATA channel> at channel 2 on mvs0
mvsch2: [ITHREAD]
mvsch3: <Marvell SATA channel> at channel 3 on mvs0
mvsch3: [ITHREAD]
uhci0: <Intel 82801EB (ICH5) USB controller USB-A> port 0xff80-0xff9f irq 16 at device 29.0 on pci0
uhci0: [ITHREAD]
usbus0: <Intel 82801EB (ICH5) USB controller USB-A> on uhci0
uhci1: <Intel 82801EB (ICH5) USB controller USB-B> port 0xff60-0xff7f irq 19 at device 29.1 on pci0
uhci1: [ITHREAD]
usbus1: <Intel 82801EB (ICH5) USB controller USB-B> on uhci1
uhci2: <Intel 82801EB (ICH5) USB controller USB-C> port 0xff40-0xff5f irq 18 at device 29.2 on pci0
uhci2: [ITHREAD]
usbus2: <Intel 82801EB (ICH5) USB controller USB-C> on uhci2
uhci3: <Intel 82801EB (ICH5) USB controller USB-D> port 0xff20-0xff3f irq 16 at device 29.3 on pci0
uhci3: [ITHREAD]
usbus3: <Intel 82801EB (ICH5) USB controller USB-D> on uhci3
ehci0: <Intel 82801EB/R (ICH5) USB 2.0 controller> mem 0xffa80800-0xffa80bff irq 23 at device 29.7 on pci0
ehci0: [ITHREAD]
usbus4: EHCI version 1.0
usbus4: <Intel 82801EB/R (ICH5) USB 2.0 controller> on ehci0
pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci6: <ACPI PCI bus> on pcib6
vgapci0: <VGA-compatible display> port 0xbc80-0xbcff mem 0xdf000000-0xdf7fffff,0xdf9f0000-0xdf9fffff irq 17 at device 13.0 on pci6
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH5 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf mem 0xdffffc00-0xdfffffff irq 18 at device 31.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata0: [ITHREAD]
ata1: <ATA channel> at channel 1 on atapci0
ata1: [ITHREAD]
atapci1: <Intel ICH5 SATA150 controller> port 0xfe00-0xfe07,0xfe10-0xfe13,0xfe20-0xfe27,0xfe30-0xfe33,0xfea0-0xfeaf irq 18 at device 31.2 on pci0
atapci1: [ITHREAD]
ata2: <ATA channel> at channel 0 on atapci1
ata2: [ITHREAD]
ata3: <ATA channel> at channel 1 on atapci1
ata3: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
atrtc0: <AT realtime clock> port 0x70-0x7f irq 8 on acpi0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xcefff,0xcf000-0xd07ff,0xd0800-0xd3fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
ppc0: cannot reserve I/O port range
p4tcc0: <CPU Frequency Thermal Control> on cpu0
fuse4bsd: version 0.3.9-pre1, FUSE ABI 7.8
Timecounter "TSC" frequency 2992706850 Hz quality 800
Timecounters tick every 1.000 msec
hpt27xx: no controller detected.
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus4
ata0: timeout waiting for read DRQ
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub4: 8 ports with 8 removable, self powered
ugen4.2: <SanDisk> at usbus4
umass0: <SanDisk Cruzer Glide, class 0/0, rev 2.00/1.26, addr 2> on usbus4
ugen3.2: <Logitech> at usbus3
ukbd0: <USB Keyboard> on usbus3
kbd2 at ukbd0
uhid0: <USB Keyboard> on usbus3
ugen3.3: <Microsoft> at usbus3
ums0: <Microsoft Microsoft Basic Optical Mouse v2.0, class 0/0, rev 1.10/1.04, addr 3> on usbus3
ums0: 3 buttons and [XYZ] coordinates ID=0
ata0: timeout waiting for read DRQ
ata0: timeout waiting for read DRQ
ada0 at mvsch0 bus 0 scbus0 target 0 lun 0
ada0: <WDC WD10EAVS-00D7B0 01.01A01> ATA-8 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada1 at mvsch1 bus 0 scbus1 target 0 lun 0
ada1: <WDC WD10EAVS-00D7B1 01.01A01> ATA-8 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada2 at mvsch2 bus 0 scbus2 target 0 lun 0
ada2: <ST32000540AS CC83> ATA-8 SATA 2.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3 at mvsch3 bus 0 scbus3 target 0 lun 0
ada3: <WDC WD15EADS-00S2B0 01.00A01> ATA-8 SATA 2.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 1430799MB (2930277168 512 byte sectors: 16H 63S/T 16383C)
da0 at umass-sim0 bus 0 scbus8 target 0 lun 0
da0: <SanDisk Cruzer Glide 1.26> Removable Direct Access SCSI-5 device
da0: 40.000MB/s transfers
da0: 15267MB (31266816 512 byte sectors: 255H 63S/T 1946C)
cd0 at ata0 bus 0 scbus4 target 0 lun 0
cd0: <PHILIPS DVD+-RW DVD8631 9D03> Removable CD-ROM SCSI-0 device
cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: cd present [13408 x 2048 byte records]
GEOM: da0s1: geometry does not match label (16h,63s != 255h,63s).
Trying to mount root from ufs:/dev/ufs/FreeNASs1a
WARNING: /data was not properly dismounted
ugen3.3: <Microsoft> at usbus3 (disconnected)
ums0: at uhub3, port 2, addr 3 (disconnected)
ugen3.3: <Microsoft> at usbus3
ums0: <Microsoft Microsoft Basic Optical Mouse v2.0, class 0/0, rev 1.10/1.04, addr 3> on usbus3
ums0: 3 buttons and [XYZ] coordinates ID=0
ZFS filesystem version 5
ZFS storage pool version 28

don't have any tunings, I had autotuning enabled on the 8.2 install

 

[paleoN]

don't have any tunings, I had autotuning enabled on the 8.2 install

Go ahead and turn autotuning on in 8.3 as well. This will limit ZFS ARC size which may be contributing to the panic.

If that doesn't help, do try with the 9.0-RELEASE iso from above. It should be quick enough to test.

 

[Aaron Lee Kafton]

No apparent joy on autotuning, same panic. The 9.0 version doesn't seem to find the drives connected to my pci-e card. however it will find them if I put drives on the only two sata ports onboard which of course makes the raid degraded, went ahead and gave it a go with the two drives.. it sat for awhile and then flashed a whole bunch of panics on my screen

 

[paleoN]

No apparent joy on autotuning, same panic.

It can't be easy can it.

The 9.0 version doesn't seem to find the drives connected to my pci-e card.

You can try a live USB/DVD. That one's based on 9.1 and likely has more drivers. Where is the Rosewill RC-218, the pci-e 4x or 16x port?

it sat for awhile and then flashed a whole bunch of panics on my screen

I take it they weren't relevant or they went by too fast? Not that I would know what to do with them.

Do you have other memory you can stick in the machine for now?




Edit: Wrong link, fixed.

 

[Aaron Lee Kafton]

Ok, I was fighting with this for several days before I even posted this, and despite your awesome help, several distros, and days of troubleshooting hardware. I'm getting nothing but panics. I threw 3 random small disks in to see to create a pool then destroy the pool and see if I'd get the same error trying to import... and no. Import runs without a problem, even with the -F switch that would give me panics every other time I'd try to run it..

I think I have to come to terms and just pay my tuition to the school of hardknocks and move on. Thank you so much for your help and now I have a lot more tools to deal with any future problems.

 

[paleoN]

I threw 3 random small disks in to see to create a pool then destroy the pool and see if I'd get the same error trying to import... and no. Import runs without a problem, even with the -F switch that would give me panics every other time I'd try to run it..

Unfortunately, this doesn't tell us as much as you would like. It's a much smaller pool and empty, or nearly so.

I think I have to come to terms and just pay my tuition to the school of hardknocks and move on. Thank you so much for your help and now I have a lot more tools to deal with any future problems.

There's is a fair chance it's what you would have to do anyway. Oh, next time don't be shy.

A few final thoughts:

• Have you tried a Solaris based distro, e.g. OpenIndiana? I would say you want something that has ZFS V28 or newer in it.
• Given that this is ZFS and you used an HBA, you can simply stick the disks in a different box and attempt recovery from there.
• With all the panics you are seeing I would be taking a hard look at hardware. FreeBSD 8.3 and 9 have updated drivers in them. It's also possible that they have some bugs in them for your hardware. Personally, I'm leaning toward your hardware and not the drivers.

You can always just leave the disks until the weekend. If you are still done with all this then, go ahead and wipe the disks. After you have been fighting this X number of hours, it gets to the point that the data is just gone and it's easier to move on.

 

[Aaron Lee Kafton]

I ran that command one last time... and I got a different result. I ran it through the web shell instead of directly into the console on the machine, I don't know why that would matter.

This is what it's showing on the console screen, while the web shell screen hasn't provided the next line so I can only assume its hung or still processing.

Code:

mvsch0: Timeout on slot 0
mvsch0: iec 00000000 sstat 00000123 serr 00000000 edma_s 00001160 dma_c 20000708 dma_s 00000008 rs 00000001 status c0
(ada0:mvsch0:0:0:0): Synchronize cache failed
mvsch0: Timeout on slot 0
mvsch0: iec 00000000 sstat 00000123 serr 00000000 edma_s 000000e0 dma_c 00000000 dma_s 00000008 rs 00000001 status 40
(ada0:mvsch0:0:0:0): lost device

 

[paleoN]

Code:

mvsch0: Timeout on slot 0
mvsch0: iec 00000000 sstat 00000123 serr 00000000 edma_s 00001160 dma_c 20000708 dma_s 00000008 rs 00000001 status c0
(ada0:mvsch0:0:0:0): Synchronize cache failed
mvsch0: Timeout on slot 0
mvsch0: iec 00000000 sstat 00000123 serr 00000000 edma_s 000000e0 dma_c 00000000 dma_s 00000008 rs 00000001 status 40
(ada0:mvsch0:0:0:0): lost device

I believe this means ada0 fell off the bus. Which would be a problem with one of the drives (ada0 the most likely), a controller problem, a driver problem or other hardware issues.

What's the output of:

Code:

smartctl -A /dev/ada0
...
smartctl -A /dev/ada2

 

[Aaron Lee Kafton]

Code:

[root@freenas] ~# smartctl -A /dev/ada0
smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   111   099   006    Pre-fail  Always       -       34883432
  3 Spin_Up_Time            0x0003   100   100   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       239
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   055   051   030    Pre-fail  Always       -       124566298009
  9 Power_On_Hours          0x0032   098   098   000    Old_age   Always       -       1864
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       154
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   096   096   000    Old_age   Always       -       4
190 Airflow_Temperature_Cel 0x0022   061   043   045    Old_age   Always   In_the_past 39 (0 8 39 38 0)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       96
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       210
194 Temperature_Celsius     0x0022   039   057   000    Old_age   Always       -       39 (0 22 0 0 0)
195 Hardware_ECC_Recovered  0x001a   035   027   000    Old_age   Always       -       34883432
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       135115376166462
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       3186702969
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       3243232457

[root@freenas] ~# smartctl -A /dev/ada1
smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   226   158   021    Pre-fail  Always       -       3658
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       570
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   097   097   000    Old_age   Always       -       2473
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       165
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       30
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       568
194 Temperature_Celsius     0x0022   115   101   000    Old_age   Always       -       35
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       1
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   117   000    Old_age   Always       -       8762
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

[root@freenas] ~# smartctl -A /dev/ada2
smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       1326
  3 Spin_Up_Time            0x0027   253   140   021    Pre-fail  Always       -       3816
  4 Start_Stop_Count        0x0032   098   098   000    Old_age   Always       -       2695
  5 Reallocated_Sector_Ct   0x0033   199   199   140    Pre-fail  Always       -       1
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   086   086   000    Old_age   Always       -       10657
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       1791
192 Power-Off_Retract_Count 0x0032   198   198   000    Old_age   Always       -       1587
193 Load_Cycle_Count        0x0032   181   181   000    Old_age   Always       -       58568
194 Temperature_Celsius     0x0022   115   095   000    Old_age   Always       -       37
196 Reallocated_Event_Count 0x0032   199   199   000    Old_age   Always       -       1
197 Current_Pending_Sector  0x0032   200   197   000    Old_age   Always       -       8
198 Offline_Uncorrectable   0x0030   200   199   000    Old_age   Offline      -       2
199 UDMA_CRC_Error_Count    0x0032   200   198   000    Old_age   Always       -       475
200 Multi_Zone_Error_Rate   0x0008   200   193   000    Old_age   Offline      -       1

[root@freenas] ~# smartctl -A /dev/ada3
smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p4 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   232   154   021    Pre-fail  Always       -       3400
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       272
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   051    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   090   090   000    Old_age   Always       -       7577
 10 Spin_Retry_Count        0x0032   100   100   051    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   051    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       135
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       16
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       271
194 Temperature_Celsius     0x0022   116   083   000    Old_age   Always       -       34
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       4
200 Multi_Zone_Error_Rate   0x0008   200   200   051    Old_age   Offline      -       0