truenas not starting due to "uncorrectable I/O failure and has been suspended"

Grinas · Aug 4, 2023

So i had this thread open as i had a number of failing drives.

Help with identifying which drives are bad.

Sorry for the stupid question but im confused as to which drives are actually degraded. So long story short i just realized today that i have no gotten an email from truenas in about a year so decides to have a look at the storage to make sure everything is ok but I see 2 drives degraded. I am...

www.truenas.com

After replacing one drive and 12 hours of re silvering. truenas froze and i was unable to run any command via ssh or WebUI. I tired to restart it but after an hour or so it still had not rebooted and console said waiting on PID.

I did a hard reboot of the VM and now truenas wont start. When starting i get the error followed by errors you see below.

Code:

Solaris: WARNING: Pool 'ThreeTBM' has encountered an uncorrectable I/O failure and has been suspended

From looking at other threads with that error truenas seems to boot for them and they can see pool info. I have done a few drive changes such as replace new drive the old drive it replaced but the drives may be in different bays as I took out all the drives and labels them when i was changing the drive.

I left this boot for 12 hours and it was still not responding to ping, webUI access and console access.

I know the pool is probably gone but any one have any suggestions on how to get Truenas to boot so i can at least see the disks and see what has failed?

Arwen · Aug 4, 2023

First, the pool may not be "gone". ZFS is telling you their is a problem. This is likely due to your in-complete disk replacement. ZFS will normally resume the disk replacement on reboot. But, it could be you lost blocks or another whole disk in your RAID-Z1. That would either be file loss or pool loss, depending on what we see.

Please supply the output of the following command, in CODE tags;
zpool import

ZFS was specifically designed to survive a graceless, (aka a crash or hard power off), reboot. And do so without data loss, (except any data in flight, like any other file system). That said, hardware problems have had an impact on graceless reboots, causing data or pool loss.

One last note. When replacing a failing but not failed disk, if you have a spare disk slot, do a replace in place. This allows ZFS to keep the existing failing disk as a source of data, in addition to the other disks. With a RAID-Z1 pool this can be a life / pool saver, in case a bad block is found on another disk, but the failing disk has that block stripe still functional.

Grinas · Aug 4, 2023

Arwen said:
First, the pool may not be "gone". ZFS is telling you their is a problem. This is likely due to your in-complete disk replacement. ZFS will normally resume the disk replacement on reboot. But, it could be you lost blocks or another whole disk in your RAID-Z1. That would either be file loss or pool loss, depending on what we see.

Please supply the output of the following command, in CODE tags;
zpool import

ZFS was specifically designed to survive a graceless, (aka a crash or hard power off), reboot. And do so without data loss, (except any data in flight, like any other file system). That said, hardware problems have had an impact on graceless reboots, causing data or pool loss.

One last note. When replacing a failing but not failed disk, if you have a spare disk slot, do a replace in place. This allows ZFS to keep the existing failing disk as a source of data, in addition to the other disks. With a RAID-Z1 pool this can be a life / pool saver, in case a bad block is found on another disk, but the failing disk has that block stripe still functional.

Sorry I should of been more clear in my first post truenas is not booting so i cant run those commands.

Will these commands work if i boot into another mode?

Arwen · Aug 4, 2023

Hmm, if the server is not booting, then something is wrong with your boot-pool. This is complicated because you are running a virtual machine.

Did you pass the LSI HBA through to your TrueNAS Core VM?

Get 2 new boot devices, one loaded with TrueNAS Core, and another available for installing TrueNAS Core. Install TrueNAS Core to the available one. Then boot off the new installed device. You should then be able to see your various pool's status. Unless you restore a saved configuration, the pools won't auto-import.

Grinas · Aug 8, 2023

So created and I installed a truenas 13 on a new vm.

It booted and i imported the affected pool. all the backups of this truenas were either on the pool or on gdrive with i had to get rid of delete a few week ago since they changed their licensing.

After importing the pool i still cant access the pool or disk info either via console or via Webui. It just shows loading. I have left it 30+ mins but still the same.

Should i leave it longer or does anyone else have any other suggestions?

sretalla · Aug 8, 2023

How are the pool disks arriving in the VM? is the HBA in PCIe passthrough?

Grinas · Aug 8, 2023

sretalla said:
How are the pool disks arriving in the VM? is the HBA in PCIe passthrough?

yes.

sretalla · Aug 8, 2023

OK; that's relatively good news.

so can you see the disks?

camcontrol devlist

And is the pool really imported yet?

zpool import

Grinas · Aug 8, 2023

sretalla said:
OK; that's relatively good news.

so can you see the disks?

camcontrol devlist

Code:

root@truenas[~]# camcontrol devlist
<NECVMWar VMware SATA CD00 1.00>   at scbus2 target 0 lun 0 (cd0,pass0)
<VMware Virtual disk 2.0>          at scbus32 target 0 lun 0 (pass1,da0)
<ATA WDC WD30EFRX-68E 0A82>        at scbus33 target 1 lun 0 (pass2,da1)
<ATA TOSHIBA DT01ACA3 ABB0>        at scbus33 target 3 lun 0 (pass3,da2)
<ATA WDC WD30EFRX-68E 0A82>        at scbus33 target 5 lun 0 (pass4,da3)
<ATA WDC WD30PURZ-85G 0A80>        at scbus33 target 7 lun 0 (pass5,da4)
<ATA NT-1TB 117D>                  at scbus33 target 8 lun 0 (pass6,da5)
<ATA WDC WD30PURX-64P 2B.3>        at scbus33 target 9 lun 0 (pass7,da6)
<ATA WDC WD42PURZ-85B 0A80>        at scbus33 target 12 lun 0 (pass8,da7)
<ATA WDC WD40EFZX-68A 0B81>        at scbus33 target 13 lun 0 (pass9,da8)

And is the pool really imported yet?

zpool import

I ran this command and got nothing back yet after waiting 5+ minutes.

How long would the pool normally take to import? Is there somewhere i can see the logs to see progress of import?

sretalla · Aug 8, 2023

Grinas said:
How long would the pool normally take to import? Is there somewhere i can see the logs to see progress of import?

Minutes at most is "normal"... sometimes there can be issues.

dmesg will give you some clues perhaps

CAM messages are of particular interest if there are any.

Grinas · Aug 8, 2023

sretalla said:
Minutes at most is "normal"... sometimes there can be issues.

dmesg will give you some clues perhaps

CAM messages are of particular interest if there are any.

all i got from dmesg is below. No time stamp either so dont know when that message is from.

Code:

hwpc_core: unknown PMC architecture: 1
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI>
Security policy loaded: MAC/ntpd (mac_ntpd)
Solaris: WARNING: Pool 'ThreeTBM' has encountered an uncorrectable I/O failure and has been suspended.

got a few errors about da2 having readable sectors in ESXI console too.

sretalla · Aug 8, 2023

Grinas said:
got a few errors about da2 having readable sectors in ESXI console too.

That's strange... if that's a disk in TrueNAS, and attached to the PCIe HBA which is in passthrough to the VM, ESXi can't see it.

Grinas said:
all i got from dmesg is below. No time stamp either so dont know when that message is from.

That seems unreasonably short... are we saying the boot drive (virtual disk) is corrupt again? (is that what's on da2?)

sretalla · Aug 8, 2023

Grinas said:
Pool 'ThreeTBM' has encountered an uncorrectable I/O failure and has been suspended.

That does seem to not be your boot pool though, so it's not great news for the pool import.

Grinas · Aug 8, 2023

That's strange... if that's a disk in TrueNAS, and attached to the PCIe HBA which is in passthrough to the VM, ESXi can't see it.

That seems unreasonably short... are we saying the boot drive (virtual disk) is corrupt again? (is that what's on da2?)

When i say ESXI console i mean console of VM in ESXI so it is being seen by truenas VM.

No thats just the last 10 lines, the rest seems standard enough,

Here is the full `dmesg` output.

Code:

Copyright (c) 1992-2021 The FreeBSD Project.

Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994

    The Regents of the University of California. All rights reserved.

FreeBSD is a registered trademark of The FreeBSD Foundation.

FreeBSD 13.1-RELEASE-p7 n245428-4dfb91682c1 TRUENAS amd64

FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)

VT(vga): text 80x25

CPU: Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz (3192.61-MHz K8-class CPU)

  Origin="GenuineIntel"  Id=0x306c3  Family=0x6  Model=0x3c  Stepping=3

  Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>

  Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>

  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>

  AMD Features2=0x21<LAHF,ABM>

  Structured Extended Features=0x27ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG>

  Structured Extended Features3=0xbc000400<MD_CLEAR,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>

  XSAVE Features=0x1<XSAVEOPT>

  IA32_ARCH_CAPS=0xc<RSBA,SKIP_L1DFL_VME>

  TSC: P-state invariant

Hypervisor: Origin = "VMwareVMware"

real memory  = 21474836480 (20480 MB)

avail memory = 20802605056 (19838 MB)

Event timer "LAPIC" quality 600

ACPI APIC Table: <PTLTD       APIC  >

FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs

FreeBSD/SMP: 4 package(s) x 1 core(s)

random: registering fast source Intel Secure Key RNG

random: fast provider: "Intel Secure Key RNG"

arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled.

MADT: Forcing active-low polarity and level trigger for SCI

ioapic0 <Version 2.0> irqs 0-23

Launching APs: 2 3 1

random: entropy device external interface

kbd1 at kbdmux0

vtvga0: <VT VGA driver>

aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>

padlock0: No ACE support.

acpi0: <INTEL 440BX>

acpi0: Power Button (fixed)

hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0

Timecounter "HPET" frequency 14318180 Hz quality 950

cpu0: <ACPI CPU> numa-domain 0 on acpi0

attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0

Timecounter "i8254" frequency 1193182 Hz quality 0

Event timer "i8254" frequency 1193182 Hz quality 100

atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0

atrtc0: registered as a time-of-day clock, resolution 1.000000s

Event timer "RTC" frequency 32768 Hz quality 0

Timecounter "ACPI-fast" frequency 3579545 Hz quality 900

acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0

pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0

pci0: <ACPI PCI bus> on pcib0

pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0

pci1: <ACPI PCI bus> on pcib1

isab0: <PCI-ISA bridge> at device 7.0 on pci0

isa0: <ISA bus> on isab0

atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1060-0x106f at device 7.1 on pci0

ata0: <ATA channel> at channel 0 on atapci0

ata1: <ATA channel> at channel 1 on atapci0

pci0: <bridge> at device 7.3 (no driver attached)

vgapci0: <VGA-compatible display> port 0x1070-0x107f mem 0xe8000000-0xefffffff,0xfe000000-0xfe7fffff irq 16 at device 15.0 on pci0

vgapci0: Boot video device

pcib2: <ACPI PCI-PCI bridge> at device 17.0 on pci0

pci2: <ACPI PCI bus> on pcib2

uhci0: <UHCI (generic) USB controller> port 0x2000-0x201f irq 18 at device 0.0 on pci2

usbus0 on uhci0

usbus0: 12Mbps Full Speed USB v1.0

ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfd5ff000-0xfd5fffff irq 19 at device 1.0 on pci2

usbus1: EHCI version 1.0

usbus1 on ehci0

usbus1: 480Mbps High Speed USB v2.0

ahci0: <AHCI SATA controller> mem 0xfd5fe000-0xfd5fefff irq 17 at device 3.0 on pci2

ahci0: AHCI v1.30 with 30 6Gbps ports, Port Multiplier not supported

ahcich0: <AHCI channel> at channel 0 on ahci0

ahcich1: <AHCI channel> at channel 1 on ahci0

ahcich2: <AHCI channel> at channel 2 on ahci0

ahcich3: <AHCI channel> at channel 3 on ahci0

ahcich4: <AHCI channel> at channel 4 on ahci0

ahcich5: <AHCI channel> at channel 5 on ahci0

ahcich6: <AHCI channel> at channel 6 on ahci0

ahcich7: <AHCI channel> at channel 7 on ahci0

ahcich8: <AHCI channel> at channel 8 on ahci0

ahcich9: <AHCI channel> at channel 9 on ahci0

ahcich10: <AHCI channel> at channel 10 on ahci0

ahcich11: <AHCI channel> at channel 11 on ahci0

ahcich12: <AHCI channel> at channel 12 on ahci0

ahcich13: <AHCI channel> at channel 13 on ahci0

ahcich14: <AHCI channel> at channel 14 on ahci0

ahcich15: <AHCI channel> at channel 15 on ahci0

ahcich16: <AHCI channel> at channel 16 on ahci0

ahcich17: <AHCI channel> at channel 17 on ahci0

ahcich18: <AHCI channel> at channel 18 on ahci0

ahcich19: <AHCI channel> at channel 19 on ahci0

ahcich20: <AHCI channel> at channel 20 on ahci0

ahcich21: <AHCI channel> at channel 21 on ahci0

ahcich22: <AHCI channel> at channel 22 on ahci0

ahcich23: <AHCI channel> at channel 23 on ahci0

ahcich24: <AHCI channel> at channel 24 on ahci0

ahcich25: <AHCI channel> at channel 25 on ahci0

ahcich26: <AHCI channel> at channel 26 on ahci0

ahcich27: <AHCI channel> at channel 27 on ahci0

ahcich28: <AHCI channel> at channel 28 on ahci0

ahcich29: <AHCI channel> at channel 29 on ahci0

pcib3: <ACPI PCI-PCI bridge> at device 21.0 on pci0

pci3: <ACPI PCI bus> on pcib3

mpt0: <LSILogic SAS/SATA Adapter> port 0x4000-0x40ff mem 0xfd4ec000-0xfd4effff,0xfd4f0000-0xfd4fffff irq 18 at device 0.0 on pci3

mpt0: MPI Version=1.5.0.0

pcib4: <ACPI PCI-PCI bridge> at device 21.1 on pci0

pcib5: <ACPI PCI-PCI bridge> at device 21.2 on pci0

pcib6: <ACPI PCI-PCI bridge> at device 21.3 on pci0

pcib7: <ACPI PCI-PCI bridge> at device 21.4 on pci0

pcib8: <ACPI PCI-PCI bridge> at device 21.5 on pci0

pcib9: <ACPI PCI-PCI bridge> at device 21.6 on pci0

pcib10: <ACPI PCI-PCI bridge> at device 21.7 on pci0

pcib11: <ACPI PCI-PCI bridge> at device 22.0 on pci0

pci4: <ACPI PCI bus> on pcib11

mps0: <Avago Technologies (LSI) SAS2308> port 0x5000-0x50ff mem 0xfd3f0000-0xfd3fffff,0xfd380000-0xfd3bffff irq 19 at device 0.0 on pci4

mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd

mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>

pcib12: <ACPI PCI-PCI bridge> at device 22.1 on pci0

pcib13: <ACPI PCI-PCI bridge> at device 22.2 on pci0

pcib14: <ACPI PCI-PCI bridge> at device 22.3 on pci0

pcib15: <ACPI PCI-PCI bridge> at device 22.4 on pci0

pcib16: <ACPI PCI-PCI bridge> at device 22.5 on pci0

pcib17: <ACPI PCI-PCI bridge> at device 22.6 on pci0

pcib18: <ACPI PCI-PCI bridge> at device 22.7 on pci0

pcib19: <ACPI PCI-PCI bridge> at device 23.0 on pci0

pci5: <ACPI PCI bus> on pcib19

vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x6000-0x600f mem 0xfd2fc000-0xfd2fcfff,0xfd2fd000-0xfd2fdfff,0xfd2fe000-0xfd2fffff irq 16 at device 0.0 on pci5

vmx0: Using 512 TX descriptors and 512 RX descriptors

vmx0: Using 4 RX queues 4 TX queues

vmx0: Using MSI-X interrupts with 5 vectors

vmx0: Ethernet address: 00:0c:29:e9:75:05

pcib20: <ACPI PCI-PCI bridge> at device 23.1 on pci0

pcib21: <ACPI PCI-PCI bridge> at device 23.2 on pci0

pcib22: <ACPI PCI-PCI bridge> at device 23.3 on pci0

pcib23: <ACPI PCI-PCI bridge> at device 23.4 on pci0

pcib24: <ACPI PCI-PCI bridge> at device 23.5 on pci0

pcib25: <ACPI PCI-PCI bridge> at device 23.6 on pci0

pcib26: <ACPI PCI-PCI bridge> at device 23.7 on pci0

pcib27: <ACPI PCI-PCI bridge> at device 24.0 on pci0

pcib28: <ACPI PCI-PCI bridge> at device 24.1 on pci0

pcib29: <ACPI PCI-PCI bridge> at device 24.2 on pci0

pcib30: <ACPI PCI-PCI bridge> at device 24.3 on pci0

pcib31: <ACPI PCI-PCI bridge> at device 24.4 on pci0

pcib32: <ACPI PCI-PCI bridge> at device 24.5 on pci0

pcib33: <ACPI PCI-PCI bridge> at device 24.6 on pci0

pcib34: <ACPI PCI-PCI bridge> at device 24.7 on pci0

acpi_acad0: <AC Adapter> on acpi0

atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0

atkbd0: <AT Keyboard> irq 1 on atkbdc0

kbd0 at atkbd0

atkbd0: [GIANT-LOCKED]

psm0: <PS/2 Mouse> irq 12 on atkbdc0

psm0: [GIANT-LOCKED]

WARNING: Device "psm" is Giant locked and may be deleted before FreeBSD 14.0.

psm0: model IntelliMouse, device ID 3

acpi_syscontainer0: <System Container> on acpi0

orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc9fff,0xca000-0xcbfff,0xcc000-0xccfff,0xdc000-0xdffff,0xe0000-0xe7fff pnpid ORM0000 on isa0

Timecounter "TSC-low" frequency 1596303500 Hz quality 1000

Timecounters tick every 10.000 msec

ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled

ugen1.1: <(0x15ad) EHCI root HUB> at usbus1

ugen0.1: <(0x15ad) UHCI root HUB> at usbus0

uhub0 on usbus1

uhub0: <(0x15ad) EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1

uhub1 on usbus0

uhub1: <(0x15ad) UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0

random: unblocking device.

Trying to mount root from cd9660:iso9660/TRUENAS []...

uhub1: 2 ports with 2 removable, self powered

Root mount waiting for: CAM usbus0 usbus1

ugen0.2: <VMware VMware Virtual USB Mouse> at usbus0

uhid0 on uhub1

uhid0: <VMware> on usbus0

ums0 on uhub1

ums0: <VMware> on usbus0

ums0: 16 buttons and [XYZT] coordinates ID=0

Root mount waiting for: CAM usbus0 usbus1

ugen0.3: <vendor 0x0e0f VMware Virtual USB Hub> at usbus0

uhub2 on uhub1

uhub2: <VMware Virtual USB Hub> on usbus0

uhub0: 6 ports with 6 removable, self powered

da0 at mpt0 bus 0 scbus32 target 0 lun 0

da0: <VMware Virtual disk 2.0> Fixed Direct Access SPC-4 SCSI device

da0: 300.000MB/s transfers

da0: Command Queueing enabled

da0: 8192MB (16777216 512 byte sectors)

da0: quirks=0x140<RETRY_BUSY,STRICT_UNMAP>

cd0 at ahcich0 bus 0 scbus2 target 0 lun 0

cd0: <NECVMWar VMware SATA CD00 1.00> Removable CD-ROM SCSI device

cd0: Serial Number 00000000000000000001

cd0: 600.000MB/s transfers (SATA 3.x, UDMA2, ATAPI 12bytes, PIO 8192bytes)

cd0: 998MB (511399 2048 byte sectors)

da5 at mps0 bus 0 scbus33 target 8 lun 0

da5: <ATA NT-1TB 117D> Fixed Direct Access SPC-4 SCSI device

da5: Serial Number 9100113100090

da5: 600.000MB/s transfers

da5: Command Queueing enabled

da5: 976762MB (2000409264 512 byte sectors)

da8 at mps0 bus 0 scbus33 target 13 lun 0

da8: <ATA WDC WD40EFZX-68A 0B81> Fixed Direct Access SPC-4 SCSI device

da8: Serial Number WD-WX12D514R32V

da8: 600.000MB/s transfers

da8: Command Queueing enabled

da8: 3815447MB (7814037168 512 byte sectors)

da3 at mps0 bus 0 scbus33 target 5 lun 0

da3: <ATA WDC WD30EFRX-68E 0A82> Fixed Direct Access SPC-4 SCSI device

da3: Serial Number WD-WCC4N6SS9F16

da3: 600.000MB/s transfers

da3: Command Queueing enabled

da3: 2861588MB (5860533168 512 byte sectors)

da3: quirks=0x8<4K>

da1 at mps0 bus 0 scbus33 target 1 lun 0

da1: <ATA WDC WD30EFRX-68E 0A82> Fixed Direct Access SPC-4 SCSI device

da1: Serial Number WD-WCC4N0XP648R

da1: 600.000MB/s transfers

da1: Command Queueing enabled

da1: 2861588MB (5860533168 512 byte sectors)

da1: quirks=0x8<4K>

da6 at mps0 bus 0 scbus33 target 9 lun 0

da6: <ATA WDC WD30PURX-64P 2B.3> Fixed Direct Access SPC-4 SCSI device

da6: Serial Number WD-WCC4N6VVXV6T

da6: 300.000MB/s transfers

da6: Command Queueing enabled

da6: 2862443MB (5862285072 512 byte sectors)

da6: quirks=0x8<4K>

da7 at mps0 bus 0 scbus33 target 12 lun 0

da7: <ATA WDC WD42PURZ-85B 0A80> Fixed Direct Access SPC-4 SCSI device

da7: Serial Number WD-WX32D917Z45D

da7: 600.000MB/s transfers

da7: Command Queueing enabled

da7: 3815447MB (7814037168 512 byte sectors)

da4 at mps0 bus 0 scbus33 target 7 lun 0

da4: <ATA WDC WD30PURZ-85G 0A80> Fixed Direct Access SPC-4 SCSI device

da4: Serial Number WD-WCC4N7AY14Z9

da4: 600.000MB/s transfers

da4: Command Queueing enabled

da4: 2861588MB (5860533168 512 byte sectors)

da2 at mps0 bus 0 scbus33 target 3 lun 0

da2: <ATA TOSHIBA DT01ACA3 ABB0> Fixed Direct Access SPC-4 SCSI device

da2: Serial Number X3V9SV5GS

da2: 600.000MB/s transfers

da2: Command Queueing enabled

da2: 2861588MB (5860533168 512 byte sectors)

uhub2: 7 ports with 7 removable, self powered

md0 attached to /data/base.ufs.uzip

Trying to mount root from ufs:/dev/md0.uzip [rw]...

ZFS filesystem version: 5

ZFS storage pool version: features support (5000)

Waiting (max 60 seconds) for system process `vnlru' to stop... done

Waiting (max 60 seconds) for system process `syncer' to stop...

Syncing disks, vnodes remaining... 0 0 done

All buffers synced.

Uptime: 23m30s

mps0: Sending StopUnit: path (xpt0:mps0:0:8:ffffffff):  handle 14

uhub0: detached

mps0: Incrementing SSU count

uhid0: detached

ums0: detached

mps0: Decrementing SSU count.

mps0: Completing stop unit for (xpt0:mps0:0:8:ffffffff):

---<<BOOT>>---

Copyright (c) 1992-2021 The FreeBSD Project.

Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994

    The Regents of the University of California. All rights reserved.

FreeBSD is a registered trademark of The FreeBSD Foundation.

FreeBSD 13.1-RELEASE-p7 n245428-4dfb91682c1 TRUENAS amd64

FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)

VT(vga): text 80x25

CPU: Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz (3192.61-MHz K8-class CPU)

  Origin="GenuineIntel"  Id=0x306c3  Family=0x6  Model=0x3c  Stepping=3

  Features=0xf83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS>

  Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>

  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>

  AMD Features2=0x21<LAHF,ABM>

  Structured Extended Features=0x27ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG>

  Structured Extended Features3=0xbc000400<MD_CLEAR,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>

  XSAVE Features=0x1<XSAVEOPT>

  IA32_ARCH_CAPS=0xc<RSBA,SKIP_L1DFL_VME>

  TSC: P-state invariant

Hypervisor: Origin = "VMwareVMware"

real memory  = 21474836480 (20480 MB)

avail memory = 20789882880 (19826 MB)

Event timer "LAPIC" quality 600

ACPI APIC Table: <PTLTD       APIC  >

FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs

FreeBSD/SMP: 4 package(s) x 1 core(s)

random: registering fast source Intel Secure Key RNG

random: fast provider: "Intel Secure Key RNG"

arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled.

MADT: Forcing active-low polarity and level trigger for SCI

ioapic0 <Version 2.0> irqs 0-23

Launching APs: 2 1 3

random: entropy device external interface

kbd1 at kbdmux0

vtvga0: <VT VGA driver>

aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>

padlock0: No ACE support.

acpi0: <INTEL 440BX>

acpi0: Power Button (fixed)

hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0

Timecounter "HPET" frequency 14318180 Hz quality 950

cpu0: <ACPI CPU> numa-domain 0 on acpi0

attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0

Timecounter "i8254" frequency 1193182 Hz quality 0

Event timer "i8254" frequency 1193182 Hz quality 100

atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0

atrtc0: registered as a time-of-day clock, resolution 1.000000s

Event timer "RTC" frequency 32768 Hz quality 0

Timecounter "ACPI-fast" frequency 3579545 Hz quality 900

acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0

pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0

pci0: <ACPI PCI bus> on pcib0

pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0

pci1: <ACPI PCI bus> on pcib1

isab0: <PCI-ISA bridge> at device 7.0 on pci0

isa0: <ISA bus> on isab0

atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1060-0x106f at device 7.1 on pci0

ata0: <ATA channel> at channel 0 on atapci0

ata1: <ATA channel> at channel 1 on atapci0

pci0: <bridge> at device 7.3 (no driver attached)

vgapci0: <VGA-compatible display> port 0x1070-0x107f mem 0xe8000000-0xefffffff,0xfe000000-0xfe7fffff irq 16 at device 15.0 on pci0

vgapci0: Boot video device

pcib2: <ACPI PCI-PCI bridge> at device 17.0 on pci0

pci2: <ACPI PCI bus> on pcib2

uhci0: <UHCI (generic) USB controller> port 0x2000-0x201f irq 18 at device 0.0 on pci2

usbus0 on uhci0

usbus0: 12Mbps Full Speed USB v1.0

ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfd5ff000-0xfd5fffff irq 19 at device 1.0 on pci2

usbus1: EHCI version 1.0

usbus1 on ehci0

usbus1: 480Mbps High Speed USB v2.0

ahci0: <AHCI SATA controller> mem 0xfd5fe000-0xfd5fefff irq 17 at device 3.0 on pci2

ahci0: AHCI v1.30 with 30 6Gbps ports, Port Multiplier not supported

ahcich0: <AHCI channel> at channel 0 on ahci0

ahcich1: <AHCI channel> at channel 1 on ahci0

ahcich2: <AHCI channel> at channel 2 on ahci0

ahcich3: <AHCI channel> at channel 3 on ahci0

ahcich4: <AHCI channel> at channel 4 on ahci0

ahcich5: <AHCI channel> at channel 5 on ahci0

ahcich6: <AHCI channel> at channel 6 on ahci0

ahcich7: <AHCI channel> at channel 7 on ahci0

ahcich8: <AHCI channel> at channel 8 on ahci0

ahcich9: <AHCI channel> at channel 9 on ahci0

ahcich10: <AHCI channel> at channel 10 on ahci0

ahcich11: <AHCI channel> at channel 11 on ahci0

ahcich12: <AHCI channel> at channel 12 on ahci0

ahcich13: <AHCI channel> at channel 13 on ahci0

ahcich14: <AHCI channel> at channel 14 on ahci0

ahcich15: <AHCI channel> at channel 15 on ahci0

ahcich16: <AHCI channel> at channel 16 on ahci0

ahcich17: <AHCI channel> at channel 17 on ahci0

ahcich18: <AHCI channel> at channel 18 on ahci0

ahcich19: <AHCI channel> at channel 19 on ahci0

ahcich20: <AHCI channel> at channel 20 on ahci0

ahcich21: <AHCI channel> at channel 21 on ahci0

ahcich22: <AHCI channel> at channel 22 on ahci0

ahcich23: <AHCI channel> at channel 23 on ahci0

ahcich24: <AHCI channel> at channel 24 on ahci0

ahcich25: <AHCI channel> at channel 25 on ahci0

ahcich26: <AHCI channel> at channel 26 on ahci0

ahcich27: <AHCI channel> at channel 27 on ahci0

ahcich28: <AHCI channel> at channel 28 on ahci0

ahcich29: <AHCI channel> at channel 29 on ahci0

pcib3: <ACPI PCI-PCI bridge> at device 21.0 on pci0

pci3: <ACPI PCI bus> on pcib3

mpt0: <LSILogic SAS/SATA Adapter> port 0x4000-0x40ff mem 0xfd4ec000-0xfd4effff,0xfd4f0000-0xfd4fffff irq 18 at device 0.0 on pci3

mpt0: MPI Version=1.5.0.0

pcib4: <ACPI PCI-PCI bridge> at device 21.1 on pci0

pcib5: <ACPI PCI-PCI bridge> at device 21.2 on pci0

pcib6: <ACPI PCI-PCI bridge> at device 21.3 on pci0

pcib7: <ACPI PCI-PCI bridge> at device 21.4 on pci0

pcib8: <ACPI PCI-PCI bridge> at device 21.5 on pci0

pcib9: <ACPI PCI-PCI bridge> at device 21.6 on pci0

pcib10: <ACPI PCI-PCI bridge> at device 21.7 on pci0

pcib11: <ACPI PCI-PCI bridge> at device 22.0 on pci0

pci4: <ACPI PCI bus> on pcib11

mps0: <Avago Technologies (LSI) SAS2308> port 0x5000-0x50ff mem 0xfd3f0000-0xfd3fffff,0xfd380000-0xfd3bffff irq 19 at device 0.0 on pci4

mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd

mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>

pcib12: <ACPI PCI-PCI bridge> at device 22.1 on pci0

pcib13: <ACPI PCI-PCI bridge> at device 22.2 on pci0

pcib14: <ACPI PCI-PCI bridge> at device 22.3 on pci0

pcib15: <ACPI PCI-PCI bridge> at device 22.4 on pci0

pcib16: <ACPI PCI-PCI bridge> at device 22.5 on pci0

pcib17: <ACPI PCI-PCI bridge> at device 22.6 on pci0

pcib18: <ACPI PCI-PCI bridge> at device 22.7 on pci0

pcib19: <ACPI PCI-PCI bridge> at device 23.0 on pci0

pci5: <ACPI PCI bus> on pcib19

vmx0: <VMware VMXNET3 Ethernet Adapter> port 0x6000-0x600f mem 0xfd2fc000-0xfd2fcfff,0xfd2fd000-0xfd2fdfff,0xfd2fe000-0xfd2fffff irq 16 at device 0.0 on pci5

vmx0: Using 512 TX descriptors and 512 RX descriptors

vmx0: Using 4 RX queues 4 TX queues

vmx0: Using MSI-X interrupts with 5 vectors

vmx0: Ethernet address: 00:0c:29:e9:75:05

pcib20: <ACPI PCI-PCI bridge> at device 23.1 on pci0

pcib21: <ACPI PCI-PCI bridge> at device 23.2 on pci0

pcib22: <ACPI PCI-PCI bridge> at device 23.3 on pci0

pcib23: <ACPI PCI-PCI bridge> at device 23.4 on pci0

pcib24: <ACPI PCI-PCI bridge> at device 23.5 on pci0

pcib25: <ACPI PCI-PCI bridge> at device 23.6 on pci0

pcib26: <ACPI PCI-PCI bridge> at device 23.7 on pci0

pcib27: <ACPI PCI-PCI bridge> at device 24.0 on pci0

pcib28: <ACPI PCI-PCI bridge> at device 24.1 on pci0

pcib29: <ACPI PCI-PCI bridge> at device 24.2 on pci0

pcib30: <ACPI PCI-PCI bridge> at device 24.3 on pci0

pcib31: <ACPI PCI-PCI bridge> at device 24.4 on pci0

pcib32: <ACPI PCI-PCI bridge> at device 24.5 on pci0

pcib33: <ACPI PCI-PCI bridge> at device 24.6 on pci0

pcib34: <ACPI PCI-PCI bridge> at device 24.7 on pci0

acpi_acad0: <AC Adapter> on acpi0

atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0

atkbd0: <AT Keyboard> irq 1 on atkbdc0

kbd0 at atkbd0

atkbd0: [GIANT-LOCKED]

psm0: <PS/2 Mouse> irq 12 on atkbdc0

psm0: [GIANT-LOCKED]

WARNING: Device "psm" is Giant locked and may be deleted before FreeBSD 14.0.

psm0: model IntelliMouse, device ID 3

acpi_syscontainer0: <System Container> on acpi0

orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc9fff,0xca000-0xcbfff,0xcc000-0xccfff,0xdc000-0xdffff,0xe0000-0xe7fff pnpid ORM0000 on isa0

Timecounter "TSC-low" frequency 1596303500 Hz quality 1000

Timecounters tick every 10.000 msec

ZFS filesystem version: 5

ZFS storage pool version: features support (5000)

ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled

ugen0.1: <(0x15ad) UHCI root HUB> at usbus0

ugen1.1: <(0x15ad) EHCI root HUB> at usbus1

uhub0 on usbus0

uhub0: <(0x15ad) UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0

uhub1 on usbus1

uhub1: <(0x15ad) EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1

random: unblocking device.

Trying to mount root from zfs:boot-pool/ROOT/default []...

uhub0: 2 ports with 2 removable, self powered

Root mount waiting for: CAM usbus0 usbus1

ugen0.2: <VMware VMware Virtual USB Mouse> at usbus0

uhid0 on uhub0

uhid0: <VMware> on usbus0

ums0 on uhub0

ums0: <VMware> on usbus0

ums0: 16 buttons and [XYZT] coordinates ID=0

ugen0.3: <vendor 0x0e0f VMware Virtual USB Hub> at usbus0

uhub2 on uhub0

uhub2: <VMware Virtual USB Hub> on usbus0

Root mount waiting for: CAM usbus0 usbus1

uhub1: 6 ports with 6 removable, self powered

da0 at mpt0 bus 0 scbus32 target 0 lun 0

da0: <VMware Virtual disk 2.0> Fixed Direct Access SPC-4 SCSI device

da0: 300.000MB/s transfers

da0: Command Queueing enabled

da0: 8192MB (16777216 512 byte sectors)

da0: quirks=0x140<RETRY_BUSY,STRICT_UNMAP>

da5 at mps0 bus 0 scbus33 target 8 lun 0

da5: <ATA NT-1TB 117D> Fixed Direct Access SPC-4 SCSI device

da5: Serial Number 9100113100090

da5: 600.000MB/s transfers

da5: Command Queueing enabled

da5: 976762MB (2000409264 512 byte sectors)

cd0 at ahcich0 bus 0 scbus2 target 0 lun 0

cd0: <NECVMWar VMware SATA CD00 1.00> Removable CD-ROM SCSI device

cd0: Serial Number 00000000000000000001

cd0: 600.000MB/s transfers (SATA 3.x, UDMA2, ATAPI 12bytes, PIO 8192bytes)

cd0: 998MB (511399 2048 byte sectors)

da3 at mps0 bus 0 scbus33 target 5 lun 0

da3: <ATA WDC WD30EFRX-68E 0A82> Fixed Direct Access SPC-4 SCSI device

da3: Serial Number WD-WCC4N6SS9F16

da3: 600.000MB/s transfers

da3: Command Queueing enabled

da3: 2861588MB (5860533168 512 byte sectors)

da3: quirks=0x8<4K>

da1 at mps0 bus 0 scbus33 target 1 lun 0

da1: <ATA WDC WD30EFRX-68E 0A82> Fixed Direct Access SPC-4 SCSI device

da1: Serial Number WD-WCC4N0XP648R

da1: 600.000MB/s transfers

da1: Command Queueing enabled

da1: 2861588MB (5860533168 512 byte sectors)

da1: quirks=0x8<4K>

da8 at mps0 bus 0 scbus33 target 13 lun 0

da8: <ATA WDC WD40EFZX-68A 0B81> Fixed Direct Access SPC-4 SCSI device

da8: Serial Number WD-WX12D514R32V

da8: 600.000MB/s transfers

da8: Command Queueing enabled

da8: 3815447MB (7814037168 512 byte sectors)

da6 at mps0 bus 0 scbus33 target 9 lun 0

da6: <ATA WDC WD30PURX-64P 2B.3> Fixed Direct Access SPC-4 SCSI device

da6: Serial Number WD-WCC4N6VVXV6T

da6: 300.000MB/s transfers

da6: Command Queueing enabled

da6: 2862443MB (5862285072 512 byte sectors)

da6: quirks=0x8<4K>

da7 at mps0 bus 0 scbus33 target 12 lun 0

da7: <ATA WDC WD42PURZ-85B 0A80> Fixed Direct Access SPC-4 SCSI device

da7: Serial Number WD-WX32D917Z45D

da7: 600.000MB/s transfers

da7: Command Queueing enabled

da7: 3815447MB (7814037168 512 byte sectors)

da4 at mps0 bus 0 scbus33 target 7 lun 0

da4: <ATA WDC WD30PURZ-85G 0A80> Fixed Direct Access SPC-4 SCSI device

da4: Serial Number WD-WCC4N7AY14Z9

da4: 600.000MB/s transfers

da4: Command Queueing enabled

da4: 2861588MB (5860533168 512 byte sectors)

uhub2: 7 ports with 7 removable, self powered

da2 at mps0 bus 0 scbus33 target 3 lun 0

da2: <ATA TOSHIBA DT01ACA3 ABB0> Fixed Direct Access SPC-4 SCSI device

da2: Serial Number X3V9SV5GS

da2: 600.000MB/s transfers

da2: Command Queueing enabled

da2: 2861588MB (5860533168 512 byte sectors)

intsmb0: <Intel PIIX4 SMBUS Interface> port 0x1040-0x104f at device 7.3 on pci0

intsmb0: intr SMI disabled revision 0

smbus0: <System Management Bus> on intsmb0

lo0: link state changed to UP

vmx_modinit: processor does not support VMX operation

module_register_init: MOD_LOAD (vmm, 0xffffffff83604200, 0) error 6

CPU: Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz (3192.61-MHz K8-class CPU)

  Origin="GenuineIntel"  Id=0x306c3  Family=0x6  Model=0x3c  Stepping=3

  Features=0xf8bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,SS>

  Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>

  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>

  AMD Features2=0x21<LAHF,ABM>

  Structured Extended Features=0x27ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG>

  Structured Extended Features3=0xbc000400<MD_CLEAR,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>

  XSAVE Features=0x1<XSAVEOPT>

  IA32_ARCH_CAPS=0xc<RSBA,SKIP_L1DFL_VME>

  TSC: P-state invariant

Hypervisor: Origin = "VMwareVMware"

VMware memory control driver initialized

vmx0: link state changed to UP

hwpc_core: unknown PMC architecture: 1

hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI>

Security policy loaded: MAC/ntpd (mac_ntpd)

Solaris: WARNING: Pool 'ThreeTBM' has encountered an uncorrectable I/O failure and has been suspended.

sretalla said:
That does seem to not be your boot pool though, so it's not great news for the pool import.

I would of thought it would of least been able to tell me the problematic drives so i can try to switch them with the old degraded ones. Now i dont know what drives are good and whats not to create a new pool.

These are the errors i see on truenas vm console

sretalla · Aug 8, 2023

You should still be able to run zpool status and see something in that state (I would expect).

How about smartctl -a /dev/da3 (etc.)?

Grinas · Aug 8, 2023

sretalla said:
You should still be able to run zpool status and see something in that state (I would expect).

How about smartctl -a /dev/da3 (etc.)?

10 minutes since i ran zpool status and zero output. Doesnt appear to have locked up as when i press enter it goes to new line.

smartctl -a /dev/da3 this does seem to work.

da1

Code:

smartctl -a /dev/da1
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD30EFRX-68EUZN0
Serial Number:    WD-WCC4N0XP648R
LU WWN Device Id: 5 0014ee 2640bcb11
Firmware Version: 82.00A82
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Aug  8 06:18:19 2023 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (40080) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      ( 402) minutes.
Conveyance self-test routine
recommended polling time:      (   5) minutes.
SCT capabilities:            (0x703d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       3
  3 Spin_Up_Time            0x0027   182   180   021    Pre-fail  Always       -       5866
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       233
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   039   039   000    Old_age   Always       -       44592
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       233
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       232
193 Load_Cycle_Count        0x0032   199   199   000    Old_age   Always       -       5198
194 Temperature_Celsius     0x0022   113   097   000    Old_age   Always       -       37
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       1

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     35590         -
# 2  Extended offline    Completed without error       00%     32877         -
# 3  Extended offline    Completed without error       00%     32853         -
# 4  Extended offline    Completed without error       00%     32783         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

da2

Code:

smartctl -a /dev/da2
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Toshiba 3.5" DT01ACA... Desktop HDD
Device Model:     TOSHIBA DT01ACA300
Serial Number:    X3V9SV5GS
LU WWN Device Id: 5 000039 ff4d2885a
Firmware Version: MX6OABB0
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Aug  8 06:19:39 2023 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84)    Offline data collection activity
                    was suspended by an interrupting command from host.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (22652) seconds.
Offline data collection
capabilities:              (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   1) minutes.
Extended self-test routine
recommended polling time:      ( 378) minutes.
SCT capabilities:            (0x003d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   139   139   054    Pre-fail  Offline      -       70
  3 Spin_Up_Time            0x0007   147   147   024    Pre-fail  Always       -       415 (Average 365)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       591
  5 Reallocated_Sector_Ct   0x0033   094   094   005    Pre-fail  Always       -       276
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   124   124   020    Pre-fail  Offline      -       33
  9 Power_On_Hours          0x0012   092   092   000    Old_age   Always       -       61005
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       403
192 Power-Off_Retract_Count 0x0032   099   099   000    Old_age   Always       -       1341
193 Load_Cycle_Count        0x0012   099   099   000    Old_age   Always       -       1341
194 Temperature_Celsius     0x0002   171   171   000    Old_age   Always       -       35 (Min/Max 13/57)
196 Reallocated_Event_Count 0x0032   093   093   000    Old_age   Always       -       307
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       120
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       1102

SMART Error Log Version: 1
ATA Error Count: 1153 (device log contains only the most recent five errors)
    CR = Command Register [HEX]
    FR = Features Register [HEX]
    SC = Sector Count Register [HEX]
    SN = Sector Number Register [HEX]
    CL = Cylinder Low Register [HEX]
    CH = Cylinder High Register [HEX]
    DH = Device/Head Register [HEX]
    DC = Device Command Register [HEX]
    ER = Error register [HEX]
    ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 1153 occurred at disk power-on lifetime: 60867 hours (2536 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 00 00 4f c2 00  Error: ABRT

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  b0 d5 01 01 4f c2 00 00      05:46:54.563  SMART READ LOG
  b0 d5 01 00 4f c2 00 00      05:46:54.563  SMART READ LOG
  b0 da 00 00 4f c2 00 00      05:46:54.396  SMART RETURN STATUS
  b0 d1 01 01 4f c2 00 00      05:46:54.231  SMART READ ATTRIBUTE THRESHOLDS [OBS-4]
  b0 d0 01 00 4f c2 00 00      05:46:53.963  SMART READ DATA

Error 1152 occurred at disk power-on lifetime: 60867 hours (2536 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 c1 3f 02 00 00  Error: ICRC, ABRT at LBA = 0x0000023f = 575

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 e0 10 20 a0 50 40 00      05:46:11.191  READ FPDMA QUEUED
  60 e0 08 20 9e 50 40 00      05:46:11.191  READ FPDMA QUEUED
  60 e0 00 20 02 00 40 00      05:46:11.191  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      05:46:11.190  READ LOG EXT
  60 e0 18 20 a0 50 40 00      05:46:11.183  READ FPDMA QUEUED

Error 1151 occurred at disk power-on lifetime: 60867 hours (2536 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 c1 3f 00 00 00  Error: ICRC, ABRT at LBA = 0x0000003f = 63

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 e0 18 20 a0 50 40 00      05:46:11.183  READ FPDMA QUEUED
  60 e0 10 20 9e 50 40 00      05:46:11.183  READ FPDMA QUEUED
  60 e0 08 20 02 00 40 00      05:46:11.183  READ FPDMA QUEUED
  60 e0 00 20 00 00 40 00      05:46:11.183  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      05:46:11.182  READ LOG EXT

Error 1150 occurred at disk power-on lifetime: 60867 hours (2536 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 c1 3f 00 00 00  Error: ICRC, ABRT at LBA = 0x0000003f = 63

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 e0 18 20 a0 50 40 00      05:46:11.153  READ FPDMA QUEUED
  60 e0 10 20 9e 50 40 00      05:46:11.153  READ FPDMA QUEUED
  60 e0 08 20 02 00 40 00      05:46:11.153  READ FPDMA QUEUED
  60 e0 00 20 00 00 40 00      05:46:11.153  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      05:46:11.152  READ LOG EXT

Error 1149 occurred at disk power-on lifetime: 60867 hours (2536 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 d1 2f 9e 50 0d  Error: ICRC, ABRT at LBA = 0x0d509e2f = 223387183

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 e0 18 20 00 00 40 00      05:46:11.125  READ FPDMA QUEUED
  60 e0 10 20 02 00 40 00      05:46:11.124  READ FPDMA QUEUED
  60 e0 08 20 a0 50 40 00      05:46:11.124  READ FPDMA QUEUED
  60 e0 00 20 9e 50 40 00      05:46:11.124  READ FPDMA QUEUED
  ef 10 02 00 00 00 00 00      05:46:11.073  SET FEATURES [Enable SATA feature]

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     60410         -
# 2  Short offline       Completed without error       00%     60398         -
# 3  Short offline       Completed without error       00%     60242         -
# 4  Short offline       Completed without error       00%     60230         -
# 5  Short offline       Completed without error       00%     60026         -
# 6  Short offline       Completed without error       00%     60014         -
# 7  Short offline       Completed without error       00%     59858         -
# 8  Short offline       Completed without error       00%     59846         -
# 9  Short offline       Completed without error       00%     59690         -
#10  Short offline       Completed without error       00%     59678         -
#11  Short offline       Completed without error       00%     59525         -
#12  Short offline       Completed without error       00%     59513         -
#13  Short offline       Completed without error       00%     59285         -
#14  Short offline       Completed without error       00%     59273         -
#15  Short offline       Completed without error       00%     59117         -
#16  Short offline       Completed without error       00%     59105         -
#17  Short offline       Completed without error       00%     58949         -
#18  Short offline       Completed without error       00%     58937         -
#19  Short offline       Completed without error       00%     58781         -
#20  Short offline       Completed without error       00%     58769         -
#21  Short offline       Completed without error       00%     58565         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

da3

Code:

smartctl -a /dev/da3
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD30EFRX-68EUZN0
Serial Number:    WD-WCC4N6SS9F16
LU WWN Device Id: 5 0014ee 20e541427
Firmware Version: 82.00A82
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Aug  8 06:20:06 2023 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (40020) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      ( 401) minutes.
Conveyance self-test routine
recommended polling time:      (   5) minutes.
SCT capabilities:            (0x703d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       1
  3 Spin_Up_Time            0x0027   181   178   021    Pre-fail  Always       -       5925
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       263
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   028   028   000    Old_age   Always       -       53028
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       263
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       248
193 Load_Cycle_Count        0x0032   193   193   000    Old_age   Always       -       22575
194 Temperature_Celsius     0x0022   111   100   000    Old_age   Always       -       39
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     52767         -
# 2  Short offline       Completed without error       00%     52755         -
# 3  Short offline       Completed without error       00%     52599         -
# 4  Short offline       Completed without error       00%     52587         -
# 5  Short offline       Completed without error       00%     52431         -
# 6  Short offline       Completed without error       00%     52419         -
# 7  Short offline       Completed without error       00%     52264         -
# 8  Short offline       Completed without error       00%     52252         -
# 9  Short offline       Completed without error       00%     52048         -
#10  Short offline       Completed without error       00%     52036         -
#11  Short offline       Completed without error       00%     51880         -
#12  Short offline       Completed without error       00%     51868         -
#13  Short offline       Completed without error       00%     51712         -
#14  Short offline       Completed without error       00%     51700         -
#15  Short offline       Completed without error       00%     51548         -
#16  Short offline       Completed without error       00%     51536         -
#17  Short offline       Completed without error       00%     51308         -
#18  Short offline       Completed without error       00%     51296         -
#19  Short offline       Completed without error       00%     51140         -
#20  Short offline       Completed without error       00%     51128         -
#21  Short offline       Completed without error       00%     50972         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

da4

Code:

smartctl -a /dev/da4
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Purple
Device Model:     WDC WD30PURZ-85GU6Y0
Serial Number:    WD-WCC4N7AY14Z9
LU WWN Device Id: 5 0014ee 265d1adc5
Firmware Version: 80.00A80
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Aug  8 06:20:23 2023 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (39600) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      ( 398) minutes.
Conveyance self-test routine
recommended polling time:      (   5) minutes.
SCT capabilities:            (0x703d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   184   182   021    Pre-fail  Always       -       5766
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       126
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   060   060   000    Old_age   Always       -       29213
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       126
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       124
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       648
194 Temperature_Celsius     0x0022   116   092   000    Old_age   Always       -       34
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     20214         -
# 2  Extended offline    Completed without error       00%     17496         -
# 3  Extended offline    Completed without error       00%     17472         -
# 4  Extended offline    Completed without error       00%     17402         -
# 5  Extended offline    Aborted by host               90%     17364         -
# 6  Short offline       Completed without error       00%     17179         -
# 7  Extended offline    Completed without error       00%     17137         -
# 8  Conveyance offline  Completed without error       00%     17129         -
# 9  Extended offline    Aborted by host               90%     17129         -
#10  Short offline       Completed without error       00%     17129         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

da6

Code:

smartctl -a /dev/da6
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Purple
Device Model:     WDC WD30PURX-64P6ZY0
Serial Number:    WD-WCC4N6VVXV6T
LU WWN Device Id: 0 000000 000000000
Firmware Version: 01.12B.3
User Capacity:    3,001,489,956,864 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
Local Time is:    Tue Aug  8 06:20:55 2023 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (42300) seconds.
Offline data collection
capabilities:              (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      ( 424) minutes.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0xfc77   200   200   051    Pre-fail  Always       -       60
  3 Spin_Up_Time            0xe455   221   218   021    Pre-fail  Offline      -       3933
  4 Start_Stop_Count        0xe054   100   100   000    Old_age   Offline      -       108
  5 Reallocated_Sector_Ct   0xf455   200   200   140    Pre-fail  Offline      -       0
  7 Seek_Error_Rate         0xe455   200   200   051    Pre-fail  Offline      -       0
  9 Power_On_Hours          0xec74   063   063   000    Old_age   Offline      -       27110
 10 Spin_Retry_Count        0xd675   100   100   051    Pre-fail  Offline      -       0
 11 Calibration_Retry_Count 0xac14   100   100   051    Old_age   Offline      -       0
 12 Power_Cycle_Count       0xfc34   100   100   000    Old_age   Offline      -       108
184 End-to-End_Error        0xf477   100   100   097    Pre-fail  Always       -       0
187 Reported_Uncorrect      0xb414   100   100   000    Old_age   Offline      -       0
188 Command_Timeout         0xb436   100   099   000    Old_age   Always       -       1
190 Airflow_Temperature_Cel 0xac76   065   042   000    Old_age   Always       -       35
192 Power-Off_Retract_Count 0xa474   200   200   000    Old_age   Offline      -       107
193 Load_Cycle_Count        0xa476   200   200   000    Old_age   Always       -       0
194 Temperature_Celsius     0xc45c   115   092   000    Old_age   Offline      -       35
195 Hardware_ECC_Recovered  0xdc7e   200   200   000    Old_age   Always       -       0
196 Reallocated_Event_Count 0xbc74   200   200   000    Old_age   Offline      -       0
197 Current_Pending_Sector  0xe674   200   200   000    Old_age   Offline      -       0
198 Offline_Uncorrectable   0xfc7e   100   253   000    Old_age   Always       -       0
199 UDMA_CRC_Error_Count    0xc456   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0xbc57   200   200   051    Pre-fail  Always       -       28

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed: read failure       30%     26986         4191536
# 2  Extended offline    Completed: read failure       10%     18117         4198216
# 3  Extended offline    Completed: read failure       90%     15402         4198216
# 4  Extended offline    Completed: read failure       90%     15380         4198216
# 5  Extended offline    Completed: read failure       90%     15380         4198216
# 6  Extended offline    Completed: read failure       90%     15378         4198216
# 7  Extended offline    Completed: read failure       90%     15308         4198216

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Grinas · Aug 8, 2023

needed second post as i hit char limit. I left out Da5 as that is a SSD used for jails

da7

Code:

smartctl -a /dev/da7
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD42PURZ-85B4YY0
Serial Number:    WD-WX32D917Z45D
LU WWN Device Id: 5 0014ee 269f574a5
Firmware Version: 80.00A80
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5040 rpm
Form Factor:      3.5 inches
TRIM Command:     Available
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Aug  8 06:21:08 2023 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         ( 1800) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      ( 438) minutes.
Conveyance self-test routine
recommended polling time:      (   5) minutes.
SCT capabilities:            (0x3039)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   206   206   021    Pre-fail  Always       -       2700
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       20
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   088   088   000    Old_age   Always       -       9029
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       19
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       17
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2
194 Temperature_Celsius     0x0022   115   095   000    Old_age   Always       -       32
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%        46         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

da8

Code:

smartctl -a /dev/da8
smartctl 7.2 2021-09-14 r5236 [FreeBSD 13.1-RELEASE-p7 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD40EFZX-68AWUN0
Serial Number:    WD-WX12D514R32V
LU WWN Device Id: 5 0014ee 214503a33
Firmware Version: 81.00B81
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Aug  8 06:21:41 2023 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (41520) seconds.
Offline data collection
capabilities:              (0x11) SMART execute Offline immediate.
                    No Auto Offline data collection support.
                    Suspend Offline collection upon new
                    command.
                    No Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    No Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      ( 442) minutes.
SCT capabilities:            (0x303d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   218   218   021    Pre-fail  Always       -       4058
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       7
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       131
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       6
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       4
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       27
194 Temperature_Celsius     0x0022   118   107   000    Old_age   Always       -       32
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

Selective Self-tests/Logging not supported

Disk da2 is gone so ill see if i can replace that with another drive and if that solves it. Unfortunelly this ESXI host runs my firewall/router and another few things that i need for day to day usage so not the most convenient thing to reboot.

sretalla · Aug 8, 2023

Only 3 disks with no errors in SMART... that's possibly involved here.

I see Multi-zone and read errors.

da2 is certainly the worst impacted and I suspect may be the one stopping everything with timeouts... can you get it to run without that drive connected?

Grinas · Aug 8, 2023

sretalla said:
Only 3 disks with no errors in SMART... that's possibly involved here.

I see Multi-zone and read errors.

da2 is certainly the worst impacted and I suspect may be the one stopping everything with timeouts... can you get it to run without that drive connected?

Tried to shutdown Vm but it was stuck after 30+ mins so had to power it off in ESXI and then shut down host and removed that drive and replaced with a spare.

Seems new truenas vm is very slow to boot or wont boot like the other one as been waiting 20+ minutes for ssh and UI to respond.

these are the errors i get in truenas console in ESXI.

sretalla · Aug 8, 2023

Looks like your boot pool is in trouble again...

Important Announcement for the TrueNAS Community.

truenas not starting due to "uncorrectable I/O failure and has been suspended"

Contributor

MVP

Contributor

MVP

Contributor

Attachments

Powered by Neutrality

Contributor

Attachments

Powered by Neutrality

Contributor

Powered by Neutrality

Contributor

Powered by Neutrality

Powered by Neutrality

Contributor

Attachments

Powered by Neutrality

Contributor

Contributor

Powered by Neutrality

Contributor

Attachments

Powered by Neutrality

Important Announcement for the TrueNAS Community.

Related topics on forums.truenas.com for thread: "truenas not starting due to "uncorrectable I/O failure and has been suspended""

Similar threads