SOLVED Please Help! Pool Status Offline

[bent98]

I have a config back up from 13.0-U3.1 and I can boot to that one. Would that help?

 

[bent98]

Why would I get I/O Error? I also tried reseating in all the SATA cables.

 

[bent98]

I see nothing wrong with the HDDs, but the last long test appears to be around 5 days old. You could try running a long test on each drive and see if errors pop out.

Puzzled about the situation.

How do I do that?

 

[Davvo]

I have a config back up from 13.0-U3.1 and I can boot to that one. Would that help?

Unlikely, but trying shouldn't do any damage. Do not apply the config backup unless you have a U6.1 as well.

Why would I get I/O Error?

Knowing that would be helpful.
Do you get anything with zpool history BIGASSRAID?

How do I do that?

smartctl -t long /dev/ada0
smartctl -t long /dev/ada1
...

 

[bent98]

Knowing that would be helpful.
Do you get anything with zpool history BIGASSRAID?
...

gives me cannot open 'BIGASSRAID': no such pool

 

[Davvo]

I'm out of ideas right now, and it's late in my time zone. There are users way more capable than me, some of which have already posted here: I will let them cook; since it's 25/12 (Merry Christmas!) you will maybe have to wait a bit.
Hope you will get your data back although I am not optimistic about it.

 

[bent98]

I took a video on my phone when its booting up and then took screenshots

here

 

[rvassar]

How old is this pool? The drives are at 53k hours, so I'm thinking ~6 years. I'm wondering if it's old enough to hit a backwards compatibility bug with the features flags. That isn't anything I can offer a recommended action on, but maybe someone will come along with a Christmas miracle for you.

I would try booting the previous update. I believe you should be able to interrupt the boot process and select an older kernel, etc... If that works and the pool imports, I'd run a scrub. If the scrub comes thru clean, I'd do my homework on maybe a pool feature flags update after the guests leave.

If the previous running version throws the same errors.... You're probably in a restore from backup scenario.

 

[bent98]

I’ve been steady upgrading over the years . No issues . I tried booting into truenas 13.0 u3 which was stable for many number of months . And tried to do a pool import with the same error . Are you recommending something else ?

 

[rvassar]

I’ve been steady upgrading over the years . No issues . I tried booting into truenad 13.0 u3 which was stable for many number of months . And tried to do a pool imprint with the same error . Are you recommending something else ?

I was referring to upgrading the pool features, not the version of TrueNAS. But that is something you only want to try on a healthy imported pool. If you moved back to the last stable running version and it still doesn't import, you're in trouble. I would go ahead and run the long SMART tests.

 

[bent98]

I’ve been steady upgrading over the years . No issues . I tried booting into truenad 13.0 u3 which was stable for many number of months . And tried to do a pool imprint with the same error . Are you recommending something else

I was referring to upgrading the pool features, not the version of TrueNAS. But that is something you only want to try on a healthy imported pool. If you moved back to the last stable running version and it still doesn't import, you're in trouble. I would go ahead and run the long SMART tests.

i am running long tests now

 

[HoneyBadger]

Bad block pointers and attempting pool rewind already. That's not a good sign for the health of the pool I'm afraid.

Show : Here be Dragons

Set these two sysctl's to speed up pool rewind on import

Code:

sysctl vfs.zfs.spa.load_verify_metadata=0
sysctl vfs.zfs.spa.load_verify_data=0

then do zpool import -FXfn BIGASSRAID and after some time it should kick back

Code:

Would be able to return pool to its state as of (INSERT DATE AND TIME HERE)
Would discard approximately N seconds of transations.

Make your judgement call if you can live with that, then issue
zpool import -FXf BIGASSRAID

If it imports, reboot the system through the webUI, make sure your pool re-imports automatically, and fire off a scrub.

Best of luck.

 

[joeschmuck]

I'm with @HoneyBadger , I saw the block pointer errors and that is not a good thing.

If at anytime you are able to gain access to your files, BACK THEM UP IMMEDIATELY somewhere because you never know if you will get the system working for any length of time right now.

Please post in code brackets the file

Code:

/var/logs/messages

and it will be a long file. It is a simple text file. I don't think there is an easy way for you to grab this file as you do not have any dataset shares at this moment. I tried to create an SMB share of /var/logs but that failed.

What you can do is enter

Code:

cat /var/logs/messages | more

and then copy and paste the data page by page into a simple text file. It is time consuming but I can't think of any other options. Maybe you could email it to yourself using the 'mail' command, eh, it's late, I think this is possible but I'm tired and I don't want to give you any bad commands.

Okay, from the command line/shell

Code:

mail -s "Messages File" youremailaddress.com < /var/log/messages

and you can use this to export any text file. "Messages File" is the Subject Line.

If you need to create a text file, enter the command you want to run followed by

Code:

 > /tmp/my_new_file_name

and then you can use that file path/name to mail a file. This creates a file in the /tmp directory which is a RAM Disk and goes away upon reboot, so the files you create will not stick around.

What I'd like to see is if there is only one drive throwing errors and if we could identify that, just disconnect the data cable and boot the system up. Maybe it will work? I'm grasping at straws. There are other people better at this than myself. Let's hope for a Christmas Miracle.

 

[bent98]

Bad block pointers and attempting pool rewind already. That's not a good sign for the health of the pool I'm afraid.

Show : Here be Dragons

Set these two sysctl's to speed up pool rewind on import

Code:

sysctl vfs.zfs.spa.load_verify_metadata=0
sysctl vfs.zfs.spa.load_verify_data=0

then do zpool import -FXfn BIGASSRAID and after some time it should kick back

Code:

Would be able to return pool to its state as of (INSERT DATE AND TIME HERE)
Would discard approximately N seconds of transations.

Make your judgement call if you can live with that, then issue
zpool import -FXf BIGASSRAID

If it imports, reboot the system through the webUI, make sure your pool re-imports automatically, and fire off a scrub.

Best of luck.

I was running a long smartctl -t long /dev/adax. I guess I'll wait until thats over first which will be 2pm EST today and then if there are no errors, execute on your suggestion. How do I retrieve the results of the long tests?

 

[joeschmuck]

I was running a long smartctl -t long /dev/adax. I guess I'll wait until thats over first which will be 2pm EST today

No need to wait, just do not reboot your machine and the smart test will complete. It runs in the background if you want to think of it that way.

Please run the mail command using your email address and copy the data into a text file, then as an attachment here. Hopefully this will help us out.

To get the smart data form the test is the same as you retrieve any smart result

Code:

smartctl -a /dev/adax

or you can use -x vice -a for more information. Right now we are looking to ensure the smart long/extended test completes without errors. Even if it completes without errors, all it means is your drives are in good working order, it says nothing to if the data is corrupt or not.

Merry Christmas.

 

[bent98]

I did manage to find a back up config from 13-u3. Would that help in restoring pool if I went back to u3 version ?

Merry Christmas

 

[bent98]

No need to wait, just do not reboot your machine and the smart test will complete. It runs in the background if you want to think of it that way.

Please run the mail command using your email address and copy the data into a text file, then as an attachment here. Hopefully this will help us out.

To get the smart data form the test is the same as you retrieve any smart result

Code:

smartctl -a /dev/adax

or you can use -x vice -a for more information. Right now we are looking to ensure the smart long/extended test completes without errors. Even if it completes without errors, all it means is your drives are in good working order, it says nothing to if the data is corrupt or not.

Merry Christmas.

Code:

Dec 24 18:23:26 freenas newsyslog[1177]: logfile first created
Dec 24 18:23:26 freenas syslog-ng[1287]: syslog-ng starting up; version='3.35.1'
Dec 24 18:23:26 freenas
Dec 24 18:23:26 freenas Waiting (max 60 seconds) for system process `vnlru' to stop... done
Dec 24 18:23:26 freenas Waiting (max 60 seconds) for system process `syncer' to stop...
Dec 24 18:23:26 freenas Syncing disks, vnodes remaining... 0 0 0 done
Dec 24 18:23:26 freenas All buffers synced.
Dec 24 18:23:26 freenas Uptime: 54m20s
Dec 24 18:23:26 freenas ---<<BOOT>>---
Dec 24 18:23:26 freenas Copyright (c) 1992-2021 The FreeBSD Project.
Dec 24 18:23:26 freenas Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Dec 24 18:23:26 freenas The Regents of the University of California. All rights reserved.
Dec 24 18:23:26 freenas FreeBSD is a registered trademark of The FreeBSD Foundation.
Dec 24 18:23:26 freenas FreeBSD 13.1-RELEASE-p2 n245412-484f039b1d0 TRUENAS amd64
Dec 24 18:23:26 freenas FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
Dec 24 18:23:26 freenas VT(vga): text 80x25
Dec 24 18:23:26 freenas CPU: Intel(R) Core(TM) i3-6100 CPU @ 3.70GHz (3700.00-MHz K8-class CPU)
Dec 24 18:23:26 freenas Origin="GenuineIntel" Id=0x506e3 Family=0x6 Model=0x5e Stepping=3
Dec 24 18:23:26 freenas Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Dec 24 18:23:26 freenas Features2=0x7ffafbbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
Dec 24 18:23:26 freenas AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
Dec 24 18:23:26 freenas AMD Features2=0x121<LAHF,ABM,Prefetch>
Dec 24 18:23:26 freenas Structured Extended Features=0x29c67af<FSGSBASE,TSCADJ,SGX,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
Dec 24 18:23:26 freenas XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
Dec 24 18:23:26 freenas VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
Dec 24 18:23:26 freenas TSC: P-state invariant, performance statistics
Dec 24 18:23:26 freenas real memory = 21474836480 (20480 MB)
Dec 24 18:23:26 freenas avail memory = 20664094720 (19706 MB)
Dec 24 18:23:26 freenas Event timer "LAPIC" quality 600
Dec 24 18:23:26 freenas ACPI APIC Table: <ALASKA A M I >
Dec 24 18:23:26 freenas FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
Dec 24 18:23:26 freenas FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 hardware threads
Dec 24 18:23:26 freenas random: registering fast source Intel Secure Key RNG
Dec 24 18:23:26 freenas random: fast provider: "Intel Secure Key RNG"
Dec 24 18:23:26 freenas random: unblocking device.
Dec 24 18:23:26 freenas ioapic0 <Version 2.0> irqs 0-119
Dec 24 18:23:26 freenas Launching APs: 1 2 3
Dec 24 18:23:26 freenas random: entropy device external interface
Dec 24 18:23:26 freenas kbd0 at kbdmux0
Dec 24 18:23:26 freenas vtvga0: <VT VGA driver>
Dec 24 18:23:26 freenas aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>
Dec 24 18:23:26 freenas padlock0: No ACE support.
Dec 24 18:23:26 freenas acpi0: <ALASKA A M I >
Dec 24 18:23:26 freenas acpi0: Power Button (fixed)
Dec 24 18:23:26 freenas cpu0: <ACPI CPU> on acpi0
Dec 24 18:23:26 freenas hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Dec 24 18:23:26 freenas Timecounter "HPET" frequency 24000000 Hz quality 950
Dec 24 18:23:26 freenas Event timer "HPET" frequency 24000000 Hz quality 550
Dec 24 18:23:26 freenas Event timer "HPET1" frequency 24000000 Hz quality 440
Dec 24 18:23:26 freenas Event timer "HPET2" frequency 24000000 Hz quality 440
Dec 24 18:23:26 freenas Event timer "HPET3" frequency 24000000 Hz quality 440
Dec 24 18:23:26 freenas Event timer "HPET4" frequency 24000000 Hz quality 440
Dec 24 18:23:26 freenas atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
Dec 24 18:23:26 freenas atrtc0: Warning: Couldn't map I/O.
Dec 24 18:23:26 freenas atrtc0: registered as a time-of-day clock, resolution 1.000000s
Dec 24 18:23:26 freenas Event timer "RTC" frequency 32768 Hz quality 0
Dec 24 18:23:26 freenas attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Dec 24 18:23:26 freenas Timecounter "i8254" frequency 1193182 Hz quality 0
Dec 24 18:23:26 freenas Event timer "i8254" frequency 1193182 Hz quality 100
Dec 24 18:23:26 freenas apei0: <ACPI Platform Error Interface> on acpi0
Dec 24 18:23:26 freenas Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
Dec 24 18:23:26 freenas acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
Dec 24 18:23:26 freenas pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
Dec 24 18:23:26 freenas pci0: <ACPI PCI bus> on pcib0
Dec 24 18:23:26 freenas pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
Dec 24 18:23:26 freenas pci1: <ACPI PCI bus> on pcib1
Dec 24 18:23:26 freenas pcib2: <ACPI PCI-PCI bridge> irq 16 at device 1.1 on pci0
Dec 24 18:23:26 freenas pci2: <ACPI PCI bus> on pcib2
Dec 24 18:23:26 freenas ahci0: <Marvell 88SE9235 AHCI SATA controller> port 0xe050-0xe057,0xe040-0xe043,0xe030-0xe037,0xe020-0xe023,0xe000-0xe01f mem 0xdf140000-0xdf1407ff irq 17 at device 0.0 on pci2
Dec 24 18:23:26 freenas ahci0: AHCI v1.00 with 4 6Gbps ports, Port Multiplier supported with FBS
Dec 24 18:23:26 freenas ahcich0: <AHCI channel> at channel 0 on ahci0
Dec 24 18:23:26 freenas ahcich1: <AHCI channel> at channel 1 on ahci0
Dec 24 18:23:26 freenas ahcich2: <AHCI channel> at channel 2 on ahci0
Dec 24 18:23:26 freenas ahcich3: <AHCI channel> at channel 3 on ahci0
Dec 24 18:23:26 freenas vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xde000000-0xdeffffff,0xc0000000-0xcfffffff irq 16 at device 2.0 on pci0
Dec 24 18:23:26 freenas vgapci0: Boot video device
Dec 24 18:23:26 freenas xhci0: <Intel Sunrise Point USB 3.0 controller> mem 0xdf220000-0xdf22ffff irq 16 at device 20.0 on pci0
Dec 24 18:23:26 freenas xhci0: 32 bytes context size, 64-bit DMA
Dec 24 18:23:26 freenas usbus0 on xhci0
Dec 24 18:23:26 freenas usbus0: 5.0Gbps Super Speed USB v3.0
Dec 24 18:23:26 freenas pci0: <simple comms> at device 22.0 (no driver attached)
Dec 24 18:23:26 freenas uart2: <100 Series/C230 Series Chipset Family KT Redirection> port 0xf0a0-0xf0a7 mem 0xdf23f000-0xdf23ffff irq 19 at device 22.3 on pci0
Dec 24 18:23:26 freenas uart2: Using 1 MSI message
Dec 24 18:23:26 freenas ahci1: <Intel Sunrise Point AHCI SATA controller> port 0xf090-0xf097,0xf080-0xf083,0xf060-0xf07f mem 0xdf230000-0xdf231fff,0xdf23e000-0xdf23e0ff,0xdf23d000-0xdf23d7ff irq 16 at device 23.0 on pci0
Dec 24 18:23:26 freenas ahci1: AHCI v1.31 with 6 6Gbps ports, Port Multiplier not supported
Dec 24 18:23:26 freenas ahcich4: <AHCI channel> at channel 0 on ahci1
Dec 24 18:23:26 freenas ahcich5: <AHCI channel> at channel 1 on ahci1
Dec 24 18:23:26 freenas ahcich6: <AHCI channel> at channel 2 on ahci1
Dec 24 18:23:26 freenas ahcich7: <AHCI channel> at channel 3 on ahci1
Dec 24 18:23:26 freenas ahcich8: <AHCI channel> at channel 4 on ahci1
Dec 24 18:23:26 freenas ahcich9: <AHCI channel> at channel 5 on ahci1
Dec 24 18:23:26 freenas ahciem0: <AHCI enclosure management bridge> on ahci1
Dec 24 18:23:26 freenas pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.0 on pci0
Dec 24 18:23:26 freenas pci3: <ACPI PCI bus> on pcib3
Dec 24 18:23:26 freenas em0: <Intel(R) PRO/1000 PT 82571EB/82571GB (Copper)> port 0xd020-0xd03f mem 0xdf060000-0xdf07ffff,0xdf040000-0xdf05ffff irq 18 at device 0.0 on pci3
Dec 24 18:23:26 freenas em0: EEPROM V5.6-2
Dec 24 18:23:26 freenas em0: Using 1024 TX descriptors and 1024 RX descriptors
Dec 24 18:23:26 freenas em0: Using an MSI interrupt
Dec 24 18:23:26 freenas kernel: em0: Ethernet address: 00:15:17:2e:f0:0e
Dec 24 18:23:26 freenas em1: <Intel(R) PRO/1000 PT 82571EB/82571GB (Copper)> port 0xd000-0xd01f mem 0xdf020000-0xdf03ffff,0xdf000000-0xdf01ffff irq 19 at device 0.1 on pci3
Dec 24 18:23:26 freenas em1: EEPROM V5.6-2
Dec 24 18:23:26 freenas em1: Using 1024 TX descriptors and 1024 RX descriptors
Dec 24 18:23:26 freenas em1: Using an MSI interrupt
Dec 24 18:23:26 freenas kernel: em1: Ethernet address: 00:15:17:2e:f0:0f
Dec 24 18:23:26 freenas isab0: <PCI-ISA bridge> at device 31.0 on pci0
Dec 24 18:23:26 freenas isa0: <ISA bus> on isab0
Dec 24 18:23:26 freenas pci0: <memory> at device 31.2 (no driver attached)
Dec 24 18:23:26 freenas em2: <Intel(R) I219-LM SPT-H(2)> mem 0xdf200000-0xdf21ffff irq 16 at device 31.6 on pci0
Dec 24 18:23:26 freenas em2: EEPROM V0.8-4
Dec 24 18:23:26 freenas em2: Using 1024 TX descriptors and 1024 RX descriptors
Dec 24 18:23:26 freenas em2: Using an MSI interrupt
Dec 24 18:23:26 freenas kernel: em2: Ethernet address: e0:07:1b:ff:26:7e
Dec 24 18:23:26 freenas pci0: <unknown> at device 31.7 (no driver attached)
Dec 24 18:23:26 freenas acpi_button0: <Sleep Button> on acpi0
Dec 24 18:23:26 freenas acpi_button1: <Power Button> on acpi0
Dec 24 18:23:26 freenas acpi_tz0: <Thermal Zone> on acpi0
Dec 24 18:23:26 freenas acpi_tz1: <Thermal Zone> on acpi0
Dec 24 18:23:26 freenas orm0: <ISA Option ROMs> at iomem 0xc0000-0xcffff,0xd0000-0xd2fff,0xd3000-0xd3fff pnpid ORM0000 on isa0
Dec 24 18:23:26 freenas coretemp0: <CPU On-Die Thermal Sensors> on cpu0
Dec 24 18:23:26 freenas hwpstate_intel0: <Intel Speed Shift> on cpu0
Dec 24 18:23:26 freenas hwpstate_intel1: <Intel Speed Shift> on cpu1
Dec 24 18:23:26 freenas hwpstate_intel2: <Intel Speed Shift> on cpu2
Dec 24 18:23:26 freenas hwpstate_intel3: <Intel Speed Shift> on cpu3
Dec 24 18:23:26 freenas Timecounter "TSC-low" frequency 1847999864 Hz quality 1000
Dec 24 18:23:26 freenas Timecounters tick every 1.000 msec
Dec 24 18:23:26 freenas ZFS filesystem version: 5
Dec 24 18:23:26 freenas ZFS storage pool version: features support (5000)
Dec 24 18:23:26 freenas ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
Dec 24 18:23:26 freenas ugen0.1: <Intel XHCI root HUB> at usbus0
Dec 24 18:23:26 freenas uhub0 on usbus0
Dec 24 18:23:26 freenas uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Dec 24 18:23:26 freenas Trying to mount root from zfs:freenas-boot/ROOT/13.0-U3.1 []...
Dec 24 18:23:26 freenas ses0 at ahciem0 bus 0 scbus10 target 0 lun 0
Dec 24 18:23:26 freenas ses0: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
Dec 24 18:23:26 freenas ses0: SEMB SES Device
Dec 24 18:23:26 freenas ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
Dec 24 18:23:26 freenas ada0: <WDC WD80EFZX-68UW8N0 83.H0A83> ACS-2 ATA SATA 3.x device
Dec 24 18:23:26 freenas ada0: Serial Number VK1EX8HY
Dec 24 18:23:26 freenas ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Dec 24 18:23:26 freenas ada0: Command Queueing enabled
Dec 24 18:23:26 freenas ada0: 7630885MB (15628053168 512 byte sectors)
Dec 24 18:23:26 freenas ses0: pass3 in 'Slot 01', SATA Slot: scbus5 target 0
Dec 24 18:23:26 freenas ses0: pass4 in 'Slot 02', SATA Slot: scbus6 target 0
Dec 24 18:23:26 freenas ada1 at ahcich2 bus 0 scbus2 target 0 lun 0
Dec 24 18:23:26 freenas ada1: <WDC WD80EFZX-68UW8N0 83.H0A83> ACS-2 ATA SATA 3.x device
Dec 24 18:23:26 freenas ada1: Serial Number VK1GHX4Y
Dec 24 18:23:26 freenas ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Dec 24 18:23:26 freenas ada1: Command Queueing enabled
Dec 24 18:23:26 freenas ada1: 7630885MB (15628053168 512 byte sectors)
Dec 24 18:23:26 freenas ses0: pass5 in 'Slot 03', SATA Slot: scbus7 target 0
Dec 24 18:23:26 freenas ada2 at ahcich3 bus 0 scbus3 target 0 lun 0
Dec 24 18:23:26 freenas ada2: <WDC WD80EFZX-68UW8N0 83.H0A83> ACS-2 ATA SATA 3.x device
Dec 24 18:23:26 freenas ada2: Serial Number VJH5RDPX
Dec 24 18:23:26 freenas ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Dec 24 18:23:26 freenas ada2: Command Queueing enabled
Dec 24 18:23:26 freenas ada2: 7630885MB (15628053168 512 byte sectors)
Dec 24 18:23:26 freenas ses0: pass6 in 'Slot 04', SATA Slot: scbus8 target 0
Dec 24 18:23:26 freenas ada3 at ahcich5 bus 0 scbus5 target 0 lun 0
Dec 24 18:23:26 freenas ada3: <SanDisk SDSSDA120G Z33130RL> ACS-2 ATA SATA 3.x device
Dec 24 18:23:26 freenas ada3: Serial Number 172773801991
Dec 24 18:23:26 freenas ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
Dec 24 18:23:26 freenas ada3: Command Queueing enabled
Dec 24 18:23:26 freenas ada3: 114473MB (234441648 512 byte sectors)
Dec 24 18:23:26 freenas ada4 at ahcich6 bus 0 scbus6 target 0 lun 0
Dec 24 18:23:26 freenas ada4: <WDC WD80EFZX-68UW8N0 83.H0A83> ACS-2 ATA SATA 3.x device
Dec 24 18:23:26 freenas ada4: Serial Number VK1GJA0Y
Dec 24 18:23:26 freenas ada4: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Dec 24 18:23:26 freenas ada4: Command Queueing enabled
Dec 24 18:23:26 freenas ada4: 7630885MB (15628053168 512 byte sectors)
Dec 24 18:23:26 freenas ada5 at ahcich7 bus 0 scbus7 target 0 lun 0
Dec 24 18:23:26 freenas ada5: <WDC WD80EFZX-68UW8N0 83.H0A83> ACS-2 ATA SATA 3.x device
Dec 24 18:23:26 freenas ada5: Serial Number VK1EXGGY
Dec 24 18:23:26 freenas ada5: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Dec 24 18:23:26 freenas ada5: Command Queueing enabled
Dec 24 18:23:26 freenas ada5: 7630885MB (15628053168 512 byte sectors)
Dec 24 18:23:26 freenas ada6 at ahcich8 bus 0 scbus8 target 0 lun 0
Dec 24 18:23:26 freenas ada6: <WDC WD80EFZX-68UW8N0 83.H0A83> ACS-2 ATA SATA 3.x device
Dec 24 18:23:26 freenas ada6: Serial Number VK1EBZTY
Dec 24 18:23:26 freenas ada6: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Dec 24 18:23:26 freenas ada6: Command Queueing enabled
Dec 24 18:23:26 freenas ada6: 7630885MB (15628053168 512 byte sectors)
Dec 24 18:23:26 freenas ada7 at ahcich9 bus 0 scbus9 target 0 lun 0
Dec 24 18:23:26 freenas ada7: <WDC WD80EFZX-68UW8N0 83.H0A83> ACS-2 ATA SATA 3.x device
Dec 24 18:23:26 freenas ada7: Serial Number VK1EH4BY
Dec 24 18:23:26 freenas ada7: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Dec 24 18:23:26 freenas ada7: Command Queueing enabled
Dec 24 18:23:26 freenas ada7: 7630885MB (15628053168 512 byte sectors)
Dec 24 18:23:26 freenas ses0: pass7,ada7 in 'Slot 05', SATA Slot: scbus9 target 0
Dec 24 18:23:26 freenas Root mount waiting for: usbus0
Dec 24 18:23:26 freenas uhub0: 26 ports with 26 removable, self powered
Dec 24 18:23:26 freenas ugen0.2: <CPS CP 1500C> at usbus0
Dec 24 18:23:26 freenas Root mount waiting for: usbus0
Dec 24 18:23:26 freenas ugen0.3: <Chicony Saitek Eclipse Keyboard> at usbus0
Dec 24 18:23:26 freenas ukbd0 on uhub0
Dec 24 18:23:26 freenas ukbd0: <Chicony Saitek Eclipse Keyboard, class 0/0, rev 2.00/1.30, addr 2> on usbus0
Dec 24 18:23:26 freenas kbd1 at ukbd0
Dec 24 18:23:26 freenas uhid0 on uhub0
Dec 24 18:23:26 freenas uhid0: <Chicony Saitek Eclipse Keyboard, class 0/0, rev 2.00/1.30, addr 2> on usbus0
Dec 24 18:23:26 freenas ichsmb0: <Intel Sunrise Point-H SMBus controller> port 0xf040-0xf05f mem 0xdf23c000-0xdf23c0ff irq 16 at device 31.4 on pci0
Dec 24 18:23:26 freenas smbus0: <System Management Bus> on ichsmb0
Dec 24 18:23:26 freenas kernel: lo0: link state changed to UP
Dec 24 18:23:26 freenas kernel: em0: link state changed to UP
Dec 24 18:23:26 freenas debugnet_any_ifnet_update: Bad dn_init result from em0 (ifp 0xfffff80005031800), ignoring.
Dec 24 18:23:26 freenas kernel: em1: link state changed to UP
Dec 24 18:23:26 freenas debugnet_any_ifnet_update: Bad dn_init result from em1 (ifp 0xfffff80004acc800), ignoring.
Dec 24 18:23:26 freenas CPU: Intel(R) Core(TM) i3-6100 CPU @ 3.70GHz (3696.00-MHz K8-class CPU)
Dec 24 18:23:26 freenas Origin="GenuineIntel" Id=0x506e3 Family=0x6 Model=0x5e Stepping=3
Dec 24 18:23:26 freenas Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Dec 24 18:23:26 freenas Features2=0x7ffafbbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
Dec 24 18:23:26 freenas AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
Dec 24 18:23:26 freenas AMD Features2=0x121<LAHF,ABM,Prefetch>
Dec 24 18:23:26 freenas Structured Extended Features=0x29c67af<FSGSBASE,TSCADJ,SGX,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
Dec 24 18:23:26 freenas Structured Extended Features3=0x9c002e00<MCUOPT,MD_CLEAR,TSXFA,IBPB,STIBP,L1DFL,SSBD>
Dec 24 18:23:26 freenas XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
Dec 24 18:23:26 freenas VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
Dec 24 18:23:26 freenas TSC: P-state invariant, performance statistics
Dec 24 18:23:26 freenas kernel: lagg0: link state changed to UP
Dec 24 18:23:26 freenas kernel: em1: link state changed to DOWN
Dec 24 18:23:26 freenas kernel: lagg0: link state changed to DOWN
Dec 24 18:23:26 freenas kernel: em1: link state changed to UP
Dec 24 18:23:26 freenas kernel: lagg0: link state changed to UP
Dec 24 18:23:26 freenas hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/4/48/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/48/0x67<INT,USR,SYS,REA,WRI>
Dec 24 18:23:26 freenas 1 2023-12-24T18:23:26.849127-05:00 freenas.local ntpd 1354 - - ntpd 4.2.8p15-a (1): Starting
Dec 24 18:23:26 freenas 1 2023-12-24T18:23:26.849429-05:00 freenas.local ntpd 1354 - - Command line: /usr/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f /var/db/ntp/ntpd.drift -g
Dec 24 18:23:26 freenas 1 2023-12-24T18:23:26.849442-05:00 freenas.local ntpd 1354 - - ----------------------------------------------------
Dec 24 18:23:26 freenas 1 2023-12-24T18:23:26.849451-05:00 freenas.local ntpd 1354 - - ntp-4 is maintained by Network Time Foundation,
Dec 24 18:23:26 freenas 1 2023-12-24T18:23:26.849470-05:00 freenas.local ntpd 1354 - - Inc. (NTF), a non-profit 501(c)(3) public-benefit
Dec 24 18:23:26 freenas 1 2023-12-24T18:23:26.849482-05:00 freenas.local ntpd 1354 - - corporation. Support and training for ntp-4 are
Dec 24 18:23:26 freenas 1 2023-12-24T18:23:26.849498-05:00 freenas.local ntpd 1354 - - available at https://www.nwtime.org/support
Dec 24 18:23:26 freenas 1 2023-12-24T18:23:26.849510-05:00 freenas.local ntpd 1354 - - ----------------------------------------------------
Dec 24 18:23:27 freenas Security policy loaded: MAC/ntpd (mac_ntpd)
Dec 24 18:23:53 freenas 1 2023-12-24T18:23:53.921439-05:00 freenas.local ntpd 1355 - - restrict: ignoring line 10, address/host '0.freebsd.pool.ntp.org' unusable.
Dec 24 18:23:57 freenas proftpd[1399]: 127.0.0.1 - ProFTPD 1.3.7c (maint) (built Fri Nov 11 2022 15:28:34 UTC) standalone mode STARTUP
Dec 24 18:23:58 freenas 1 2023-12-24T18:23:58.407094-05:00 freenas.local daemon 1461 - - 2023-12-24 18:23:58,406:wsdd WARNING(pid 1462): no interface given, using all interfaces
Dec 24 18:24:07 freenas 1 2023-12-24T18:24:07.459304-05:00 freenas.local daemon 1693 - - 2023-12-24 18:24:07,459:wsdd WARNING(pid 1694): no interface given, using all interfaces
Dec 25 00:00:00 freenas syslog-ng[1287]: Configuration reload request received, reloading configuration;
Dec 25 00:00:00 freenas syslog-ng[1287]: Configuration reload finished;

 

[joeschmuck]

I did manage to find a back up config from 13-u3. Would that help in restoring pool if I went back to u3 version ?

Merry Christmas

You can use that to restore any version of 13.x or greater, even SCALE if you really wanted to. I'm not sure about going back to 12.x as I've never tried that. The file will restore all the user accounts the way they were when that config file was created. You should have two files in the compressed file. One is your configuration and one is a password file, assuming you saved that.

You can save a new config file right now if you desire and it should be saved with today's date on it. I don't think you config file is corrupt but it's always good to be cautious.

 

[bent98]

You can use that to restore any version of 13.x or greater, even SCALE if you really wanted to. I'm not sure about going back to 12.x as I've never tried that. The file will restore all the user accounts the way they were when that config file was created. You should have two files in the compressed file. One is your configuration and one is a password file, assuming you saved that.

You can save a new config file right now if you desire and it should be saved with today's date on it. I don't think you config file is corrupt but it's always good to be cautious.

Sorry for asking dumb questions as I am a noob when it comes to this and trying to follow what is being recommended.

If all my disks check out as good after the long test, should I try to restore from config or do what honeybadger is recommending below?

Code:
sysctl vfs.zfs.spa.load_verify_metadata=0
sysctl vfs.zfs.spa.load_verify_data=0

then do zpool import -FXfn BIGASSRAID and after some time it should kick back
Code:
Would be able to return pool to its state as of (INSERT DATE AND TIME HERE)
Would discard approximately N seconds of transations.

Make your judgement call if you can live with that, then issue
zpool import -FXf BIGASSRAID

 

[joeschmuck]

I would recommend you do what @HoneyBadger is suggesting. It may be the only way to recover your data. Make sure you wait the time for it to respond. I don't know if it takes 2 minutes or 6 hours. I suspect it depends on how much data you have.

In the meantime while waiting on your drives to finish SMART testing, the data you attached for the 'messages' file had no usable data that I could see to help. Let's try this one:

Code:

mail -s "dmesg.today" youremailaddr < /var/log/dmesg.today

and do the same for file

Code:

/var/log/dmesg.yesterday

and why not add one more file,

Code:

/var/log/console.log

as I suspect that will list the data you previously posted.

You can take a look in this log file area and see all the files there. To look at them on the system you just need to type

Code:

cat filename | more

and it will display the file on the screen one page at a time. Press the SPACE BAR to advance to the next screen. I don't look in this area often but you can see there are some interesting things here.