I have a 4-bay iXsystems FreeNAS mini from 2017 running 13.x core.
I've been running and updating this box since 2017 with no problems. As far as I can tell all drives and hardware are fine.
Recently I updated to 13.something core (I can't see the exact version since I can't boot)
I had updated and rebooted with no problem at least a couple times without a problem, so the update to 13.0 or 13.1 or whatever it was seemed to have gone fine.
The last things I was doing before it failed to reboot was I had tried to install pkg and ports on the host and had created a new base jail to run unifi controller, which was all actually working fine. I had that running for a couple days before the problem.
The problem happened when I noticed that the host console was outputting a constant repeating error message about a killed syslog-ng process. I assume that was caused either by the 13.x upgrade or by my messing with pkg and ports. I didn't intentionally touch anything related to syslog but who knows maybe just running pkg upgrade or installing some other pkgdid something.
Anyway, the last thing I did before failing to boot was I noticed the syslog-ng process on the host console log window in the gui, removed all old boot environments (12.x and lower) from the gui, and then did an intentional/graceful reboot using the reboot button in the gui, and that reboot halted at mountroot> and I've never been able to get past that since then. It looks like it's successfully loading the bootloader and even the kernel, and only failing at mounting to root fs.
What do I do now?
Here is a capture from the serial console:
There is no usb devices plugged in. Until just now I'd never opened the case even. There's no usb boot drive inside. It's a small pcb plugged into an sata plug on the mother board, which appears to be a 16G ssd. The hardware, other than the main drives, is all right from iXsystems new and until just now I'd never even broken the warranty sticker. Inside everything looks fine, not even dusty, fan works etc. It's been powered from a apc smart-ups 2u rackmount the whole time and there was no storm or power event recently. I believe the problem must be the result of deleting the old boot environments. Though I don't know why since I'm definitely running 13.x and I definitely only removed some 12.x and 11.x environments. The main drives are 4 x 4T wd red 3.5" NASware (well you can see those in the console log so nevermind...)
It looks like, the kernel says it's 13.1, but the failing mountroot says it's trying to mount something with 13.0 in the name. No idea how that came about.
What can I do from here?
I actually don't mind reinstalling the host and all jails as long as I can preserve the main zfs pool. I don't have a recent config backup, but I do have some old config backups, from before the 13.0 update. The few jails I have don't have anything irreplaceable in the jails themselves. I can lose the host and all jails but do not want to lose the main zfs data pool.
I've been running and updating this box since 2017 with no problems. As far as I can tell all drives and hardware are fine.
Recently I updated to 13.something core (I can't see the exact version since I can't boot)
I had updated and rebooted with no problem at least a couple times without a problem, so the update to 13.0 or 13.1 or whatever it was seemed to have gone fine.
The last things I was doing before it failed to reboot was I had tried to install pkg and ports on the host and had created a new base jail to run unifi controller, which was all actually working fine. I had that running for a couple days before the problem.
The problem happened when I noticed that the host console was outputting a constant repeating error message about a killed syslog-ng process. I assume that was caused either by the 13.x upgrade or by my messing with pkg and ports. I didn't intentionally touch anything related to syslog but who knows maybe just running pkg upgrade or installing some other pkgdid something.
Anyway, the last thing I did before failing to boot was I noticed the syslog-ng process on the host console log window in the gui, removed all old boot environments (12.x and lower) from the gui, and then did an intentional/graceful reboot using the reboot button in the gui, and that reboot halted at mountroot> and I've never been able to get past that since then. It looks like it's successfully loading the bootloader and even the kernel, and only failing at mounting to root fs.
What do I do now?
Here is a capture from the serial console:
Code:
F ______ ____ _____ _____ - | ____| | _ \ / ____| __ \ | |___ _ __ ___ ___ | |_) | (___ | | | | | ___| '__/ _ \/ _ \| _ < \___ \| | | | | | | | | __/ __/| |_) |____) | |__| | | | | | | | || | | | |_| |_| \___|\___||____/|_____/|_____/ ``` ` +-------------------------- Welcome to FreeBSD ----------------------------+/ | +o .--` /y:` |. | 1. Boot FreeBSD [Enter] yo`:. :o `+| | 2. Boot FreeBSD (Serial Console) y/ -/` -o/| | 3. Escape to loader prompt .- ::/sy+:| | 4. Reboot / `-- | | `: |` | Options: `: |` | 5. Kernel: default/kernel (1 of 2) / | | 6. Boot Options .- -| | -- -.| | `:` `:` | | .-- `--. | | .---.....----. | +--------------------------------------------------------------------------+ Loading kernel... /boot/kernel/kernel text=0x1830d8 text=0xe03b10 text=0x8270fc data=0x140 data=0x25d3bc+0x5a1c44 syms=[0x8+0x1b3090+0x8+0x20d623] Loading configured modules... /boot/entropy size=0x1000 can't find '/etc/hostid' KDB: debugger backends: ddb KDB: current backend: ddb ---<<BOOT>>--- Copyright (c) 1992-2021 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 13.1-RELEASE n245376-eba770b30ff TRUENAS amd64 FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303) VT(vga): resolution 640x480 CPU: Intel(R) Atom(TM) CPU C2750 @ 2.40GHz (2400.10-MHz K8-class CPU) Origin="GenuineIntel" Id=0x406d8 Family=0x6 Model=0x4d Stepping=8 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> AMD Features2=0x101<LAHF,Prefetch> Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 17179869184 (16384 MB) avail memory = 16596303872 (15827 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <INTEL TIANO > WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1) FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 1 package(s) x 8 core(s) random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" random: unblocking device. ioapic0 <Version 2.0> irqs 0-23 Launching APs: 3 1 2 5 7 4 6 random: entropy device external interface kbd1 at kbdmux0 vtvga0: <VT VGA driver> aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> padlock0: No ACE support. acpi0: <ALASKA A M I > acpi0: Power Button (fixed) cpu0: <ACPI CPU> on acpi0 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0 atrtc0: Warning: Couldn't map I/O. atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib1: <ACPI PCI-PCI bridge> mem 0xdf740000-0xdf75ffff irq 16 at device 1.0 on pci0 pci1: <ACPI PCI bus> on pcib1 pcib2: <ACPI PCI-PCI bridge> mem 0xdf720000-0xdf73ffff irq 20 at device 3.0 on pci0 pci2: <ACPI PCI bus> on pcib2 pcib3: <ACPI PCI-PCI bridge> mem 0xdf500000-0xdf51ffff irq 22 at device 0.0 on pci2 pci3: <ACPI PCI bus> on pcib3 pcib4: <PCI-PCI bridge> irq 23 at device 1.0 on pci3 pci4: <PCI bus> on pcib4 ahci0: <Marvell 88SE9172 AHCI SATA controller> port 0xc040-0xc047,0xc030-0xc033,0xc020-0xc027,0xc010-0xc013,0xc000-0xc00f mem 0xdf410000-0xdf4101ff irq 23 at device 0.0 on pci4 ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS ahci0: quirks=0x1000000<IOMMU_BUSWIDE> ahcich0: <AHCI channel> at channel 0 on ahci0 ahcich1: <AHCI channel> at channel 1 on ahci0 pcib5: <PCI-PCI bridge> irq 23 at device 5.0 on pci3 pci5: <PCI bus> on pcib5 pcib6: <PCI-PCI bridge> irq 23 at device 0.0 on pci5 pci6: <PCI bus> on pcib6 vgapci0: <VGA-compatible display> port 0xb000-0xb07f mem 0xde000000-0xdeffffff,0xdf000000-0xdf01ffff irq 23 at device 0.0 on pci6 vgapci0: Boot video device pcib7: <PCI-PCI bridge> irq 21 at device 7.0 on pci3 pci7: <PCI bus> on pcib7 igb0: <Intel(R) I210 (Copper)> port 0xa000-0xa01f mem 0xdf300000-0xdf37ffff,0xdf380000-0xdf383fff irq 21 at device 0.0 on pci7 igb0: EEPROM V3.16-0 eTrack 0x800004d6 igb0: Using 1024 TX descriptors and 1024 RX descriptors igb0: Using 4 RX queues 4 TX queues igb0: Using MSI-X interrupts with 5 vectors igb0: Ethernet address: d0:50:99:c2:b5:23 pcib8: <PCI-PCI bridge> irq 23 at device 9.0 on pci3 pci8: <PCI bus> on pcib8 igb1: <Intel(R) I210 (Copper)> port 0x9000-0x901f mem 0xdf200000-0xdf27ffff,0xdf280000-0xdf283fff irq 23 at device 0.0 on pci8 igb1: EEPROM V3.16-0 eTrack 0x800004d6 igb1: Using 1024 TX descriptors and 1024 RX descriptors igb1: Using 4 RX queues 4 TX queues igb1: Using MSI-X interrupts with 5 vectors igb1: Ethernet address: d0:50:99:c2:b5:24 pcib9: <ACPI PCI-PCI bridge> mem 0xdf700000-0xdf71ffff at device 4.0 on pci0 pci9: <ACPI PCI bus> on pcib9 ahci1: <Marvell 88SE9230 AHCI SATA controller> port 0xd050-0xd057,0xd040-0xd043,0xd030-0xd037,0xd020-0xd023,0xd000-0xd01f mem 0xdf610000-0xdf6107ff irq 23 at device 0.0 on pci9 ahci1: AHCI v1.20 with 8 6Gbps ports, Port Multiplier not supported ahci1: quirks=0x1000900<NOBSYRES,ALTSIG,IOMMU_BUSWIDE> ahcich2: <AHCI channel> at channel 0 on ahci1 ahcich3: <AHCI channel> at channel 1 on ahci1 ahcich4: <AHCI channel> at channel 2 on ahci1 ahcich5: <AHCI channel> at channel 3 on ahci1 ahcich6: <AHCI channel> at channel 4 on ahci1 ahcich7: <AHCI channel> at channel 5 on ahci1 ahcich8: <AHCI channel> at channel 6 on ahci1 ahcich9: <AHCI channel> at channel 7 on ahci1 pci0: <base peripheral, IOMMU> at device 15.0 (no driver attached) ehci0: <Intel Avoton USB 2.0 controller> mem 0xdf763000-0xdf7633ff irq 23 at device 22.0 on pci0 usbus0: EHCI version 1.0 usbus0 on ehci0 usbus0: 480Mbps High Speed USB v2.0 ahci2: <Intel Avoton AHCI SATA controller> port 0xe0d0-0xe0d7,0xe0c0-0xe0c3,0xe0b0-0xe0b7,0xe0a0-0xe0a3,0xe040-0xe05f mem 0xdf762000-0xdf7627ff irq 19 at device 23.0 on pci0 ahci2: AHCI v1.30 with 4 3Gbps ports, Port Multiplier not supported ahcich10: <AHCI channel> at channel 0 on ahci2 ahcich11: <AHCI channel> at channel 1 on ahci2 ahcich12: <AHCI channel> at channel 2 on ahci2 ahcich13: <AHCI channel> at channel 3 on ahci2 ahci3: <Intel Avoton AHCI SATA controller> port 0xe090-0xe097,0xe080-0xe083,0xe070-0xe077,0xe060-0xe063,0xe020-0xe03f mem 0xdf761000-0xdf7617ff irq 19 at device 24.0 on pci0 ahci3: AHCI v1.30 with 2 6Gbps ports, Port Multiplier not supported ahcich14: <AHCI channel> at channel 0 on ahci3 ahcich15: <AHCI channel> at channel 1 on ahci3 isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 apei0: <ACPI Platform Error Interface> on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) ichwd0: <Intel Avoton/Rangeley SoC watchdog timer> on isa0 ichwd0: ICH WDT present but disabled in BIOS or hardware device_attach: ichwd0 attach returned 6 ichwd0: <Intel Avoton/Rangeley SoC watchdog timer> at port 0x430-0x437,0x460-0x47f on isa0 ichwd0: ICH WDT present but disabled in BIOS or hardware device_attach: ichwd0 attach returned 6 orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xcb000-0xcbfff pnpid ORM0000 on isa0 superio0: <Nuvoton NCT6776> at port 0x2e-0x2f on isa0 wbwd0: <Nuvoton NCT6776 (0xc3/0x33) Watchdog Timer> at WDT ldn 0x08 on superio0 atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] coretemp0: <CPU On-Die Thermal Sensors> on cpu0 est0: <Enhanced SpeedStep Frequency Control> on cpu0 Timecounter "TSC-low" frequency 1199999726 Hz quality 1000 Timecounters tick every 1.000 msec ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled Trying to mount root from zfs:freenas-boot/ROOT/13.0-U1.1 []... ugen0.1: <Intel EHCI root HUB> at usbus0 Mounting from zfs:freenas-boot/ROOT/13.0-U1.1 failed with error 2: unknown file system. uhub0 on usbus0 uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0 ada0 at ahcich3 bus 0 scbus3 target 0 lun 0 ada0: <16GB SATA Flash Drive SFDK002A> ACS-2 ATA SATA 3.x device ada0: Serial Number E0117209800000000086 ada0: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 512bytes) ada0: Command Queueing enabled ada0: 15272MB (31277232 512 byte sectors) ada1 at ahcich10 bus 0 scbus10 target 0 lun 0 ada1: <WDC WD40EFRX-68N32N0 82.00A82> ACS-3 ATA SATA 3.x device ada1: Serial Number WD-WCC7K3SESA43 ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 3815447MB (7814037168 512 byte sectors) ada1: quirks=0x1<4K> ada2 at ahcich11 bus 0 scbus11 target 0 lun 0 ada2: <WDC WD40EFRX-68N32N0 82.00A82> ACS-3 ATA SATA 3.x device ada2: Serial Number WD-WCC7K1RCRT25 ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada2: Command Queueing enabled ada2: 3815447MB (7814037168 512 byte sectors) ada2: quirks=0x1<4K> ada3 at ahcich12 bus 0 scbus12 target 0 lun 0 ada3: <WDC WD40EFRX-68N32N0 82.00A82> ACS-3 ATA SATA 3.x device ada3: Serial Number WD-WCC7K6TSYF4Y ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada3: Command Queueing enabled ada3: 3815447MB (7814037168 512 byte sectors) ada3: quirks=0x1<4K> ada4 at ahcich13 bus 0 scbus13 target 0 lun 0 ada4: <WDC WD40EFRX-68N32N0 82.00A82> ACS-3 ATA SATA 3.x device ada4: Serial Number WD-WCC7K6AKH5CJ ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada4: Command Queueing enabled ada4: 3815447MB (7814037168 512 byte sectors) ada4: quirks=0x1<4K> pass1 at ahcich9 bus 0 scbus9 target 0 lun 0 pass1: <Marvell Console 1.01> Removable Processor SCSI device pass1: Serial Number HKDP221516WL pass1: 150.000MB/s transfers (SATA 1.x, UDMA4, ATAPI 12bytes, PIO 8192bytes) Root mount waiting for: usbus0 Root mount waiting for: usbus0 uhub0: 8 ports with 8 removable, self powered Root mount waiting for: usbus0 ugen0.2: <vendor 0x8087 product 0x07db> at usbus0 uhub1 on uhub0 uhub1: <vendor 0x8087 product 0x07db, class 9/0, rev 2.00/0.02, addr 2> on usbus0 uhub1: 4 ports with 4 removable, self powered Root mount waiting for: usbus0 ugen0.3: <vendor 0x05e3 USB2.0 Hub> at usbus0 uhub2 on uhub1 uhub2: <vendor 0x05e3 USB2.0 Hub, class 9/0, rev 2.00/32.98, addr 3> on usbus0 uhub2: MTT enabled uhub2: 4 ports with 4 removable, self powered Root mount waiting for: usbus0 ugen0.4: <vendor 0x04d9 RPI Wired Keyboard 4> at usbus0 ukbd0 on uhub2 ukbd0: <vendor 0x04d9 RPI Wired Keyboard 4, class 0/0, rev 2.00/1.10, addr 4> on usbus0 kbd2 at ukbd0 uhid0 on uhub2 uhid0: <vendor 0x04d9 RPI Wired Keyboard 4, class 0/0, rev 2.00/1.10, addr 4> on usbus0 ugen0.5: <American Megatrends Inc. Virtual Hub> at usbus0 uhub3 on uhub1 uhub3: <7-port Hub> on usbus0 Root mount waiting for: usbus0 uhub3: 5 ports with 5 removable, self powered ugen0.6: <American Megatrends Inc. Virtual Keyboard and Mouse> at usbus0 ukbd1 on uhub3 ukbd1: <Keyboard Interface> on usbus0 kbd3 at ukbd1 ums0 on uhub3 ums0: <Mouse Interface> on usbus0 ums0: 3 buttons and [Z] coordinates ID=0 Loader variables: vfs.root.mountfrom=zfs:freenas-boot/ROOT/13.0-U1.1 Manual root filesystem specification: <fstype>:<device> [options] Mount <device> using filesystem <fstype> and with the specified (optional) option list. eg. ufs:/dev/da0s1a zfs:zroot/ROOT/default cd9660:/dev/cd0 ro (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /) ? List valid disk boot devices . Yield 1 second (for background tasks) <empty line> Abort manual input mountroot> ? List of GEOM managed disk devices: diskid/DISK-WD-WCC7K6AKH5CJp2 diskid/DISK-WD-WCC7K6AKH5CJp1 gptid/f56a0216-71e1-11e7-bcde-d05099c2b523 gptid/f5561875-71e1-11e7-bcde-d05099c2b523 diskid/DISK-WD-WCC7K6TSYF4Yp2 diskid/DISK-WD-WCC7K6TSYF4Yp1 gptid/f44af09c-71e1-11e7-bcde-d05099c2b523 gptid/f4376e2c-71e1-11e7-bcde-d05099c2b523 diskid/DISK-WD-WCC7K1RCRT25p2 diskid/DISK-WD-WCC7K1RCRT25p1 gptid/f32ee1d8-71e1-11e7-bcde-d05099c2b523 gptid/f31c3673-71e1-11e7-bcde-d05099c2b523 diskid/DISK-WD-WCC7K6AKH5CJ ada4p2 ada4p1 diskid/DISK-WD-WCC7K6TSYF4Y ada3p2 ada3p1 diskid/DISK-WD-WCC7K1RCRT25 ada2p2 ada2p1 diskid/DISK-WD-WCC7K3SESA43p2 diskid/DISK-WD-WCC7K3SESA43p1 gptid/f215e5ce-71e1-11e7-bcde-d05099c2b523 gptid/f20454d6-71e1-11e7-bcde-d05099c2b523 ada4 ada3 ada2 diskid/DISK-WD-WCC7K3SESA43 ada1p2 ada1p1 diskid/DISK-E0117209800000000086p2 diskid/DISK-E0117209800000000086p1 gptid/e9b0c5ff-68a0-11e7-b857-d05099c2b523 gptid/e9ae3b5e-68a0-11e7-b857-d05099c2b523 ada1 diskid/DISK-E0117209800000000086 ada0p2 ada0p1 ada0 mountroot>
There is no usb devices plugged in. Until just now I'd never opened the case even. There's no usb boot drive inside. It's a small pcb plugged into an sata plug on the mother board, which appears to be a 16G ssd. The hardware, other than the main drives, is all right from iXsystems new and until just now I'd never even broken the warranty sticker. Inside everything looks fine, not even dusty, fan works etc. It's been powered from a apc smart-ups 2u rackmount the whole time and there was no storm or power event recently. I believe the problem must be the result of deleting the old boot environments. Though I don't know why since I'm definitely running 13.x and I definitely only removed some 12.x and 11.x environments. The main drives are 4 x 4T wd red 3.5" NASware (well you can see those in the console log so nevermind...)
It looks like, the kernel says it's 13.1, but the failing mountroot says it's trying to mount something with 13.0 in the name. No idea how that came about.
What can I do from here?
I actually don't mind reinstalling the host and all jails as long as I can preserve the main zfs pool. I don't have a recent config backup, but I do have some old config backups, from before the 13.0 update. The few jails I have don't have anything irreplaceable in the jails themselves. I can lose the host and all jails but do not want to lose the main zfs data pool.