My Prod server started acting poorly yesterday (newly built 6 months ago), the GUI would hang for minutes at a time, NFS mounts would stall, command line would hang for 60 seconds or more, these events were continual repeating every 2-5 minutes or so. Leaving the system pretty much unusable. This was running 12.0-u3. (system details should be in signature). No hardware changes have been made in months, last configuration change was weeks ago (added new replication tasks to backup NAS).
middlewared.log
This started running version 12.0-u3, hoping it was just a fixed bug, eventually I was able to upgrade to 12.0-u4-1 (which took 2 hours to complete). No change.
Reverted back to 12.0-u3 - no change.
Disabled SMART Service - no change
Removed SMART tasks - no change
Removed the LOG drive from the pool, no change, even physically removed it, no change. (SMART was not collecting data on this drive because of a data format error)
dmsg output;
middlewared.log
Code:
[2021/06/02 08:11:27] (WARNING) middlewared._loop_monitor_thread():1418 - Task seems blocked: File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/disk_/smartctl.py", line 69, in smartctl cp = await smartctl(smartctl_args + args, check=False, stderr=subprocess.STDOUT, File "/usr/local/lib/python3.9/site-packages/middlewared/common/smart/smartctl.py", line 104, in smartctl return await run(["smartctl"] + args, **kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/utils/__init__.py", line 53, in run proc = await asyncio.create_subprocess_exec(*args, **kwargs) File "/usr/local/lib/python3.9/asyncio/subprocess.py", line 236, in create_subprocess_exec transport, protocol = await loop.subprocess_exec( File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1661, in subprocess_exec transport = await self._make_subprocess_transport( File "/usr/local/lib/python3.9/asyncio/unix_events.py", line 197, in _make_subprocess_transport transp = _UnixSubprocessTransport(self, protocol, args, shell, File "/usr/local/lib/python3.9/asyncio/base_subprocess.py", line 36, in __init__ self._start(args=args, shell=shell, stdin=stdin, stdout=stdout, File "/usr/local/lib/python3.9/asyncio/unix_events.py", line 789, in _start self._proc = subprocess.Popen( File "/usr/local/lib/python3.9/subprocess.py", line 951, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/local/lib/python3.9/subprocess.py", line 1777, in _execute_child part = os.read(errpipe_read, 50000) [2021/06/02 08:16:22] (WARNING) middlewared._loop_monitor_thread():1418 - Task seems blocked: File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/disk_/smartctl.py", line 69, in smartctl cp = await smartctl(smartctl_args + args, check=False, stderr=subprocess.STDOUT, File "/usr/local/lib/python3.9/site-packages/middlewared/common/smart/smartctl.py", line 104, in smartctl return await run(["smartctl"] + args, **kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/utils/__init__.py", line 53, in run proc = await asyncio.create_subprocess_exec(*args, **kwargs) File "/usr/local/lib/python3.9/asyncio/subprocess.py", line 236, in create_subprocess_exec transport, protocol = await loop.subprocess_exec( File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1661, in subprocess_exec transport = await self._make_subprocess_transport( File "/usr/local/lib/python3.9/asyncio/unix_events.py", line 197, in _make_subprocess_transport transp = _UnixSubprocessTransport(self, protocol, args, shell, File "/usr/local/lib/python3.9/asyncio/base_subprocess.py", line 36, in __init__ self._start(args=args, shell=shell, stdin=stdin, stdout=stdout, File "/usr/local/lib/python3.9/asyncio/unix_events.py", line 789, in _start self._proc = subprocess.Popen( File "/usr/local/lib/python3.9/subprocess.py", line 951, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/local/lib/python3.9/subprocess.py", line 1777, in _execute_child part = os.read(errpipe_read, 50000) [2021/06/02 08:17:09] (WARNING) middlewared._loop_monitor_thread():1418 - Task seems blocked: File "/usr/local/lib/python3.9/site-packages/middlewared/alert/source/ipmi_sel.py", line 151, in check return await self._produce_alerts_for_ipmitool_output(await ipmitool(["-c", "sel", "elist"])) File "/usr/local/lib/python3.9/site-packages/middlewared/alert/source/ipmi_sel.py", line 28, in __call__ result = await run(["ipmitool"] + args, check=False, encoding="utf8", errors="ignore") File "/usr/local/lib/python3.9/site-packages/middlewared/utils/__init__.py", line 53, in run proc = await asyncio.create_subprocess_exec(*args, **kwargs) File "/usr/local/lib/python3.9/asyncio/subprocess.py", line 236, in create_subprocess_exec transport, protocol = await loop.subprocess_exec( File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1661, in subprocess_exec transport = await self._make_subprocess_transport( File "/usr/local/lib/python3.9/asyncio/unix_events.py", line 197, in _make_subprocess_transport transp = _UnixSubprocessTransport(self, protocol, args, shell, File "/usr/local/lib/python3.9/asyncio/base_subprocess.py", line 36, in __init__ self._start(args=args, shell=shell, stdin=stdin, stdout=stdout, File "/usr/local/lib/python3.9/asyncio/unix_events.py", line 789, in _start self._proc = subprocess.Popen( File "/usr/local/lib/python3.9/subprocess.py", line 951, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/local/lib/python3.9/subprocess.py", line 1777, in _execute_child part = os.read(errpipe_read, 50000) ^C[2021/06/02 08:21:25] (WARNING) middlewared._loop_monitor_thread():1418 - Task seems blocked: File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/disk_/smartctl.py", line 69, in smartctl cp = await smartctl(smartctl_args + args, check=False, stderr=subprocess.STDOUT, File "/usr/local/lib/python3.9/site-packages/middlewared/common/smart/smartctl.py", line 104, in smartctl return await run(["smartctl"] + args, **kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/utils/__init__.py", line 53, in run proc = await asyncio.create_subprocess_exec(*args, **kwargs) File "/usr/local/lib/python3.9/asyncio/subprocess.py", line 236, in create_subprocess_exec transport, protocol = await loop.subprocess_exec( File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1661, in subprocess_exec transport = await self._make_subprocess_transport( File "/usr/local/lib/python3.9/asyncio/unix_events.py", line 197, in _make_subprocess_transport transp = _UnixSubprocessTransport(self, protocol, args, shell, File "/usr/local/lib/python3.9/asyncio/base_subprocess.py", line 36, in __init__ self._start(args=args, shell=shell, stdin=stdin, stdout=stdout, File "/usr/local/lib/python3.9/asyncio/unix_events.py", line 789, in _start self._proc = subprocess.Popen( File "/usr/local/lib/python3.9/subprocess.py", line 951, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/local/lib/python3.9/subprocess.py", line 1777, in _execute_child part = os.read(errpipe_read, 50000) [2021/06/02 08:22:09] (WARNING) middlewared._loop_monitor_thread():1418 - Task seems blocked: File "/usr/local/lib/python3.9/site-packages/middlewared/alert/source/ipmi_sel.py", line 151, in check return await self._produce_alerts_for_ipmitool_output(await ipmitool(["-c", "sel", "elist"])) File "/usr/local/lib/python3.9/site-packages/middlewared/alert/source/ipmi_sel.py", line 28, in __call__ result = await run(["ipmitool"] + args, check=False, encoding="utf8", errors="ignore") File "/usr/local/lib/python3.9/site-packages/middlewared/utils/__init__.py", line 53, in run proc = await asyncio.create_subprocess_exec(*args, **kwargs) File "/usr/local/lib/python3.9/asyncio/subprocess.py", line 236, in create_subprocess_exec transport, protocol = await loop.subprocess_exec( File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1661, in subprocess_exec transport = await self._make_subprocess_transport( File "/usr/local/lib/python3.9/asyncio/unix_events.py", line 197, in _make_subprocess_transport transp = _UnixSubprocessTransport(self, protocol, args, shell, File "/usr/local/lib/python3.9/asyncio/base_subprocess.py", line 36, in __init__ self._start(args=args, shell=shell, stdin=stdin, stdout=stdout, File "/usr/local/lib/python3.9/asyncio/unix_events.py", line 789, in _start self._proc = subprocess.Popen( File "/usr/local/lib/python3.9/subprocess.py", line 951, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/local/lib/python3.9/subprocess.py", line 1777, in _execute_child part = os.read(errpipe_read, 50000) [2021/06/02 08:22:18] (WARNING) middlewared._loop_monitor_thread():1418 - Task seems blocked: File "/usr/local/lib/python3.9/site-packages/middlewared/alert/source/ipmi_sel.py", line 214, in check return self._produce_alert_for_ipmitool_output(await ipmitool(["sel", "info"])) File "/usr/local/lib/python3.9/site-packages/middlewared/alert/source/ipmi_sel.py", line 28, in __call__ result = await run(["ipmitool"] + args, check=False, encoding="utf8", errors="ignore") File "/usr/local/lib/python3.9/site-packages/middlewared/utils/__init__.py", line 53, in run proc = await asyncio.create_subprocess_exec(*args, **kwargs) File "/usr/local/lib/python3.9/asyncio/subprocess.py", line 236, in create_subprocess_exec transport, protocol = await loop.subprocess_exec( File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1661, in subprocess_exec transport = await self._make_subprocess_transport( File "/usr/local/lib/python3.9/asyncio/unix_events.py", line 197, in _make_subprocess_transport transp = _UnixSubprocessTransport(self, protocol, args, shell, File "/usr/local/lib/python3.9/asyncio/base_subprocess.py", line 36, in __init__ self._start(args=args, shell=shell, stdin=stdin, stdout=stdout, File "/usr/local/lib/python3.9/asyncio/unix_events.py", line 789, in _start self._proc = subprocess.Popen( File "/usr/local/lib/python3.9/subprocess.py", line 951, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/local/lib/python3.9/subprocess.py", line 1777, in _execute_child part = os.read(errpipe_read, 50000)
This started running version 12.0-u3, hoping it was just a fixed bug, eventually I was able to upgrade to 12.0-u4-1 (which took 2 hours to complete). No change.
Reverted back to 12.0-u3 - no change.
Disabled SMART Service - no change
Removed SMART tasks - no change
Removed the LOG drive from the pool, no change, even physically removed it, no change. (SMART was not collecting data on this drive because of a data format error)
dmsg output;
Code:
---<<BOOT>>--- Copyright (c) 1992-2020 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.2-RELEASE-p6 df578562304(HEAD) TRUENAS amd64 FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2) VT(vga): text 80x25 CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (2399.37-MHz K8-class CPU) Origin="GenuineIntel" Id=0x206c2 Family=0x6 Model=0x2c Stepping=2 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x29ee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x1<LAHF> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 51539607552 (49152 MB) avail memory = 49922826240 (47610 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <HP ProLiant> FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 hardware threads random: unblocking device. Firmware Warning (ACPI): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20200430/tbfadt-850) Firmware Warning (ACPI): Invalid length for FADT/Pm2ControlBlock: 32, using default 8 (20200430/tbfadt-850) ioapic1 <Version 2.0> irqs 24-47 on motherboard ioapic0 <Version 2.0> irqs 0-23 on motherboard Launching APs: 1 14 4 8 5 3 15 6 7 10 2 9 12 13 11 Timecounter "TSC-low" frequency 1199685090 Hz quality 1000 random: entropy device external interface kbd1 at kbdmux0 mlx5en: Mellanox Ethernet driver 3.5.2 (September 2019) nexus0 vtvga0: <VT VGA driver> on motherboard aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard padlock0: No ACE support. cryptosoft0: <software crypto> on motherboard acpi0: <HP ProLiant> on motherboard acpi0: Power Button (fixed) acpi0: _OSC failed: AE_BUFFER_OVERFLOW cpu0: <ACPI CPU> on acpi0 attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 apei0: <ACPI Platform Error Interface> on acpi0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0 pcib0: <ACPI Host-PCI bridge> on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0 pci1: <ACPI PCI bus> on pcib1 ciss0: <HP Smart Array P410i> port 0x4000-0x40ff mem 0xfba00000-0xfbbfffff,0xfb9f0000-0xfb9f0fff irq 28 at device 0.0 on pci1 ciss0: PERFORMANT Transport pcib2: <ACPI PCI-PCI bridge> at device 2.0 on pci0 pci2: <ACPI PCI bus> on pcib2 pcib3: <ACPI PCI-PCI bridge> at device 3.0 on pci0 pci3: <ACPI PCI bus> on pcib3 mps0: <Avago Technologies (LSI) SSS6200> port 0x5000-0x50ff mem 0xfbdf0000-0xfbdfffff,0xfbd80000-0xfbdbffff irq 24 at device 0.0 on pci3 mps0: Firmware: 113.00.00.00, Driver: 21.02.00.00-fbsd mps0: IOCCapabilities: 984c<ScsiTaskFull,DiagTrace,EEDP,TransRetry,IR,MSIXIndex> pcib4: <ACPI PCI-PCI bridge> at device 4.0 on pci0 pci4: <ACPI PCI bus> on pcib4 pcib5: <ACPI PCI-PCI bridge> at device 5.0 on pci0 pci5: <ACPI PCI bus> on pcib5 pcib6: <ACPI PCI-PCI bridge> at device 6.0 on pci0 pci6: <ACPI PCI bus> on pcib6 pcib7: <ACPI PCI-PCI bridge> at device 7.0 on pci0 pci7: <ACPI PCI bus> on pcib7 pcib8: <ACPI PCI-PCI bridge> at device 8.0 on pci0 pci8: <ACPI PCI bus> on pcib8 pcib9: <ACPI PCI-PCI bridge> at device 9.0 on pci0 pci9: <ACPI PCI bus> on pcib9 mps1: <Avago Technologies (LSI) SAS2008> port 0x6000-0x60ff mem 0xfbef0000-0xfbef3fff,0xfbe80000-0xfbebffff irq 32 at device 0.0 on pci9 mps1: Firmware: 18.00.00.00, Driver: 21.02.00.00-fbsd mps1: IOCCapabilities: 185c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR> pcib10: <ACPI PCI-PCI bridge> at device 10.0 on pci0 pci10: <ACPI PCI bus> on pcib10 em0: <Intel(R) PRO/1000 Network Connection> port 0x7000-0x701f mem 0xfbfe0000-0xfbffffff,0xfbfc0000-0xfbfdffff irq 33 at device 0.0 on pci10 em0: Using 1024 TX descriptors and 1024 RX descriptors em0: Using an MSI interrupt em0: Ethernet address: 00:26:55:e8:44:8c em1: <Intel(R) PRO/1000 Network Connection> port 0x7020-0x703f mem 0xfbfa0000-0xfbfbffff,0xfbf80000-0xfbf9ffff irq 31 at device 0.1 on pci10 em1: Using 1024 TX descriptors and 1024 RX descriptors em1: Using an MSI interrupt em1: Ethernet address: 00:26:55:e8:44:8d pci0: <base peripheral, interrupt controller> at device 20.0 (no driver attached) pci0: <base peripheral, interrupt controller> at device 20.1 (no driver attached) pci0: <base peripheral, interrupt controller> at device 20.2 (no driver attached) pcib11: <ACPI PCI-PCI bridge> at device 28.0 on pci0 pci11: <ACPI PCI bus> on pcib11 pcib12: <ACPI PCI-PCI bridge> at device 0.0 on pci11 pci12: <ACPI PCI bus> on pcib12 bge0: <HP NC326i Dual Port PCI-E Gigabit Adapter, ASIC rev. 0x009003> mem 0xfb8f0000-0xfb8fffff,0xfb8e0000-0xfb8effff irq 16 at device 4.0 on pci12 bge0: CHIP ID 0x00009003; ASIC REV 0x09; CHIP REV 0x90; PCI-X 133 MHz miibus0: <MII bus> on bge0 brgphy0: <BCM5714 1000BASE-T media interface> PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bge0: Ethernet address: 9c:8e:99:19:b9:28 bge1: <HP NC326i Dual Port PCI-E Gigabit Adapter, ASIC rev. 0x009003> mem 0xfb8d0000-0xfb8dffff,0xfb8c0000-0xfb8cffff irq 17 at device 4.1 on pci12 bge1: CHIP ID 0x00009003; ASIC REV 0x09; CHIP REV 0x90; PCI-X 133 MHz miibus1: <MII bus> on bge1 brgphy1: <BCM5714 1000BASE-T media interface> PHY 1 on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bge1: Ethernet address: 9c:8e:99:19:b9:29 uhci0: <Intel 82801JI (ICH10) USB controller USB-A> port 0x1000-0x101f irq 20 at device 29.0 on pci0 usbus0 on uhci0 usbus0: 12Mbps Full Speed USB v1.0 uhci1: <Intel 82801JI (ICH10) USB controller USB-B> port 0x1020-0x103f irq 23 at device 29.1 on pci0 usbus1 on uhci1 usbus1: 12Mbps Full Speed USB v1.0 uhci2: <Intel 82801JI (ICH10) USB controller USB-C> port 0x1040-0x105f irq 22 at device 29.2 on pci0 usbus2 on uhci2 usbus2: 12Mbps Full Speed USB v1.0 uhci3: <Intel 82801JI (ICH10) USB controller USB-F> port 0x1060-0x107f irq 23 at device 29.3 on pci0 usbus3 on uhci3 usbus3: 12Mbps Full Speed USB v1.0 ehci0: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> mem 0xfb5f0000-0xfb5f03ff irq 20 at device 29.7 on pci0 usbus4: EHCI version 1.0 usbus4 on ehci0 usbus4: 480Mbps High Speed USB v2.0 pcib13: <ACPI PCI-PCI bridge> at device 30.0 on pci0 pci13: <ACPI PCI bus> on pcib13 vgapci0: <VGA-compatible display> port 0x3000-0x30ff mem 0xf0000000-0xf7ffffff,0xfb7f0000-0xfb7fffff irq 23 at device 3.0 on pci13 vgapci0: Boot video device uhci4: <HP iLO Standard Virtual USB controller> port 0x3800-0x381f irq 22 at device 4.4 on pci13 usbus5 on uhci4 usbus5: 12Mbps Full Speed USB v1.0 ipmi0: <IPMI System Interface> mem 0xfb6f0000-0xfb6f00ff irq 21 at device 4.6 on pci13 ipmi0: using KSC interface isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 atapci0: <Intel ICH10 SATA300 controller> port 0x10c0-0x10c7,0x10c8-0x10cb,0x10d0-0x10d7,0x10d8-0x10db,0x10e0-0x10ef,0x10f0-0x10ff irq 17 at device 31.5 on pci0 ata2: <ATA channel> at channel 0 on atapci0 ata3: <ATA channel> at channel 1 on atapci0 acpi_tz0: <Thermal Zone> on acpi0 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] acpi_syscontainer0: <System Container> port 0x2e-0x2f,0x620-0x65f,0x680-0x69f,0x600-0x61f,0x660-0x67f,0x300-0x31f on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 ichwd0: <Intel ICH10 watchdog timer> on isa0 ichwd0: ICH WDT present but disabled in BIOS or hardware device_attach: ichwd0 attach returned 6 ichwd0: <Intel ICH10 watchdog timer> at port 0x930-0x937,0x960-0x97f on isa0 ichwd0: ICH WDT present but disabled in BIOS or hardware device_attach: ichwd0 attach returned 6 orm0: <ISA Option ROMs> at iomem 0xc0000-0xcafff,0xcf000-0xd2fff pnpid ORM0000 on isa0 uart1: <Non-standard ns8250 class UART with FIFOs> at port 0x2f8 irq 3 on isa0 coretemp0: <CPU On-Die Thermal Sensors> on cpu0 est0: <Enhanced SpeedStep Frequency Control> on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 13 device_attach: est0 attach returned 6 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 13 device_attach: est1 attach returned 6 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 13 device_attach: est2 attach returned 6 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 13 device_attach: est3 attach returned 6 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 13 device_attach: est4 attach returned 6 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 13 device_attach: est5 attach returned 6 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 13 device_attach: est6 attach returned 6 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 13 device_attach: est7 attach returned 6 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 13 device_attach: est8 attach returned 6 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 13 device_attach: est9 attach returned 6 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 13 device_attach: est10 attach returned 6 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 13 device_attach: est11 attach returned 6 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 13 device_attach: est12 attach returned 6 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 13 device_attach: est13 attach returned 6 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 13 device_attach: est14 attach returned 6 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 13 device_attach: est15 attach returned 6 Timecounters tick every 1.000 msec ZFS filesystem version: 5 ZFS storage pool version: features support (5000) ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled ugen1.1: <Intel UHCI root HUB> at usbus1 ugen5.1: <HP UHCI root HUB> at usbus5 ugen0.1: <Intel UHCI root HUB> at usbus0 ugen2.1: <Intel UHCI root HUB> at usbus2 ugen3.1: <Intel UHCI root HUB> at usbus3 ugen4.1: <Intel EHCI root HUB> at usbus4 uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1 uhub1: <HP UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5 uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2 uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3 uhub4: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus4 uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 ipmi0: IPMI device rev. 1, firmware rev. 2.33, version 2.0, device support mask 0xf ipmi0: Number of channels 0 ipmi0: Attached watchdog Trying to mount root from zfs:boot-pool/ROOT/12.0-U4-1 []... Root mount waiting for: CAM usbus0 usbus1 usbus2 usbus3 usbus4 usbus5 uhub0: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered ugen5.2: <HP Virtual Keyboard> at usbus5 ukbd0 on uhub1 ukbd0: <Virtual Keyboard> on usbus5 kbd2 at ukbd0 ums0 on uhub1 Root mount waiting for: CAM usbus4 usbus5 ums0: <Virtual Mouse> on usbus5 ums0: 3 buttons and [XY] coordinates ID=0 Root mount waiting for: CAM usbus4 Root mount waiting for: CAM usbus4 uhub4: 8 ports with 8 removable, self powered Root mount waiting for: CAM ugen3.2: <Avocent Avocent DSRIQ-USB> at usbus3 ukbd1 on uhub3 ukbd1: <EP1 Interrupt> on usbus3 kbd3 at ukbd1 ums1 on uhub3 ums1: <EP2 Interrupt> on usbus3 ums1: 5 buttons and [XYZ] coordinates ID=1 Root mount waiting for: CAM Root mount waiting for: CAM ada0 at ata3 bus 0 scbus5 target 0 lun 0 ada0: <Crucial CT240M500SSD1 MU03> ACS-2 ATA SATA 3.x device ada0: Serial Number 14100C06B003 ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes) ada0: 228936MB (468862128 512 byte sectors) ada0: quirks=0x2<NCQ_TRIM_BROKEN> da0 at mps0 bus 0 scbus2 target 0 lun 0 da0: <LSI NWD-BLP4-800 0002> Fixed Direct Access SPC-4 SCSI device da0: Serial Number 2622235072208845886 da0: 150.000MB/s transfers da0: Command Queueing enabled da0: 762939MB (1562499584 512 byte sectors) da4 at mps1 bus 0 scbus3 target 9 lun 0 da4: <ATA HGST HUH728060AL T7J0> Fixed Direct Access SPC-4 SCSI device da4: Serial Number 2RG6J0VX da4: 600.000MB/s transfers da4: Command Queueing enabled da4: 5723166MB (11721045168 512 byte sectors) da1 at mps1 bus 0 scbus3 target 6 lun 0 da1: <ATA HUS726060ALE610 TDR2> Fixed Direct Access SPC-4 SCSI device da1: Serial Number K8K98N4N da1: 600.000MB/s transfers da1: Command Queueing enabled da1: 5723166MB (11721045168 512 byte sectors) da2 at mps1 bus 0 scbus3 target 7 lun 0 da2: <ATA HGST HUS726060AL TD05> Fixed Direct Access SPC-4 SCSI device da2: Serial Number K8J1V6HD da2: 600.000MB/s transfers da2: Command Queueing enabled da2: 5723166MB (11721045168 512 byte sectors) da3 at mps1 bus 0 scbus3 target 8 lun 0 da3: <ATA HGST HUS726060AL TD05> Fixed Direct Access SPC-4 SCSI device da3: Serial Number K8J251UD da3: 600.000MB/s transfers da3: Command Queueing enabled da3: 5723166MB (11721045168 512 byte sectors) da5 at mps1 bus 0 scbus3 target 11 lun 0 da5: <ATA HGST HUS726060AL TD05> Fixed Direct Access SPC-4 SCSI device da5: Serial Number K8GKN14D da5: 600.000MB/s transfers da5: Command Queueing enabled da5: 5723166MB (11721045168 512 byte sectors) lo0: link state changed to UP em0: link state changed to UP em1: link state changed to UP GEOM_MIRROR: Device mirror/swap0 launched (2/2). GEOM_MIRROR: Device mirror/swap1 launched (2/2). GEOM_ELI: Device mirror/swap0.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: hardware GEOM_ELI: Device mirror/swap1.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: hardware hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/4/48/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/48/0x67<INT,USR,SYS,REA,WRI> UCP/8/48/0x3f8<EDG,THR,REA,WRI,INV,QUA,PRC> UCF/1/48/0x60<REA,WRI> em0: link state changed to DOWN em1: link state changed to DOWN bge0: link state changed to DOWN em0: link state changed to UP em1: link state changed to UP lagg0: link state changed to UP CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (2399.37-MHz K8-class CPU) Origin="GenuineIntel" Id=0x206c2 Family=0x6 Model=0x2c Stepping=2 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x29ee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x1<LAHF> Structured Extended Features3=0x9c000000<IBPB,STIBP,L1DFL,SSBD> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics nfsd: can't register svc name Security policy loaded: MAC/ntpd (mac_ntpd) bridge0: Ethernet address: 02:2d:4c:97:d4:00 epair0a: Ethernet address: 02:7d:a5:36:20:0a epair0b: Ethernet address: 02:7d:a5:36:20:0b epair0a: link state changed to UP epair0b: link state changed to UP epair0a: changing name to 'vnet0.1' em0: promiscuous mode enabled em1: promiscuous mode enabled lagg0: promiscuous mode enabled em0: link state changed to DOWN em1: link state changed to DOWN bridge0: link state changed to UP lagg0: link state changed to DOWN vnet0.1: promiscuous mode enabled em1: link state changed to UP lo0: link state changed to UP em0: link state changed to UP lagg0: link state changed to UP