marcevan
Patron
- Joined
- Dec 15, 2013
- Messages
- 432
I went thru the pain of trying the 11.2BETA3 upgrade but then went back to 11.1U6 after having no luck with Iocage plugins reaching the outside world.
Back on 11.1U6 and all IoCage jails are off so my warden jails (4 - emby, Plex, Transmission, Sickrage) are all up, autostart, and plugins are working.
Only oddness are Sickrage and Emby have the navtree error so they don't show in leftnav under plugins, and their plugins are in red under plugins/installed. They are up and working, however.
Two to three times a day now, the server shuts down and I have to physically power it on.
Some initial descoping of the root cause:
I suspect the Emby/Sickrage IP issue is due to the IOcage, 112B3 and back tasks but not sure if that's what's shutting the server down.
Grateful for any help. Not sure if my budget is capable of a new 2U chassis, power, mobo, RAM, CPU, etc build.
Back on 11.1U6 and all IoCage jails are off so my warden jails (4 - emby, Plex, Transmission, Sickrage) are all up, autostart, and plugins are working.
Only oddness are Sickrage and Emby have the navtree error so they don't show in leftnav under plugins, and their plugins are in red under plugins/installed. They are up and working, however.
Two to three times a day now, the server shuts down and I have to physically power it on.
Some initial descoping of the root cause:
- Power to the chassis - Wattbox registers entire rack system at 117V, 2.1A with little to no fluctuation (little when I use Control4 music in a room as it has passive amps that turn on/off with demand). All power is on a 20A circuit so we're not even close to a spike nor a limit.
- /data/crash has no entries since 2016.
- Looking in var/log* I see middleware is at it again (others have posted) Code:
2018/09/16 14:24:28] (WARNING) middlewared._loop_monitor_thread():1010 - Task seems blocked: File "/usr/local/lib/python3.6/site-p ackages/middlewared/plugins/device.py", line 68, in devd_loop await devd_listen(middleware)
- #3 happens about ever 2-10 seconds according to the file (same entries time stamped) - but they are only warnings.
- Monitor shows Emby and Sickrage plugins have messages like: arp [MAC] is using my IP address [of it's jail] on epair2b/epair4b (respectively).
- Monitor shows (on bootup): ACPI BIOS warning FADT field PM2controlblock has valid length but zero address
- Another warning about VIMAGE being experimental feature
- Zpool status is clean Code:
pool: Media state: ONLINE scan: scrub in progress since Sun Sep 16 00:00:03 2018 2.27T scanned at 1.14G/s, 991G issued at 499M/s, 5.45T total 0 repaired, 17.74% done, 0 days 02:37:00 to go config: NAME STATE READ WRITE CKSUM Media ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/41804705-6726-11e3-831d-6805ca1adbd8 ONLINE 0 0 0 gptid/41eefd8e-6726-11e3-831d-6805ca1adbd8 ONLINE 0 0 0 gptid/8f0a7046-7192-11e7-806a-6805ca1adbd8 ONLINE 0 0 0 gptid/58c022d3-6757-11e8-94e5-6805ca1adbd8 ONLINE 0 0 0 gptid/d9cee92c-9662-11e3-a1dc-6805ca1adbd8 ONLINE 0 0 0 gptid/43c4e34c-6726-11e3-831d-6805ca1adbd8 ONLINE 0 0 0 errors: No known data errors pool: freenas-boot state: ONLINE scan: scrub repaired 0 in 0 days 00:05:31 with 0 errors on Thu Sep 13 03:50:32 2018 config: NAME STATE READ WRITE CKSUM freenas-boot ONLINE 0 0 0 gptid/6e7337e9-7fad-11e4-a208-6805ca1adbd8 ONLINE 0 0 0 errors: No known data errors
- dmesg.today has the entirety of the bootup. Note it's 2018, so we're out of copyright by FreeBSD Project Code:
Copyright (c) 1992-2017 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.1-STABLE #0 r321665+9902d126c39(freenas/11.1-stable): Tue Aug 21 12:24:37 EDT 2018 root@nemesis.tn.ixsystems.com:/freenas-11-releng/freenas/_BE/objs/freenas-11-releng/freenas/_BE/os/sys/FreeNAS.amd64 amd64 FreeBSD clang version 5.0.0 (tags/RELEASE_500/final 312559) (based on LLVM 5.0.0svn) VT(vga): resolution 640x480 CPU: AMD A4-6300 APU with Radeon(tm) HD Graphics (3700.06-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x610f31 Family=0x15 Model=0x13 Stepping=1 Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> Features2=0x3e98320b<SSE3,PCLMULQDQ,MON,SSSE3,FMA,CX16,SSE4.1,SSE4.2,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C> AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM> AMD Features2=0x1ebbfff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,XOP,SKINIT,WDT,LWP,FMA4,TCE,NodeId,TBM,Topology,PCXC,PNXC> Structured Extended Features=0x8<BMI1> SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=65536 TSC: P-state invariant, performance statistics real memory = 17179869184 (16384 MB) avail memory = 15740456960 (15011 MB) Event timer "LAPIC" quality 100 ACPI APIC Table: <ALASKA A M I> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) random: unblocking device. WARNING: VIMAGE (virtualized network stack) is a highly experimental feature. ACPI BIOS Warning (bug): Optional FADT field Pm2ControlBlock has valid Length but zero Address: 0x0000000000000000/0x1 (20170728/tbfadt-796) ioapic0 <Version 2.1> irqs 0-23 on motherboard SMP: AP CPU #1 Launched! Timecounter "TSC-low" frequency 1850030317 Hz quality 1000 random: entropy device external interface kbd1 at kbdmux0 module_register_init: MOD_LOAD (vesa, 0xffffffff80fc84c0, 0) error 19 nexus0 vtvga0: <VT VGA driver> on motherboard cryptosoft0: <software crypto> on motherboard aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard padlock0: No ACE support. acpi0: <ALASKA A M I> on motherboard acpi0: Power Button (fixed) cpu0: <ACPI CPU> on acpi0 cpu1: <ACPI CPU> on acpi0 attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 550 Event timer "HPET1" frequency 14318180 Hz quality 450 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pci0: <base peripheral, IOMMU> at device 0.2 (no driver attached) vgapci0: <VGA-compatible display> port 0xf000-0xf0ff mem 0xc0000000-0xcfffffff,0xfeb00000-0xfeb3ffff irq 17 at device 1.0 on pci0 vgapci0: Boot video device pci0: <multimedia, HDA> at device 1.1 (no driver attached) pcib1: <ACPI PCI-PCI bridge> irq 17 at device 5.0 on pci0 pci1: <ACPI PCI bus> on pcib1 em0: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0xd000-0xd01f mem 0xfeac0000-0xfeadffff,0xfea00000-0xfea7ffff,0xfeae0000-0xfeae3fff irq 17 at device 0.0 on pci1 em0: Using MSIX interrupts with 3 vectors em0: Ethernet address: 68:05:ca:1a:db:d8 xhci0: <XHCI (generic) USB 3.0 controller> mem 0xfeb46000-0xfeb47fff irq 18 at device 16.0 on pci0 xhci0: 32 bytes context size, 64-bit DMA xhci0: Unable to map MSI-X table usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 xhci1: <XHCI (generic) USB 3.0 controller> mem 0xfeb44000-0xfeb45fff irq 17 at device 16.1 on pci0 xhci1: 32 bytes context size, 64-bit DMA xhci1: Unable to map MSI-X table usbus1 on xhci1 usbus1: 5.0Gbps Super Speed USB v3.0 ahci0: <AMD Hudson-2 AHCI SATA controller> port 0xf140-0xf147,0xf130-0xf133,0xf120-0xf127,0xf110-0xf113,0xf100-0xf10f mem 0xfeb4d000-0xfeb4d7ff irq 19 at device 17.0 on pci0 ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier supported ahcich0: <AHCI channel> at channel 0 on ahci0 ahcich1: <AHCI channel> at channel 1 on ahci0 ahcich2: <AHCI channel> at channel 2 on ahci0 ahcich3: <AHCI channel> at channel 3 on ahci0 ahcich4: <AHCI channel> at channel 4 on ahci0 ahcich5: <AHCI channel> at channel 5 on ahci0 ohci0: <AMD FCH USB Controller> mem 0xfeb4c000-0xfeb4cfff irq 18 at device 18.0 on pci0 usbus2 on ohci0 usbus2: 12Mbps Full Speed USB v1.0 ehci0: <AMD FCH USB 2.0 controller> mem 0xfeb4b000-0xfeb4b0ff irq 17 at device 18.2 on pci0 usbus3: EHCI version 1.0 usbus3 on ehci0 usbus3: 480Mbps High Speed USB v2.0 ohci1: <AMD FCH USB Controller> mem 0xfeb4a000-0xfeb4afff irq 18 at device 19.0 on pci0 usbus4 on ohci1 usbus4: 12Mbps Full Speed USB v1.0 ehci1: <AMD FCH USB 2.0 controller> mem 0xfeb49000-0xfeb490ff irq 17 at device 19.2 on pci0 usbus5: EHCI version 1.0 usbus5 on ehci1 usbus5: 480Mbps High Speed USB v2.0 isab0: <PCI-ISA bridge> at device 20.3 on pci0 isa0: <ISA bus> on isab0 pcib2: <ACPI PCI-PCI bridge> at device 20.4 on pci0 pci2: <ACPI PCI bus> on pcib2 ohci2: <OHCI (generic) USB controller> mem 0xfeb48000-0xfeb48fff irq 18 at device 20.5 on pci0 usbus6 on ohci2 usbus6: 12Mbps Full Speed USB v1.0 amdtemp0: <AMD CPU On-Die Thermal Sensors> on hostb4 acpi_button0: <Power Button> on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 amdsbwd0: <AMD SB8xx/SB9xx/Axx Watchdog Timer> at iomem 0xfec000f0-0xfec000f3,0xfec000f4-0xfec000f7 on isa0 atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] hwpstate0: <Cool`n'Quiet 2.0> on cpu0 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec freenas_sysctl: adding account. freenas_sysctl: adding directoryservice. freenas_sysctl: adding middlewared. freenas_sysctl: adding network. freenas_sysctl: adding services. ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled ugen5.1: <AMD EHCI root HUB> at usbus5 ugen6.1: <AMD OHCI root HUB> at usbus6 uhub0: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5 uhub1: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6 ugen3.1: <AMD EHCI root HUB> at usbus3 ugen4.1: <AMD OHCI root HUB> at usbus4 uhub2: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3 uhub3: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4 ugen2.1: <AMD OHCI root HUB> at usbus2 ugen1.1: <0x1022 XHCI root HUB> at usbus1 uhub4: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2 uhub5: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1 ugen0.1: <0x1022 XHCI root HUB> at usbus0 uhub6: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 uhub1: 2 ports with 2 removable, self powered uhub3: 5 ports with 5 removable, self powered uhub4: 5 ports with 5 removable, self powered uhub5: 4 ports with 4 removable, self powered uhub6: 4 ports with 4 removable, self powered uhub2: 5 ports with 5 removable, self powered uhub0: 5 ports with 5 removable, self powered ugen3.2: <Generic Mass Storage Device> at usbus3 umass0 on uhub2 umass0: <Generic Mass Storage Device, class 0/0, rev 2.00/1.00, addr 2> on usbus3 umass0: SCSI over Bulk-Only; quirks = 0x4001 umass0:7:0: Attached to scbus7 ugen4.2: <vendor 0x0557 product 0x8021> at usbus4 uhub7 on uhub3 uhub7: <vendor 0x0557 product 0x8021, class 9/0, rev 1.10/1.00, addr 2> on usbus4 uhub7: 4 ports with 4 removable, self powered ugen5.2: <vendor 0x13fe Patriot Memory> at usbus5 umass1 on uhub0 umass1: <vendor 0x13fe Patriot Memory, class 0/0, rev 2.00/1.00, addr 2> on usbus5 umass1: SCSI over Bulk-Only; quirks = 0x8100 umass1:8:1: Attached to scbus8 ugen4.3: <ATEN International Co. Ltd B020-U08-19 V1.1.109> at usbus4 ukbd0 on uhub7 ukbd0: <ATEN International Co. Ltd B020-U08-19 V1.1.109, class 0/0, rev 1.10/1.00, addr 3> on usbus4 kbd2 at ukbd0 ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: <WDC WD30EFRX-68EUZN0 80.00A80> ACS-2 ATA SATA 3.x device da4 at umass-sim1 bus 1 scbus8 target 0 lun 0 ada0: Serial Number WD-WMC4N0602840 ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) da4: < Patriot Memory PMAP> Removable Direct Access SCSI device da4: Serial Number 07012889093C4864 da4: 40.000MB/s transfers da4: 15272MB (31277056 512 byte sectors) da4: quirks=0x2<NO_6_BYTE> ada0: Command Queueing enabled ada0: 2861588MB (5860533168 512 byte sectors) ada0: quirks=0x1<4K> ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 ada1: <WDC WD4002FFWX-68TZ4N0 83.H0A83> ACS-2 ATA SATA 3.x device ada1: Serial Number NHGBVMNY ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 3815447MB (7814037168 512 byte sectors) ada2 at ahcich2 bus 0 scbus2 target 0 lun 0 ada2: <WDC WD30EFRX-68EUZN0 80.00A80> ACS-2 ATA SATA 3.x device ada2: Serial Number WD-WCC4N0066972 ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) da0 at umass-sim0 bus 0 scbus7 target 0 lun 0 ada2: Command Queueing enabled ada2: 2861588MB (5860533168 512 byte sectors) da0: ada2: quirks=0x1<4K> <Generic USB SD Reader 1.00> Removable Direct Access SCSI device ada3 at ahcich3 bus 0 scbus3 target 0 lun 0 da0: Serial Number 058F63626376 ada3: <WDC WD30EFRX-68EUZN0 80.00A80> ACS-2 ATA SATA 3.x device da0: 40.000MB/s transfers da0: Attempt to query device size failed: NOT READY, Medium not present da0: quirks=0x2<NO_6_BYTE> ada3: Serial Number WD-WMC4N0562899 ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada3: Command Queueing enabled ada3: 2861588MB (5860533168 512 byte sectors) ada3: quirks=0x1<4K> ada4 at ahcich4 bus 0 scbus4 target 0 lun 0 ada4: <WDC WD40EFRX-68N32N0 82.00A82> ACS-3 ATA SATA 3.x device ada4: Serial Number WD-WCC7K7XNJ4SN ada4: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada4: Command Queueing enabled ada4: 3815447MB (7814037168 512 byte sectors) da1 at umass-sim0 bus 0 scbus7 target 0 lun 1 ada4: quirks=0x1<4K> da1: <Generic USB CF Reader 1.01> Removable Direct Access SCSI device da1: Serial Number 058F63626376 ada5 at ahcich5 bus 0 scbus5 target 0 lun 0 da1: 40.000MB/s transfersada5: da1: Attempt to query device size failed: NOT READY, Medium not present <WDC WD30EFRX-68EUZN0 80.00A80> ACS-2 ATA SATA 3.x device da1: quirks=0x2<NO_6_BYTE> ada5: Serial Number WD-WMC4N0337435 ada5: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada5: Command Queueing enabled ada5: 2861588MB (5860533168 512 byte sectors) ada5: quirks=0x1<4K> da2 at umass-sim0 bus 0 scbus7 target 0 lun 2 da2: <Generic USB xD/SM Reader 1.02> Removable Direct Access SCSI device da2: Serial Number 058F63626376 da2: 40.000MB/s transfers da2: Attempt to query device size failed: NOT READY, Medium not present da2: quirks=0x2<NO_6_BYTE> da3 at umass-sim0 bus 0 scbus7 target 0 lun 3 da3: <Generic USB MS Reader 1.03> Removable Direct Access SCSI device da3: Serial Number 058F63626376 da3: 40.000MB/s transfers da3: Attempt to query device size failed: NOT READY, Medium not present da3: quirks=0x2<NO_6_BYTE> Trying to mount root from zfs:freenas-boot/ROOT/11.1-U6 []... em0: link state changed to UP GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7) GEOM_MIRROR: Device mirror/swap0 launched (2/2). GEOM_MIRROR: Device mirror/swap1 launched (2/2). GEOM_MIRROR: Device mirror/swap2 launched (2/2). GEOM_ELI: Device mirror/swap0.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: hardware GEOM_ELI: Device mirror/swap1.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: hardware GEOM_ELI: Device mirror/swap2.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: hardware hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> K8/4/48/0x1ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA> em0: link state changed to DOWN uhid0 on uhub7 uhid0: <ATEN International Co. Ltd B020-U08-19 V1.1.109, class 0/0, rev 1.10/1.00, addr 3> on usbus4 ums0 on uhub7 ums0: <ATEN International Co. Ltd B020-U08-19 V1.1.109, class 0/0, rev 1.10/1.00, addr 3> on usbus4 ums0: 5 buttons and [XYZ] coordinates ID=0 ums1 on uhub7 ums1: <ATEN International Co. Ltd B020-U08-19 V1.1.109, class 0/0, rev 1.10/1.00, addr 3> on usbus4 em0: link state changed to UP nfsd: can't register svc name bridge0: Ethernet address: 02:39:95:d5:b7:00 epair0a: Ethernet address: 02:40:d0:00:04:0a epair0b: Ethernet address: 02:40:20:00:05:0b epair0a: link state changed to UP epair0b: link state changed to UP epair0a: changing name to 'vnet0:3' bridge0: link state changed to UP vnet0:3: promiscuous mode enabled epair1a: Ethernet address: 02:40:d0:00:05:0a epair1b: Ethernet address: 02:40:20:00:06:0b epair1a: link state changed to UP epair1b: link state changed to UP epair1a: changing name to 'vnet0:4' epair1b: changing name to 'epair0b' vnet0:4: promiscuous mode enabled bridge1: Ethernet address: 02:39:95:d5:b7:01 em0: promiscuous mode enabled bridge1: link state changed to UP epair2a: Ethernet address: 02:40:d0:00:07:0a epair2b: Ethernet address: 02:40:20:00:08:0b epair2a: link state changed to UP epair2b: link state changed to UP em0: link state changed to DOWN epair2a: promiscuous mode enabled arp: 02:40:d0:00:07:0a is using my IP address 192.168.1.45 on epair2b! em0: link state changed to UP epair3a: Ethernet address: 02:40:d0:00:08:0a epair3b: Ethernet address: 02:40:20:00:09:0b epair3a: link state changed to UP epair3b: link state changed to UP epair3a: promiscuous mode enabled epair4a: Ethernet address: 02:40:d0:00:09:0a epair4b: Ethernet address: 02:40:20:00:0a:0b epair4a: link state changed to UP epair4b: link state changed to UP epair4a: promiscuous mode enabled arp: 02:40:d0:00:09:0a is using my IP address 192.168.1.43 on epair4b! epair5a: Ethernet address: 02:40:d0:00:0a:0a epair5b: Ethernet address: 02:40:20:00:0b:0b epair5a: link state changed to UP epair5b: link state changed to UP epair5a: promiscuous mode enabled arp: 192.168.1.43 moved from 02:40:d0:00:0a:0a to 02:34:da:00:11:0b on epair5b Limiting closed port RST response from 597 to 200 packets/sec Limiting closed port RST response from 339 to 200 packets/sec Limiting closed port RST response from 601 to 200 packets/sec Limiting closed port RST response from 632 to 200 packets/sec
I suspect the Emby/Sickrage IP issue is due to the IOcage, 112B3 and back tasks but not sure if that's what's shutting the server down.
Grateful for any help. Not sure if my budget is capable of a new 2U chassis, power, mobo, RAM, CPU, etc build.
Last edited: