11.1U6 - shutting down 2-3x a day

[marcevan]

I went thru the pain of trying the 11.2BETA3 upgrade but then went back to 11.1U6 after having no luck with Iocage plugins reaching the outside world.

Back on 11.1U6 and all IoCage jails are off so my warden jails (4 - emby, Plex, Transmission, Sickrage) are all up, autostart, and plugins are working.

Only oddness are Sickrage and Emby have the navtree error so they don't show in leftnav under plugins, and their plugins are in red under plugins/installed. They are up and working, however.

Two to three times a day now, the server shuts down and I have to physically power it on.

Some initial descoping of the root cause:

1. Power to the chassis - Wattbox registers entire rack system at 117V, 2.1A with little to no fluctuation (little when I use Control4 music in a room as it has passive amps that turn on/off with demand). All power is on a 20A circuit so we're not even close to a spike nor a limit.
2. /data/crash has no entries since 2016.
3. Looking in var/log* I see middleware is at it again (others have posted)
   Code:

   2018/09/16 14:24:28] (WARNING) middlewared._loop_monitor_thread():1010 - Task seems blocked:  File "/usr/local/lib/python3.6/site-p
   ackages/middlewared/plugins/device.py", line 68, in devd_loop																	   
      await devd_listen(middleware)

4. #3 happens about ever 2-10 seconds according to the file (same entries time stamped) - but they are only warnings.
5. Monitor shows Emby and Sickrage plugins have messages like: arp [MAC] is using my IP address [of it's jail] on epair2b/epair4b (respectively).
6. Monitor shows (on bootup): ACPI BIOS warning FADT field PM2controlblock has valid length but zero address
7. Another warning about VIMAGE being experimental feature
8. Zpool status is clean
   Code:

   pool: Media																													   
    state: ONLINE																													 
     scan: scrub in progress since Sun Sep 16 00:00:03 2018																			
   	   2.27T scanned at 1.14G/s, 991G issued at 499M/s, 5.45T total																
   	   0 repaired, 17.74% done, 0 days 02:37:00 to go																			 
   config:																															 
   																																   
   	   NAME											STATE	 READ WRITE CKSUM												 
   	   Media										   ONLINE	   0	 0	 0												 
   		 raidz2-0									  ONLINE	   0	 0	 0												 
   		   gptid/41804705-6726-11e3-831d-6805ca1adbd8  ONLINE	   0	 0	 0												 
   		   gptid/41eefd8e-6726-11e3-831d-6805ca1adbd8  ONLINE	   0	 0	 0												 
   		   gptid/8f0a7046-7192-11e7-806a-6805ca1adbd8  ONLINE	   0	 0	 0												 
   		   gptid/58c022d3-6757-11e8-94e5-6805ca1adbd8  ONLINE	   0	 0	 0												 
   		   gptid/d9cee92c-9662-11e3-a1dc-6805ca1adbd8  ONLINE	   0	 0	 0												 
   		   gptid/43c4e34c-6726-11e3-831d-6805ca1adbd8  ONLINE	   0	 0	 0												 
   																																   
   errors: No known data errors																										
   																																   
     pool: freenas-boot																												
    state: ONLINE																													 
     scan: scrub repaired 0 in 0 days 00:05:31 with 0 errors on Thu Sep 13 03:50:32 2018											   
   config:																															 
   																																   
   	   NAME										  STATE	 READ WRITE CKSUM													
   	   freenas-boot								  ONLINE	   0	 0	 0													
   		 gptid/6e7337e9-7fad-11e4-a208-6805ca1adbd8  ONLINE	   0	 0	 0													
   																																   
   errors: No known data errors

9. dmesg.today has the entirety of the bootup. Note it's 2018, so we're out of copyright by FreeBSD Project
   Code:

   Copyright (c) 1992-2017 The FreeBSD Project.
   Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
   	The Regents of the University of California. All rights reserved.
   FreeBSD is a registered trademark of The FreeBSD Foundation.
   FreeBSD 11.1-STABLE #0 r321665+9902d126c39(freenas/11.1-stable): Tue Aug 21 12:24:37 EDT 2018
   	root@nemesis.tn.ixsystems.com:/freenas-11-releng/freenas/_BE/objs/freenas-11-releng/freenas/_BE/os/sys/FreeNAS.amd64 amd64
   FreeBSD clang version 5.0.0 (tags/RELEASE_500/final 312559) (based on LLVM 5.0.0svn)
   VT(vga): resolution 640x480
   CPU: AMD A4-6300 APU with Radeon(tm) HD Graphics	 (3700.06-MHz K8-class CPU)
     Origin="AuthenticAMD"  Id=0x610f31  Family=0x15  Model=0x13  Stepping=1
     Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
     Features2=0x3e98320b<SSE3,PCLMULQDQ,MON,SSSE3,FMA,CX16,SSE4.1,SSE4.2,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C>
     AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
     AMD Features2=0x1ebbfff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,XOP,SKINIT,WDT,LWP,FMA4,TCE,NodeId,TBM,Topology,PCXC,PNXC>
     Structured Extended Features=0x8<BMI1>
     SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=65536
     TSC: P-state invariant, performance statistics
   real memory  = 17179869184 (16384 MB)
   avail memory = 15740456960 (15011 MB)
   Event timer "LAPIC" quality 100
   ACPI APIC Table: <ALASKA A M I>
   FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
   FreeBSD/SMP: 1 package(s) x 2 core(s)
   random: unblocking device.
   WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
   ACPI BIOS Warning (bug): Optional FADT field Pm2ControlBlock has valid Length but zero Address: 0x0000000000000000/0x1 (20170728/tbfadt-796)
   ioapic0 <Version 2.1> irqs 0-23 on motherboard
   SMP: AP CPU #1 Launched!
   Timecounter "TSC-low" frequency 1850030317 Hz quality 1000
   random: entropy device external interface
   kbd1 at kbdmux0
   module_register_init: MOD_LOAD (vesa, 0xffffffff80fc84c0, 0) error 19
   nexus0
   vtvga0: <VT VGA driver> on motherboard
   cryptosoft0: <software crypto> on motherboard
   aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard
   padlock0: No ACE support.
   acpi0: <ALASKA A M I> on motherboard
   acpi0: Power Button (fixed)
   cpu0: <ACPI CPU> on acpi0
   cpu1: <ACPI CPU> on acpi0
   attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
   Timecounter "i8254" frequency 1193182 Hz quality 0
   Event timer "i8254" frequency 1193182 Hz quality 100
   atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
   atrtc0: registered as a time-of-day clock, resolution 1.000000s
   Event timer "RTC" frequency 32768 Hz quality 0
   hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
   Timecounter "HPET" frequency 14318180 Hz quality 950
   Event timer "HPET" frequency 14318180 Hz quality 550
   Event timer "HPET1" frequency 14318180 Hz quality 450
   Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
   acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
   pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
   pci0: <ACPI PCI bus> on pcib0
   pci0: <base peripheral, IOMMU> at device 0.2 (no driver attached)
   vgapci0: <VGA-compatible display> port 0xf000-0xf0ff mem 0xc0000000-0xcfffffff,0xfeb00000-0xfeb3ffff irq 17 at device 1.0 on pci0
   vgapci0: Boot video device
   pci0: <multimedia, HDA> at device 1.1 (no driver attached)
   pcib1: <ACPI PCI-PCI bridge> irq 17 at device 5.0 on pci0
   pci1: <ACPI PCI bus> on pcib1
   em0: <Intel(R) PRO/1000 Network Connection 7.6.1-k> port 0xd000-0xd01f mem 0xfeac0000-0xfeadffff,0xfea00000-0xfea7ffff,0xfeae0000-0xfeae3fff irq 17 at device 0.0 on pci1
   em0: Using MSIX interrupts with 3 vectors
   em0: Ethernet address: 68:05:ca:1a:db:d8
   xhci0: <XHCI (generic) USB 3.0 controller> mem 0xfeb46000-0xfeb47fff irq 18 at device 16.0 on pci0
   xhci0: 32 bytes context size, 64-bit DMA
   xhci0: Unable to map MSI-X table 
   usbus0 on xhci0
   usbus0: 5.0Gbps Super Speed USB v3.0
   xhci1: <XHCI (generic) USB 3.0 controller> mem 0xfeb44000-0xfeb45fff irq 17 at device 16.1 on pci0
   xhci1: 32 bytes context size, 64-bit DMA
   xhci1: Unable to map MSI-X table 
   usbus1 on xhci1
   usbus1: 5.0Gbps Super Speed USB v3.0
   ahci0: <AMD Hudson-2 AHCI SATA controller> port 0xf140-0xf147,0xf130-0xf133,0xf120-0xf127,0xf110-0xf113,0xf100-0xf10f mem 0xfeb4d000-0xfeb4d7ff irq 19 at device 17.0 on pci0
   ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier supported
   ahcich0: <AHCI channel> at channel 0 on ahci0
   ahcich1: <AHCI channel> at channel 1 on ahci0
   ahcich2: <AHCI channel> at channel 2 on ahci0
   ahcich3: <AHCI channel> at channel 3 on ahci0
   ahcich4: <AHCI channel> at channel 4 on ahci0
   ahcich5: <AHCI channel> at channel 5 on ahci0
   ohci0: <AMD FCH USB Controller> mem 0xfeb4c000-0xfeb4cfff irq 18 at device 18.0 on pci0
   usbus2 on ohci0
   usbus2: 12Mbps Full Speed USB v1.0
   ehci0: <AMD FCH USB 2.0 controller> mem 0xfeb4b000-0xfeb4b0ff irq 17 at device 18.2 on pci0
   usbus3: EHCI version 1.0
   usbus3 on ehci0
   usbus3: 480Mbps High Speed USB v2.0
   ohci1: <AMD FCH USB Controller> mem 0xfeb4a000-0xfeb4afff irq 18 at device 19.0 on pci0
   usbus4 on ohci1
   usbus4: 12Mbps Full Speed USB v1.0
   ehci1: <AMD FCH USB 2.0 controller> mem 0xfeb49000-0xfeb490ff irq 17 at device 19.2 on pci0
   usbus5: EHCI version 1.0
   usbus5 on ehci1
   usbus5: 480Mbps High Speed USB v2.0
   isab0: <PCI-ISA bridge> at device 20.3 on pci0
   isa0: <ISA bus> on isab0
   pcib2: <ACPI PCI-PCI bridge> at device 20.4 on pci0
   pci2: <ACPI PCI bus> on pcib2
   ohci2: <OHCI (generic) USB controller> mem 0xfeb48000-0xfeb48fff irq 18 at device 20.5 on pci0
   usbus6 on ohci2
   usbus6: 12Mbps Full Speed USB v1.0
   amdtemp0: <AMD CPU On-Die Thermal Sensors> on hostb4
   acpi_button0: <Power Button> on acpi0
   uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
   amdsbwd0: <AMD SB8xx/SB9xx/Axx Watchdog Timer> at iomem 0xfec000f0-0xfec000f3,0xfec000f4-0xfec000f7 on isa0
   atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
   atkbd0: <AT Keyboard> irq 1 on atkbdc0
   kbd0 at atkbd0
   atkbd0: [GIANT-LOCKED]
   hwpstate0: <Cool`n'Quiet 2.0> on cpu0
   ZFS filesystem version: 5
   ZFS storage pool version: features support (5000)
   Timecounters tick every 1.000 msec
   freenas_sysctl: adding account.
   freenas_sysctl: adding directoryservice.
   freenas_sysctl: adding middlewared.
   freenas_sysctl: adding network.
   freenas_sysctl: adding services.
   ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
   ugen5.1: <AMD EHCI root HUB> at usbus5
   ugen6.1: <AMD OHCI root HUB> at usbus6
   uhub0: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
   uhub1: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
   ugen3.1: <AMD EHCI root HUB> at usbus3
   ugen4.1: <AMD OHCI root HUB> at usbus4
   uhub2: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
   uhub3: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
   ugen2.1: <AMD OHCI root HUB> at usbus2
   ugen1.1: <0x1022 XHCI root HUB> at usbus1
   uhub4: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
   uhub5: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
   ugen0.1: <0x1022 XHCI root HUB> at usbus0
   uhub6: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
   uhub1: 2 ports with 2 removable, self powered
   uhub3: 5 ports with 5 removable, self powered
   uhub4: 5 ports with 5 removable, self powered
   uhub5: 4 ports with 4 removable, self powered
   uhub6: 4 ports with 4 removable, self powered
   uhub2: 5 ports with 5 removable, self powered
   uhub0: 5 ports with 5 removable, self powered
   ugen3.2: <Generic Mass Storage Device> at usbus3
   umass0 on uhub2
   umass0: <Generic Mass Storage Device, class 0/0, rev 2.00/1.00, addr 2> on usbus3
   umass0:  SCSI over Bulk-Only; quirks = 0x4001
   umass0:7:0: Attached to scbus7
   ugen4.2: <vendor 0x0557 product 0x8021> at usbus4
   uhub7 on uhub3
   uhub7: <vendor 0x0557 product 0x8021, class 9/0, rev 1.10/1.00, addr 2> on usbus4
   uhub7: 4 ports with 4 removable, self powered
   ugen5.2: <vendor 0x13fe Patriot Memory> at usbus5
   umass1 on uhub0
   umass1: <vendor 0x13fe Patriot Memory, class 0/0, rev 2.00/1.00, addr 2> on usbus5
   umass1:  SCSI over Bulk-Only; quirks = 0x8100
   umass1:8:1: Attached to scbus8
   ugen4.3: <ATEN International Co. Ltd B020-U08-19 V1.1.109> at usbus4
   ukbd0 on uhub7
   ukbd0: <ATEN International Co. Ltd B020-U08-19 V1.1.109, class 0/0, rev 1.10/1.00, addr 3> on usbus4
   kbd2 at ukbd0
   ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
   ada0: <WDC WD30EFRX-68EUZN0 80.00A80> ACS-2 ATA SATA 3.x device
   da4 at umass-sim1 bus 1 scbus8 target 0 lun 0
   ada0: Serial Number WD-WMC4N0602840
   ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
   da4: < Patriot Memory PMAP> Removable Direct Access SCSI device
   da4: Serial Number 07012889093C4864
   da4: 40.000MB/s transfers
   da4: 15272MB (31277056 512 byte sectors)
   da4: quirks=0x2<NO_6_BYTE>
   ada0: Command Queueing enabled
   ada0: 2861588MB (5860533168 512 byte sectors)
   ada0: quirks=0x1<4K>
   ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
   ada1: <WDC WD4002FFWX-68TZ4N0 83.H0A83> ACS-2 ATA SATA 3.x device
   ada1: Serial Number NHGBVMNY
   ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
   ada1: Command Queueing enabled
   ada1: 3815447MB (7814037168 512 byte sectors)
   ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
   ada2: <WDC WD30EFRX-68EUZN0 80.00A80> ACS-2 ATA SATA 3.x device
   ada2: Serial Number WD-WCC4N0066972
   ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
   da0 at umass-sim0 bus 0 scbus7 target 0 lun 0
   ada2: Command Queueing enabled
   ada2: 2861588MB (5860533168 512 byte sectors)
   da0: ada2: quirks=0x1<4K>
   <Generic USB SD Reader 1.00> Removable Direct Access SCSI device
   ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
   da0: Serial Number 058F63626376
   ada3: <WDC WD30EFRX-68EUZN0 80.00A80> ACS-2 ATA SATA 3.x device
   da0: 40.000MB/s transfers
   da0: Attempt to query device size failed: NOT READY, Medium not present
   da0: quirks=0x2<NO_6_BYTE>
   ada3: Serial Number WD-WMC4N0562899
   ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
   ada3: Command Queueing enabled
   ada3: 2861588MB (5860533168 512 byte sectors)
   ada3: quirks=0x1<4K>
   ada4 at ahcich4 bus 0 scbus4 target 0 lun 0
   ada4: <WDC WD40EFRX-68N32N0 82.00A82> ACS-3 ATA SATA 3.x device
   ada4: Serial Number WD-WCC7K7XNJ4SN
   ada4: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
   ada4: Command Queueing enabled
   ada4: 3815447MB (7814037168 512 byte sectors)
   da1 at umass-sim0 bus 0 scbus7 target 0 lun 1
   ada4: quirks=0x1<4K>
   da1: <Generic USB CF Reader 1.01> Removable Direct Access SCSI device
   da1: Serial Number 058F63626376
   ada5 at ahcich5 bus 0 scbus5 target 0 lun 0
   da1: 40.000MB/s transfersada5: 
   da1: Attempt to query device size failed: NOT READY, Medium not present
   <WDC WD30EFRX-68EUZN0 80.00A80> ACS-2 ATA SATA 3.x device
   da1: quirks=0x2<NO_6_BYTE>
   ada5: Serial Number WD-WMC4N0337435
   ada5: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
   ada5: Command Queueing enabled
   ada5: 2861588MB (5860533168 512 byte sectors)
   ada5: quirks=0x1<4K>
   da2 at umass-sim0 bus 0 scbus7 target 0 lun 2
   da2: <Generic USB xD/SM Reader 1.02> Removable Direct Access SCSI device
   da2: Serial Number 058F63626376
   da2: 40.000MB/s transfers
   da2: Attempt to query device size failed: NOT READY, Medium not present
   da2: quirks=0x2<NO_6_BYTE>
   da3 at umass-sim0 bus 0 scbus7 target 0 lun 3
   da3: <Generic USB MS Reader 1.03> Removable Direct Access SCSI device
   da3: Serial Number 058F63626376
   da3: 40.000MB/s transfers
   da3: Attempt to query device size failed: NOT READY, Medium not present
   da3: quirks=0x2<NO_6_BYTE>
   Trying to mount root from zfs:freenas-boot/ROOT/11.1-U6 []...
   em0: link state changed to UP
   GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
   GEOM_MIRROR: Device mirror/swap0 launched (2/2).
   GEOM_MIRROR: Device mirror/swap1 launched (2/2).
   GEOM_MIRROR: Device mirror/swap2 launched (2/2).
   GEOM_ELI: Device mirror/swap0.eli created.
   GEOM_ELI: Encryption: AES-XTS 128
   GEOM_ELI:	 Crypto: hardware
   GEOM_ELI: Device mirror/swap1.eli created.
   GEOM_ELI: Encryption: AES-XTS 128
   GEOM_ELI:	 Crypto: hardware
   GEOM_ELI: Device mirror/swap2.eli created.
   GEOM_ELI: Encryption: AES-XTS 128
   GEOM_ELI:	 Crypto: hardware
   hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> K8/4/48/0x1ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA>
   em0: link state changed to DOWN
   uhid0 on uhub7
   uhid0: <ATEN International Co. Ltd B020-U08-19 V1.1.109, class 0/0, rev 1.10/1.00, addr 3> on usbus4
   ums0 on uhub7
   ums0: <ATEN International Co. Ltd B020-U08-19 V1.1.109, class 0/0, rev 1.10/1.00, addr 3> on usbus4
   ums0: 5 buttons and [XYZ] coordinates ID=0
   ums1 on uhub7
   ums1: <ATEN International Co. Ltd B020-U08-19 V1.1.109, class 0/0, rev 1.10/1.00, addr 3> on usbus4
   em0: link state changed to UP
   nfsd: can't register svc name
   bridge0: Ethernet address: 02:39:95:d5:b7:00
   epair0a: Ethernet address: 02:40:d0:00:04:0a
   epair0b: Ethernet address: 02:40:20:00:05:0b
   epair0a: link state changed to UP
   epair0b: link state changed to UP
   epair0a: changing name to 'vnet0:3'
   bridge0: link state changed to UP
   vnet0:3: promiscuous mode enabled
   epair1a: Ethernet address: 02:40:d0:00:05:0a
   epair1b: Ethernet address: 02:40:20:00:06:0b
   epair1a: link state changed to UP
   epair1b: link state changed to UP
   epair1a: changing name to 'vnet0:4'
   epair1b: changing name to 'epair0b'
   vnet0:4: promiscuous mode enabled
   bridge1: Ethernet address: 02:39:95:d5:b7:01
   em0: promiscuous mode enabled
   bridge1: link state changed to UP
   epair2a: Ethernet address: 02:40:d0:00:07:0a
   epair2b: Ethernet address: 02:40:20:00:08:0b
   epair2a: link state changed to UP
   epair2b: link state changed to UP
   em0: link state changed to DOWN
   epair2a: promiscuous mode enabled
   arp: 02:40:d0:00:07:0a is using my IP address 192.168.1.45 on epair2b!
   em0: link state changed to UP
   epair3a: Ethernet address: 02:40:d0:00:08:0a
   epair3b: Ethernet address: 02:40:20:00:09:0b
   epair3a: link state changed to UP
   epair3b: link state changed to UP
   epair3a: promiscuous mode enabled
   epair4a: Ethernet address: 02:40:d0:00:09:0a
   epair4b: Ethernet address: 02:40:20:00:0a:0b
   epair4a: link state changed to UP
   epair4b: link state changed to UP
   epair4a: promiscuous mode enabled
   arp: 02:40:d0:00:09:0a is using my IP address 192.168.1.43 on epair4b!
   epair5a: Ethernet address: 02:40:d0:00:0a:0a
   epair5b: Ethernet address: 02:40:20:00:0b:0b
   epair5a: link state changed to UP
   epair5b: link state changed to UP
   epair5a: promiscuous mode enabled
   arp: 192.168.1.43 moved from 02:40:d0:00:0a:0a to 02:34:da:00:11:0b on epair5b
   Limiting closed port RST response from 597 to 200 packets/sec
   Limiting closed port RST response from 339 to 200 packets/sec
   Limiting closed port RST response from 601 to 200 packets/sec
   Limiting closed port RST response from 632 to 200 packets/sec

   
   
   

I suspect the Emby/Sickrage IP issue is due to the IOcage, 112B3 and back tasks but not sure if that's what's shutting the server down.

Grateful for any help. Not sure if my budget is capable of a new 2U chassis, power, mobo, RAM, CPU, etc build.

 

[marcevan]

One thing I just noticed.. If you went to iocages, the jails are defaulted to boot=ON (autostart). So I just noticed that even when manually turning them off in iocage and enabling warden jails back on, upon reboot iocage and wardens are on which causes IP conflicts.

Hence, I issued a iocage set boot=off to each jail and enabled the warden jails.... I'll see if that was the issue but would like to think that wouldn't tank the entire server.

 

[marcevan]

Server shut down at some point last night.

It's go to be hardware, as there's no trace of any log file containing anything of merit. Even the dmesg goes from last night to this morning and only thing is this morning was a power on sequence when I pushed the power button. Zero from yesterday.

I'll shut it down today and inspect the case. I'll narrow hardware down to:

1. Dust
2. PSU
3. Connectors
4. CPU Cooler

I'm assuming best to start with cooling being the issue of a forced shut down.

Disks are fine, pool is fine. I've checked those in FreeNAS. No video card. Mobo could be issue, but that's a needle in a haystack.

 

[marcevan]

Ahem, wow.

Shut it down after doing a sysctl - a | grep temperature showing 68.1 Celsius. Took a look at the reports on CPU temperature and the spikes over 70 C align with the server going down. Graph showed typically in the 60s and up to 70s then cuts off meaning a shutdown.

Since the chassis has all the 6 HDs in a front cage and case fans on the sides of that cage, they use the 2 system fan ports on the mobo. Now racked on a shelf, those fans are not capable of doing much as they are snug to the inside of the rack. I'll just unplug them and get my 2 rear Noctua fans going so we can at least exhaust out the back much of the air.

Now alarming is that it's on the stock cooler that came with A4-6300 AMD CPU!! Hadn't opened this server up in years so now that I did, I just ordered a Noctua CPU cooler for it that supports FM2 socket. I'll have to remove mobo to put in a back plate underneath, but that shouldn't be the end of the world.

Back up for now with twin Noctua exhaust fans on the back running on full (3 pin, so no PWM) but they are silent. CPU core temps in the 50s now. Hoping they stay there but all in all after I killed iocage and those jails there's less running on the server, so the shut downs went from 2x a day to once every 3 days. If I can get temps down with the Noctua CPU cooler I may be home free.

 

[toadman]

Glad you figured it out!

 

[marcevan]

Not quite. Now I think it's the PSU. Thermaltake 550W but if I calculate my A4-6300 (AMD) FM2 CPU, 1 single LAN PCI-E card, 6 SATA 3.5 drives, mouse + keyboard and 24/7 on, it's about a 275 Watt power supply at 90% utilization. I'm probably on a very inneficient, 5-year PSU and so now I'm getting one to efficiently handle the actual load. I think everyone should be doing a PSU calculation and not just guessing.

 

[marcevan]

Not the PSU... just replaced it and only it. And it boots for 5-7 minutes of life then shuts down.

Well now I'm not sure what to do.

Power supply came early (before the chassis, etc).

Server won't stay on for more than 6 minutes before powering down.
1. New power supply and it boots and I get 6 minutes of life on the URL or the plugins URLs. So doubt it's power
2. FreeNAS - nothing ever shows in crash directory so not sure where else to look.
3. Disks - when I can get into the URL for freeNAS the disks are all fine, pool is fine.
4. Motherboard - I looked at it and cleaned it. No puffy caps and no smell of burnt anything or scorch marks anywhere.
5. RAM - not sure but doubt they just go bad. Any way to quick verify in my 6 minutes of life I get?
6. CPU? Can it overheat in 6 minutes? I'd like to think not...as I don't want to put on the new Noctua cooler until I move the board to new chassis, or verify I need a new board.

This is just weird.

 

[tman5005]

You could try booting into a different operating system from a USB disk, such as an Ubuntu Live CD. If it still shuts down in 7 minutes, then it probably has nothing to do with FreeNAS.

Otherwise, if it stays up with a different operating system on a Live CD (or USB stick) then you could use that bootup to mount the FreeNAS operating system disk and look at some of the logs in the FreeNAS /var/log folder, like messages or dmesg. Assuming of course Ubuntu supports that filesystem type. Or you could use a FreeBSD installer disk and instead of choosing the install option choose to go immediately to the shell.

 

[marcevan]

Thanks tman.

What I did was reboot into BIOS to see if a watchdog process was running. Thought was to kill that and see what happens.

Upon loading BIOS, the CPU temperature is shown at 90 C and was incrementally rising to 95 C when I decided to just exit BIOS and shut down.

3 options present as possibilities:

1. The CPU fan stopped working (it's stock) and thus the CPU has no means of cooling, leading to a shut down.
2. The 4-pin CPU PWM contact on the motherboard was not working properly, leading to a shut down. This means either a) the port isn't powering the fan or b) port was providing power but not reporting temperature accurately.
3. The CPU itself was malfunctioning in some way leading to a heat escalation with the CPU fan incapable of cooling sufficiently.

Today I will receive a basic Arctic cooling CPU fan (cheap, like $35) and boot with the cover of the chassis off so I can verify if the CPU fan from Arctic is spinning. That would help isolate possibility #1. If it's not spinning at all, then I have a motherboard issue (#2) and I need a new Mobo. If it's spinning and BIOS shows the same CPU temperature, then it's likely possibility #3, but could be #2 b).

I did "test" the physical temperature after removing the server and bringing it from rack to office and popped the cover off. Blazing hot to the touch (was not bare handed though).

 

[marcevan]

Still awaiting the Arctic cooler, but the LSI 2011 HBA arrived so popped that into my Win 10 machine and flashed it according to the post by boris D (the 21 steps).

So when my 4U arrives I will move the mobo, power, RAM, the cooler (hopefully arrives tonight when the 4U shows up) and plug in the LSI HBA and put the 6 SATA drives to that for better performance.

 

[tman5005]

Wow, if your CPU is running that hot something is definitely wrong. Most likely then your shutdowns are due to the CPU shutting itself down for being too hot. Possibly your fan is not running for some reason or maybe even your thermal compound has degraded, although I've never experienced that myself. My desktop has a core i5-3570k in it that runs BOINC projects that max the CPU and the temp maxes at 65C with a Zalman CNPS10x cooler.

Similarly, I just put together a server with a Xeon E3 1230 V6 and that runs also about 60-65C when running mprime for burn-in testing with all 4 cores (8 hyperthreaded) running at 100%. That has a Noctua NH-U12S cooler.

 

[marcevan]

Whew. Got my arctic 64 Pro installed and put it all into the U4 chassis. Did the LSI HBA (already in IT mode).

I'm flummoxed by the sysctl -a | grep temperature readouts now, because they come back 2.0 C and 0.5 C after a few refreshes. Cannot be right by any stretch but they are lower than they used to show. Wish there was an accurate CPU temp read in FreeNAS/FreeBSD but short of rebooting just to go to BIOS to see temps is a bit of a pain.

Fans are all running: 2 in front drawing in, 2 in back drawing out, PSU pulling in and out, and CPU fan is now visibly running.