SOLVED daily security run output: swap_pager_getswapspace(xx): failed

CPP-IT

Dabbler
Joined
Aug 14, 2017
Messages
43
I've seen a few other threads about this, but none of them seemed to have a solution or steps to a solution.

I got my morning email about the security issues, all well and good.

Lately I've seen a list of swap errors below is an excerpt:

Code:
> swap_pager_getswapspace(2): failed
> swap_pager_getswapspace(2): failed
> swap_pager_getswapspace(2): failed
> pid 88714 (python3.6), uid 0, was killed: out of swap space
> pid 58972 (python3.6), uid 0, was killed: out of swap space
> pid 58841 (python3.6), uid 0, was killed: out of swap space


I can see that it seems to start with some python processes, but I'm not certain which these PID belong to.

I saw a thread metion freenas-debug -t When I run this command it seems to be running fine then it appears to have halted at Middleware Jobs - 'midclt call core.get_jobs' @1577737050 and then I get logged out of the web UI.

I can't tell if this is on the USB devices that has FreeNAS on it or on the HDD that are in the zpool.

=== Edit 20191230- 1404===
Swaped the link for a code block as recommended by Chris.

Code:
# dmidecode 3.1
# SMBIOS entry point at 0x000f04c0
Found SMBIOS entry point in EFI, reading table from /dev/mem.
SMBIOS 2.7 present.
27 structures occupying 1294 bytes.
Table at 0x000EBB40.

Handle 0x0001, DMI type 0, 24 bytes
BIOS Information
    Vendor: Dell Inc.
    Version: A05
    Release Date: 07/26/2012
    Address: 0xF0000
    Runtime Size: 64 kB
    ROM Size: 8192 kB
    Characteristics:
        PCI is supported
        BIOS is upgradeable
        BIOS shadowing is allowed
        Boot from CD is supported
        Selectable boot is supported
        BIOS ROM is socketed
        EDD is supported
        5.25"/1.2 MB floppy services are supported (int 13h)
        3.5"/720 kB floppy services are supported (int 13h)
        3.5"/2.88 MB floppy services are supported (int 13h)
        Print screen service is supported (int 5h)
        8042 keyboard services are supported (int 9h)
        Serial services are supported (int 14h)
        Printer services are supported (int 17h)
        ACPI is supported
        USB legacy is supported
        BIOS boot specification is supported
        Targeted content distribution is supported
        UEFI is supported
    BIOS Revision: 4.6

Handle 0x0002, DMI type 1, 27 bytes
System Information
    Manufacturer: Dell Inc.
    Product Name: Inspiron 660
    Version:
    Serial Number: CJTLCX1
    UUID: ----
    Wake-up Type: Power Switch
    SKU Number: To be filled by O.E.M.
    Family: To be filled by O.E.M.

Handle 0x0003, DMI type 2, 15 bytes
Base Board Information
    Manufacturer: Dell Inc.
    Product Name: 084J0R   
    Version: A00
    Serial Number: .CJTLCX1.----.        

Handle 0x0004, DMI type 3, 22 bytes
Chassis Information
    Manufacturer: Dell Inc.
    Type: Desktop
    Lock: Not Present
    Version: Not Specified
    Serial Number: CJTLCX1
    Asset Tag: Not Specified
    Boot-up State: Safe
    Power Supply State: Safe
    Thermal State: Safe
    Height: Unspecified
    Number Of Power Cords: 1
    Contained Elements: 0
    SKU Number: To be filled by O.E.M.

Handle 0x0005, DMI type 9, 17 bytes
System Slot Information
    Designation: PCIEX16
    Type: x16 PCI Express
    Current Usage: In Use
    Length: Long
    ID: 0
    Characteristics:
        3.3 V is provided
        Opening is shared
        PME signal is supported
    Bus Address: 0000:00:01.0

Handle 0x0006, DMI type 9, 17 bytes
System Slot Information
    Designation: PCIEX1
    Type: x1 PCI Express
    Current Usage: In Use
    Length: Short
    ID: 1
    Characteristics:
        3.3 V is provided
        Opening is shared
        PME signal is supported
    Bus Address: 0000:00:1c.3

Handle 0x0007, DMI type 9, 17 bytes
System Slot Information
    Designation: PCIEX1
    Type: x1 PCI Express
    Current Usage: In Use
    Length: Short
    ID: 2
    Characteristics:
        3.3 V is provided
        Opening is shared
        PME signal is supported
    Bus Address: 0000:00:1c.4

Handle 0x0008, DMI type 9, 17 bytes
System Slot Information
    Designation: PCIEX1
    Type: x1 PCI Express
    Current Usage: In Use
    Length: Short
    ID: 3
    Characteristics:
        3.3 V is provided
        Opening is shared
        PME signal is supported
    Bus Address: 0000:00:1c.5

Handle 0x0009, DMI type 9, 17 bytes
System Slot Information
    Designation: PCIEX1
    Type: x1 PCI Express
    Current Usage: In Use
    Length: Short
    ID: 4
    Characteristics:
        3.3 V is provided
        Opening is shared
        PME signal is supported
    Bus Address: 0000:00:1c.6

Handle 0x000A, DMI type 11, 5 bytes
OEM Strings
    String 1: Dell System
    String 2: 1[0581]
    String 3: 3[3.3]
    String 4: 4[0001]
    String 5: 5[0000]
    String 6: 6[D0, DA]
    String 7: 7[]
    String 8: 8[]
    String 9: 9[]
    String 10: 10[A05]
    String 11: 11[]
    String 12: 12[]
    String 13: 13[D11M]
    String 14: 14[0]
    String 15: 15[11]

Handle 0x000B, DMI type 32, 20 bytes
System Boot Information
    Status: No errors detected

Handle 0x000C, DMI type 7, 19 bytes
Cache Information
    Socket Designation: Not Specified
    Configuration: Enabled, Not Socketed, Level 1
    Operational Mode: Varies With Memory Address
    Location: Internal
    Installed Size: 128 kB
    Maximum Size: 128 kB
    Supported SRAM Types:
        Unknown
    Installed SRAM Type: Unknown
    Speed: Unknown
    Error Correction Type: Unknown
    System Type: Other
    Associativity: 8-way Set-associative

Handle 0x000E, DMI type 7, 19 bytes
Cache Information
    Socket Designation: Not Specified
    Configuration: Enabled, Not Socketed, Level 2
    Operational Mode: Varies With Memory Address
    Location: Internal
    Installed Size: 512 kB
    Maximum Size: 512 kB
    Supported SRAM Types:
        Unknown
    Installed SRAM Type: Unknown
    Speed: Unknown
    Error Correction Type: Unknown
    System Type: Unified
    Associativity: 8-way Set-associative

Handle 0x000F, DMI type 7, 19 bytes
Cache Information
    Socket Designation: Not Specified
    Configuration: Enabled, Not Socketed, Level 3
    Operational Mode: Varies With Memory Address
    Location: Internal
    Installed Size: 3072 kB
    Maximum Size: 3072 kB
    Supported SRAM Types:
        Unknown
    Installed SRAM Type: Unknown
    Speed: Unknown
    Error Correction Type: Unknown
    System Type: Unified
    Associativity: Other

Handle 0x0010, DMI type 16, 23 bytes
Physical Memory Array
    Location: System Board Or Motherboard
    Use: System Memory
    Error Correction Type: None
    Maximum Capacity: 8 GB
    Error Information Handle: Not Provided
    Number Of Devices: 2

Handle 0x0011, DMI type 4, 42 bytes
Processor Information
    Socket Designation: CPU 1
    Type: Central Processor
    Family: Pentium
    Manufacturer: Intel
    ID: A9 06 03 00 FF FB EB BF
    Signature: Type 0, Family 6, Model 58, Stepping 9
    Flags:
        FPU (Floating-point unit on-chip)
        VME (Virtual mode extension)
        DE (Debugging extension)
        PSE (Page size extension)
        TSC (Time stamp counter)
        MSR (Model specific registers)
        PAE (Physical address extension)
        MCE (Machine check exception)
        CX8 (CMPXCHG8 instruction supported)
        APIC (On-chip APIC hardware supported)
        SEP (Fast system call)
        MTRR (Memory type range registers)
        PGE (Page global enable)
        MCA (Machine check architecture)
        CMOV (Conditional move instruction supported)
        PAT (Page attribute table)
        PSE-36 (36-bit page size extension)
        CLFSH (CLFLUSH instruction supported)
        DS (Debug store)
        ACPI (ACPI supported)
        MMX (MMX technology supported)
        FXSR (FXSAVE and FXSTOR instructions supported)
        SSE (Streaming SIMD extensions)
        SSE2 (Streaming SIMD extensions 2)
        SS (Self-snoop)
        HTT (Multi-threading)
        TM (Thermal monitor supported)
        PBE (Pending break enabled)
    Version: Intel(R) Pentium(R) CPU G2020 @ 2.90GHz
    Voltage: 0.0 V
    External Clock: 100 MHz
    Max Speed: 2900 MHz
    Current Speed: 2900 MHz
    Status: Populated, Enabled
    Upgrade: Socket BGA1155
    L1 Cache Handle: 0x000C
    L2 Cache Handle: 0x000E
    L3 Cache Handle: 0x000F
    Serial Number: Not Specified
    Asset Tag: Fill By OEM
    Part Number: Fill By OEM
    Core Count: 2
    Core Enabled: 2
    Thread Count: 2
    Characteristics:
        64-bit capable

Handle 0x0012, DMI type 17, 34 bytes
Memory Device
    Array Handle: 0x0010
    Error Information Handle: Not Provided
    Total Width: 64 bits
    Data Width: 64 bits
    Size: 4096 MB
    Form Factor: DIMM
    Set: None
    Locator: DIMM0
    Bank Locator: 00
    Type: DDR3
    Type Detail: Synchronous
    Speed: 1600 MT/s
    Manufacturer: Hynix/Hyundai
    Serial Number: 154C41AB
    Asset Tag: 9876543210
    Part Number: HMT351U6CFR8C-PB
    Rank: 2
    Configured Clock Speed: 1333 MT/s

Handle 0x0013, DMI type 20, 35 bytes
Memory Device Mapped Address
    Starting Address: 0x00000000000
    Ending Address: 0x000FFFFFFFF
    Range Size: 4 GB
    Physical Device Handle: 0x0012
    Memory Array Mapped Address Handle: 0x0016
    Partition Row Position: 1
    Interleave Position: 1
    Interleaved Data Depth: 1

Handle 0x0014, DMI type 17, 34 bytes
Memory Device
    Array Handle: 0x0010
    Error Information Handle: Not Provided
    Total Width: 64 bits
    Data Width: 64 bits
    Size: 4096 MB
    Form Factor: DIMM
    Set: None
    Locator: DIMM1
    Bank Locator: 00
    Type: DDR3
    Type Detail: Synchronous
    Speed: 1600 MT/s
    Manufacturer: Samsung
    Serial Number: 148E3FA3
    Asset Tag: 9876543210
    Part Number: M378B5273CH0-CK0
    Rank: 2
    Configured Clock Speed: 1333 MT/s

Handle 0x0015, DMI type 20, 35 bytes
Memory Device Mapped Address
    Starting Address: 0x00100000000
    Ending Address: 0x001FFFFFFFF
    Range Size: 4 GB
    Physical Device Handle: 0x0014
    Memory Array Mapped Address Handle: 0x0016
    Partition Row Position: 1
    Interleave Position: 2
    Interleaved Data Depth: 1

Handle 0x0016, DMI type 19, 31 bytes
Memory Array Mapped Address
    Starting Address: 0x00000000000
    Ending Address: 0x001FFFFFFFF
    Range Size: 8 GB
    Physical Array Handle: 0x0010
    Partition Width: 2

Handle 0x0018, DMI type 129, 8 bytes
OEM-specific Type
    Header and Data:
        81 08 18 00 01 01 02 01
    Strings:
        Intel_ASF
        Intel_ASF_001

Handle 0x001B, DMI type 127, 4 bytes
End Of Table


=== EDIT 20200103-0852 ===
Output of camcontrol devlist

Code:
sudo camcontrol devlist
<WDC WD30EFRX-68EUZN0 82.00A82>    at scbus0 target 0 lun 0 (pass0,ada0)
<MATSHITA DVD+-RW SW820 D.03>      at scbus1 target 0 lun 0 (pass1,cd0)
<ST2000DM001-1ER164 CC26>          at scbus2 target 0 lun 0 (pass2,ada1)
<SanDisk Ultra Fit 1.00>           at scbus4 target 0 lun 0 (pass3,da0)
<Generic- Multi-Card 1.00>         at scbus5 target 0 lun 0 (pass4,da1)
<SanDisk Ultra Fit 1.00>           at scbus6 target 0 lun 0 (pass5,da2)


Contents of /etc/fstab:

Code:
fdescfs /dev/fd fdescfs rw      0 0
/dev/mirror/swap0.eli   none    swap    sw      0       0
 
Last edited:

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080

CPP-IT

Dabbler
Joined
Aug 14, 2017
Messages
43
I've updated the main post with the output fromdmidecode as that seems a more human readable group of info.

If it would help I can also put up a dmesg output too.

Chris, The box was rebooted just today.

This is currently running off of a repurposed desktop box. There is hope to transition it to a genuine server box, but budget and time have not allowed for that yet.
 
Last edited:

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
You can use [code] your text here [/code] tags to put long lists from the console into your post on the forum.
 
Last edited:

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Here is the example, what I provide about my NAS:

Built in 2018, I reused the name, but this is the 8th FreeNAS unit I have built for home.
Case: Chenbro 4U 48 bay Chassis NR40700
- new old stock - http://www.chenbro.com/en-global/products/RackmountChassis/4U_Chassis/NR40700
System board: SuperMicro Motherboard X9SRL-F, LGA 2011/Socket R, IPMI
- new old stock - http://www.supermicro.com/products/motherboard/Xeon/C600/X9SRL-F.cfm
Processor: Intel Xeon E5-2650 V2, 2.6GHz 8 Core (16 thread)
- https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E5-2650+v2+@+2.60GHz&id=2042
Memory: 64 GB of 8GB sticks Samsung brand PC3-12800R, DDR3 Registered ECC
HBA: LSI/Broadcom SAS9207-8i, 6Gbps SAS PCI-E 3.0 HBA - flashed to IT Firmware: 20.00.07.00
- https://www.broadcom.com/products/storage/host-bus-adapters/sas-9207-8i
Connected to: two 6Gb/s 24-port 3.5" mini-SAS expander backplanes (80H10024001A0)
Emily pool: 2 vdevs
vdev-0 = 6 x 4 TB drives in RAID-z2 (6 Seagate Desktop drives - ST4000DM000-1F2168)
vdev-1 = 6 x 4 TB drives in RAID-z2 (6 Seagate Desktop drives - ST4000DM000-1F2168)
Irene pool: 2 vdevs
vdev-0 = 6 x 4 TB drives in RAID-z2 (6 Seagate Desktop drives - ST4000DM000-1F2168)
vdev-1 = 6 x 4 TB drives in RAID-z2 (6 Seagate Desktop drives - ST4000DM000-1F2168)
Backup pool: 2 vdevs
vdev-0 = 4 x 6 TB drives in RAID-z1 (4 WD Gold drives - WD6002FRYZ)
vdev-1 = 4 x 6 TB drives in RAID-z1 (4 WD Gold drives - WD6002FRYZ)
Boot pool: 1 vdev with 2 x 40 GB notebook drives in mirror (2 drives total - FUJITSU MHW2040BS)
Jails: running 1 warden jail (Plex) originally from the plugin - manually updating
Additional NIC: MELLANOX 10GB CONNECTX2 - MNPA19-XTR
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
What is your hard drive situation in your NAS because the swap space is on disk and we have not talked about what you have for storage yet.
 

CPP-IT

Dabbler
Joined
Aug 14, 2017
Messages
43
System Summary: This is currently a repurposed desktop system.

CPU: Intel Core2 Duo G2020 @2.90GHZ
Main Board: Dell -- I'll get the part number when I can get back to the box
RAM: 8GB PC1333 DDR NON-ecc
HDDs:
  • ada0 - Western Digital RED - WD-WCC4N5DKUS01 - 3TB
  • ada1 - Segate (I think) - Z4Z3MF1D - 2TB
  • da1 (boot pool) - SanDisk SureFit - 32GB
  • da2 (boot pool) - SanDisk SureFit - 32GB
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
The swap space should be on the two drives that are not part of the boot pool. How are they configured?
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
You can just give us the output of zpool status, and enclose the text in [CODE] tags like this[/CODE], to preserve the formatting . It should come out looking somethng like this:

Code:
  pool: Irene
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 0 days 03:42:32 with 0 errors on Wed Aug  7 03:42:33 2019
config:

        NAME                                            STATE     READ WRITE CKSUM
        Irene                                           ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/8710385b-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/87e94156-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/88db19ad-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/89addd3b-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/8a865453-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/8b66b1ef-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
          raidz2-1                                      ONLINE       0     0     0
            gptid/8c69bc72-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/8d48655d-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/8e2b6d1f-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/8efea929-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/8fd4d25c-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/90c2759a-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0

errors: No known data errors
 

CPP-IT

Dabbler
Joined
Aug 14, 2017
Messages
43
I believe the swap is part of the "data" pool.

Code:
  pool: cpp-bak
 state: ONLINE
  scan: scrub repaired 0 in 0 days 04:44:49 with 0 errors on Tue Dec  3 00:44:51 2019
config:

    NAME                                                STATE     READ WRITE CKSUM
    cpp-bak                                             ONLINE       0     0     0
      mirror-0                                          ONLINE       0     0     0
        gptid/bdb09b58-36d7-11e9-8c40-a41f726bb5e5.eli  ONLINE       0     0     0
        gptid/bf1de3a8-36d7-11e9-8c40-a41f726bb5e5.eli  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:02:04 with 0 errors on Thu Dec 19 03:47:04 2019
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        da0p2   ONLINE       0     0     0
        da2p2   ONLINE       0     0     0

errors: No known data errors

 

CPP-IT

Dabbler
Joined
Aug 14, 2017
Messages
43
Anyone seeing anything like this in their setups?

I've added the output of camcontrol devlist to the original post.
 
Joined
Oct 7, 2016
Messages
29
Anyone seeing anything like this in their setups?

I've added the output of camcontrol devlist to the original post.
To avoid this issue I have been running with the sysctl tunable vm.disable_swapspace_pageouts=1 for the last few years..
It seems to disable pageouts which can eat swap space, you do need plenty of memory for your workload though.

Paul
 

CPP-IT

Dabbler
Joined
Aug 14, 2017
Messages
43
To avoid this issue I have been running with the sysctl tunable vm.disable_swapspace_pageouts=1 for the last few years..
It seems to disable pageouts which can eat swap space, you do need plenty of memory for your workload though.

Paul

I appreciate the info. I'll look into it as a possibility.
 

CPP-IT

Dabbler
Joined
Aug 14, 2017
Messages
43
In other trouble shooting "news" --

I did a nmap ping sweep of our network, and I found an active ping on an address that was going to be an SAMBA AD in a jail. But when I do a jls or iocage list they both list nothing running.

So I'm wondering if a config busted somewhere and that jail is still trying to be active/shutdown and failing which is causing these errors.

I'm not certain how I'd fix such an issue though.
 

CPP-IT

Dabbler
Joined
Aug 14, 2017
Messages
43
I'm not certain this solved the issue correctly, but I did some dataset clean up yesterday. As part of that clean up I deleted the dataset that was for the jails. Thankfully in my case there was no loss from the dataset deletion, all those jails were basically experimental.

Today, there was no long winded email about swap failures. So I'm marking this solved.

If someone knows of a less drastic way to accomplish this task, that would be appreciated.
 
Top