SOLVED Any idea why my idle FreeNAS is swapping?

[NASbox]

My FreeNAS box is sitting mostly idle (have an ssh session with tmux running), and a couple of Samba shares that aren't doing much, but for some reason there is over 500MB on the swap. I have 32GB of RAM and 2 Pools 8x6TB RaidZ2 and a 2x4TB mirror.

Is there something I need to change/tune?

I don't know how long this data has been on the swap, but I should mention that when I booted FreeNAS, the first thing I did was replace and resilver a 6TB that had been removed/RMA's after a disk failure on my main RaidZ2 pool. Does resilvering cause swapping to occur?

I also ran this script: (to see if it would provide any clue)
https://github.com/ocochard/myscripts/blob/master/FreeBSD/freebsd-memory.sh

Code:

SYSTEM MEMORY INFORMATION:
mem_wire:       32223371264 (  30730MB) [ 96%] Wired: disabled for paging out
mem_active:  +     24322048 (     23MB) [  0%] Active: recently referenced
mem_inactive:+    254951424 (    243MB) [  0%] Inactive: recently not referenced
mem_cache:   +            0 (      0MB) [  0%] Cached: almost avail. for allocation
mem_free:    +    489963520 (    467MB) [  1%] Free: fully available for allocation
mem_gap_vm:  +    276844544 (    264MB) [  0%] Memory gap: UNKNOWN
______________ ____________ ___________ ______
mem_all:     =  33269452800 (  31728MB) [100%] Total real memory managed
mem_gap_sys: +    867729408 (    827MB)        Memory gap: Kernel?!
______________ ____________ ___________
mem_phys:    =  34137182208 (  32555MB)        Total real memory available
mem_gap_hw:  +    222556160 (    212MB)        Memory gap: Segment Mappings?!
______________ ____________ ___________
mem_hw:      =  34359738368 (  32768MB)        Total real memory installed

SYSTEM MEMORY SUMMARY:
mem_used:       33614823424 (  32057MB) [ 97%] Logically used memory
mem_avail:   +    744914944 (    710MB) [  2%] Logically available memory
______________ ____________ __________ _______
mem_total:   =  34359738368 (  32768MB) [100%] Logically total memory

 

[dlavigne]

Were you able to determine the cause?

 

[jgreco]

Were you able to determine the cause?

Because he has relatively little memory, and at some point there was momentary memory pressure and a massive amount of swap-eligible user pages.

Even an incredibly heavily resourced FreeNAS system can see this at times.

All that's really going on is that there's a brief userland spike in memory demand. Maybe someone logged into the GUI and a bunch of stuff started executing, or even just a cron job ran. If this happens at the same time as ZFS has allocated all the "free" memory and is actively using it as ARC, you can get a momentary run on memory. ZFS is supposed to free ARC in order to "make way" for userland, but this doesn't actually happen instantaneously, and the kernel may instead consider it a memory shortfall and start the process of swapping stuff out. Then ZFS releases some ARC and the memory pressure is gone, but in the meantime some stuff has been swapped out. The exact interaction has changed a bit over the years as ZFS has been made to behave better.

 

[NASbox]

Were you able to determine the cause?

Thanks @dlavigne for giving this thread a poke.

Because he has relatively little memory, and at some point there was momentary memory pressure and a massive amount of swap-eligible user pages.

Even an incredibly heavily resourced FreeNAS system can see this at times.

All that's really going on is that there's a brief userland spike in memory demand. Maybe someone logged into the GUI and a bunch of stuff started executing, or even just a cron job ran. If this happens at the same time as ZFS has allocated all the "free" memory and is actively using it as ARC, you can get a momentary run on memory. ZFS is supposed to free ARC in order to "make way" for userland, but this doesn't actually happen instantaneously, and the kernel may instead consider it a memory shortfall and start the process of swapping stuff out. Then ZFS releases some ARC and the memory pressure is gone, but in the meantime some stuff has been swapped out. The exact interaction has changed a bit over the years as ZFS has been made to behave better.

Thanks @jgreco for the explanation... I'm pretty sure that this is the explanation. I'll bet that resilvering sucked up all available memory. I did a reboot and haven't seen this again recently.

 

[jgreco]

Yup, resilvering can place pressure on the ARC to "hold more stuff." And under normal operations, a lot of that FreeNAS middleware and GUIware that's in user memory isn't active, therefore looking like a juicy target for swapout by the kernel. The ARC size management code is supposed to make sure that this doesn't actually happen to idle pages, by leaving some free space for userland.

 

[Chris Moore]

for some reason there is over 500MB on the swap

Please don't take it as a criticism, I am just curious, why are you worried about a little swap usage?

 

[danb35]

why are you worried about a little swap usage?

Probably because there are so many threads here saying, with some variation, "If your FreeNAS box is using swap, you're doing it wrong." It's never really been true, but any use of swap has long been stated to indicate that your system needs more RAM.

 

[NASbox]

Yup, resilvering can place pressure on the ARC to "hold more stuff." And under normal operations, a lot of that FreeNAS middleware and GUIware that's in user memory isn't active, therefore looking like a juicy target for swapout by the kernel. The ARC size management code is supposed to make sure that this doesn't actually happen to idle pages, by leaving some free space for userland.

I think you are right about this as I had never seen that behaviour before, and AFAIK, rebooting has cleared it up (I haven't really been looking but didn't anything either.)

Please don't take it as a criticism, I am just curious, why are you worried about a little swap usage?

@Chris Moore - no offense taken, your input is always constructinve and valued

Probably because there are so many threads here saying, with some variation, "If your FreeNAS box is using swap, you're doing it wrong." It's never really been true, but any use of swap has long been stated to indicate that your system needs more RAM.

"Worried" is a bit too stong... I was curious as to why an "idle" system would be using so much swap, and as @danb35 indicates, I though it might mean that I need to do some optimization, or indicate a software problem that needed to be addressed.

 

[jgreco]

"Worried" is a bit too stong... I was curious as to why an "idle" system would be using so much swap, and as @danb35 indicates, I though it might mean that I need to do some optimization, or indicate a software problem that needed to be addressed.

500MB is *nothing* on a system that routinely runs large multi-hundred-MB executables.

The debate over whether one should build systems (like FreeNAS) running huge piggy software stacks is a different one; about 75% of the FreeBSD VM's around here run 256MB RAM and that's quite sufficient for many purposes.

 

[NASbox]

about 75% of the FreeBSD VM's around here run 256MB RAM and that's quite sufficient for many purposes.

Just curious... are you talking about VMs running on a FreeNAS? If so what type of applicatons/environment?

 

[jgreco]

No, service provider operations, mostly running under ESXi. Some of us make the world go round. ;-)

 

[NASbox]

No, service provider operations, mostly running under ESXi. Some of us make the world go round. ;-)

Pardon my ignorance, when you say service provider operations I'm assuming you mean things like dns, radiaus, LDAP, ntp, mail server, SQL server etc. or is it more like microservices for web apps? (or all of the above)

 

[jgreco]

All of the above fits well enough, but there's also a whole range of network-building tools such as routers (not your home NAT gateway), VPN, and other fun stuff.

UNIX systems have gotten fat. Back in the day, one of the two big UUCP hubs I ran was a Sun 3/60LE workstation, 12MB RAM on a 20MHz 68020 host. It ran several concurrent public user dial-ins (FreeBSD as a serial terminal server took that from two up to six), mail hub connecting dozens of sites, Cnews taking a full news feed, and oh it also ran X11 on its head.

Maintaining large complicated UNIX systems is a nightmare I'm familiar with. The last quarter century I've spent breaking things up into their component bits and making a truism of Sun's "The network is the computer." Both jails and virtual machines play out well, though they fit somewhat different roles.

It's been a blast to watch the rest of the world "discover" containers and software-defined networking. Turns out I'd been doing this stuff for years.

So I still run lots of infrastructure for various little corners of the Internet.

 

[NASbox]

Cool...

UNIX systems have gotten fat.

Unfortunately everything has gotten fat... It's amazing how what used to run in 50MB is now 400-500MB.

I was installing some simple apps on my wife's tablet tonight and some of them were 50-60MB! Even today I've seen whole linux systems on a raspberry pi in less space! Computers/memory/storage has gotten cheap and human labour has gotten expensive, so efficeincy doesn't seem to matter any more as it is too expensive.

 

[jgreco]

Cool... Unfortunately everything has gotten fat... It's amazing how what used to run in 50MB is now 400-500MB.

I was installing some simple apps on my wife's tablet tonight and some of them were 50-60MB! Even today I've seen whole linux systems on a raspberry pi in less space! Computers/memory/storage has gotten cheap and human labour has gotten expensive, so efficeincy doesn't seem to matter any more as it is too expensive.

This wouldn't bother me, except that the fat is typically a symptom of bloat. As you create more interdependencies, the number of attack vectors increases dramatically, and not all of it comes with a meaningful return on investment.

One of the things I'm known for in the FreeBSD community was creating the first framework (XKERNEL) to build an "appliance" version of FreeBSD, which eventually became PicoBSD. At the time, it was possible (difficult but possible) to strip down a FreeBSD kernel and custom userland to fit onto a 1.44MB floppy. I had a bunch of diskless 486 PC's and good graphics cards and I wanted to make into Xterminals without the drama of needing a netboot ROM using an ethernet card that supported that. You just needed to get the environment bootstrapped and then it could launch X11 from NFS.

Today's FreeBSD 12.1R kernel is maybe 18MB, which can be zipped down to 7MB, but basically no hope of cramming it onto a floppy.

 

[NASbox]

This wouldn't bother me, except that the fat is typically a symptom of bloat. As you create more interdependencies, the number of attack vectors increases dramatically, and not all of it comes with a meaningful return on investment.

One of the things I'm known for in the FreeBSD community was creating the first framework (XKERNEL) to build an "appliance" version of FreeBSD, which eventually became PicoBSD. At the time, it was possible (difficult but possible) to strip down a FreeBSD kernel and custom userland to fit onto a 1.44MB floppy. I had a bunch of diskless 486 PC's and good graphics cards and I wanted to make into Xterminals without the drama of needing a netboot ROM using an ethernet card that supported that. You just needed to get the environment bootstrapped and then it could launch X11 from NFS.

Today's FreeBSD 12.1R kernel is maybe 18MB, which can be zipped down to 7MB, but basically no hope of cramming it onto a floppy.

We are a bit off topic, but I'm enjoying the conversation. Wow! That's impressive.... I've got a lot of respect for people that can handle that low level work like building/optimizing kernels and the like. I'm very late to the game FreeNAS was my first real nix box. Now with Microsoft's predatory business practices I'm working as hard as I can to get everything on Linux. I'm pretty comfortable on Linux after about 2 years, but I must say I find FreeBSD WAY HARDER!

As you create more interdependencies, the number of attack vectors increases dramatically, and not all of it comes with a meaningful return on investment.

A Truer word has never been spoken.... In some cases NO ROI (at least for the user). With just about every app "phoning home" for updates and websites that make use of 10+ "cloud" services with all the potential xss vulnerabilites, attempting to do IT security is a nightmare. I'm coming to the conclusion that most internet activities need to be run in containers that are regularly discarded and refreshed, and the interfaces between those containers highly defined and firewalled/monitored.

Given that you are a FreeBSD expert, I'm wondering if I could avail of your good nature an have you take a look at this...
https://www.ixsystems.com/community/threads/disk-identification.81146/#post-562125
This problem has been driving me crazy for a couple of weeks. My intuition tells me it's a simple problem -IF, and big IF you know FreeBSD intimately.

BTW Happy New Year

 

[jgreco]

We are a bit off topic, but I'm enjoying the conversation. Wow! That's impressive.... I've got a lot of respect for people that can handle that low level work like building/optimizing kernels and the like. I'm very late to the game FreeNAS was my first real nix box. Now with Microsoft's predatory business practices I'm working as hard as I can to get everything on Linux. I'm pretty comfortable on Linux after about 2 years, but I must say I find FreeBSD WAY HARDER!

Just the opposite here. I find all the various versions of Linux with their multitude of ways of doing things to be very distressing. I don't mind that there could be multiple ways to do things, but I find it hard to get a handle on it when developers actively try to create as many competing systems as possible. Over time, FreeBSD has a much greater level of overall stability. Things do morph slowly, but it's generally not dramatic change like "oh systemd!"

The upside to all of *BSD, Linux is that everything's out there for you to discover if you try hard enough. In the worst case you read the source code.

A Truer word has never been spoken.... In some cases NO ROI (at least for the user). With just about every app "phoning home" for updates and websites that make use of 10+ "cloud" services with all the potential xss vulnerabilites, attempting to do IT security is a nightmare. I'm coming to the conclusion that most internet activities need to be run in containers that are regularly discarded and refreshed, and the interfaces between those containers highly defined and firewalled/monitored.

So, yeah. That.

As part of the generation that brought the Internet to the public, I fear we made a bunch of mistakes. A lot of this is that we've made it wicked hard for people to run their own services -- some of them, such as e-mail, are incredibly difficult to set up and maintain in a fully standards-compliant fashion. Making things like DAV difficult mean that most people simply sync their contacts and calendars out to Google or Yahoo!. Even just running a website is harder than it ought to be.

For years, I've worked to build service-provider grade infrastructure, which basically means systems that can hold their own out on the public Internet, without needing to be behind some specialized firewall gear. Part of the business model here used to be that we shipped gear that was placed on service provider networks, and you never knew how "raw" that would be, so you had to assume the worst. Getting things like DNS, NTP, SSL, etc., to work correctly and reliably can be a bit challenging. I've got a set of system building tools that do most of it automatically.

It was fun watching the Linux guys develop containers, "hey look we have this brand new idea that's never been done before" - wait, PHK did jails a decade before... haha. I'm usually a bit disappointed at the real world application of jails/containers, because they generally involve putting a full OS install into the container. The much more technically challenging way is to build a jail that just contains the needed bits.

But however you do it, creating strong barriers between bits of your services is a good idea.

Given that you are a FreeBSD expert, I'm wondering if I could avail of your good nature an have you take a look at this...
https://www.ixsystems.com/community/threads/disk-identification.81146/#post-562125
This problem has been driving me crazy for a couple of weeks. My intuition tells me it's a simple problem -IF, and big IF you know FreeBSD intimately.

BTW Happy New Year

So your question there is.... what? The GPT is somehow screwed up? That's what it sounds like.

What happens when you do

# glabel list ada3p1

Which might not show up since it sounds like you wiped the swap partition? Or

# glabel list ada3p2

Which you should be able to compare side-by-side with other entries such as maybe ada2p2.

Is the kernel throwing errors when you do any of this, like the always delightful "the {primary,secondary} GPT table is corrupt" message? Because normally if it is P.O.'ed at you it will tell you. There's a lot of soft probing that you can do with glabel to zero in on what's going on.

 

[NASbox]

Just the opposite here. I find all the various versions of Linux with their multitude of ways of doing things to be very distressing. I don't mind that there could be multiple ways to do things, but I find it hard to get a handle on it when developers actively try to create as many competing systems as possible. Over time, FreeBSD has a much greater level of overall stability. Things do morph slowly, but it's generally not dramatic change like "oh systemd!"

I'll agree with you on this one. Where I disagree is that there have been some of the GNU utilities have added a lot of very useful options (that still fit with the unix philosophy-do one thing and one thing well) that are absent in FreeBSD.

The upside to all of *BSD, Linux is that everything's out there for you to discover if you try hard enough. In the worst case you read the source code.

You can still read the source code applies to linux as well. Only problem is that the skill level is WAY WAY HIGHER.

Linux to it's credit has made "nix" accessable to a much wider audience by lowering the slope on the learning curve. You can start out using a GUI (babys start with picutre books), and as you mature in your knowledge (older children/adults learn to read) you can move to the command line. If I want to sync a directory with a remote server, give me a command line and rsync/ssh... If I want to move a bunch of random files that don't fit a pattern I'll take the GUI please. I think the knowledge level in the FreeBSD community is (by necessity) so high that they can not empathize with the newer/less experienced user. For a desktop, it leaves a lot to be desired from a usability point of view (until you scale the steep learning curve). For a hardened systems like embedded systems or service-provider grade infrastructure it's also great. That's why I use FreeNAS/pfSense... it's an open product and I believe that there is a highly skilled and dedicated community behind it.

To their credit, distributions like Linux Mint, Manjaro, Linux Lite, etc have made it easier to install linux than windows, and on old/lower power hardware. I put linux on my wife's lap top faster than the damn thing did the windows updates from the 6 month period when the laptop recieved it's install image and the time we turned it on. Bloody thing took hours. Download, run install programs, boot, download, run install programs, amd boot again (repeat several times). Then change all the privacy settings and uninstall the crap that has been foised on your system.

As part of the generation that brought the Internet to the public, I fear we made a bunch of mistakes. A lot of this is that we've made it wicked hard for people to run their own services -- some of them, such as e-mail, are incredibly difficult to set up and maintain in a fully standards-compliant fashion. Making things like DAV difficult mean that most people simply sync their contacts and calendars out to Google or Yahoo!. Even just running a website is harder than it ought to be.

For years, I've worked to build service-provider grade infrastructure, which basically means systems that can hold their own out on the public Internet, without needing to be behind some specialized firewall gear. Part of the business model here used to be that we shipped gear that was placed on service provider networks, and you never knew how "raw" that would be, so you had to assume the worst. Getting things like DNS, NTP, SSL, etc., to work correctly and reliably can be a bit challenging. I've got a set of system building tools that do most of it automatically.

Are your tools things that you have built, or generally available tools? That sounds like a really smart idea. It would be a great idea to build projects so that it would be easy to install a buch of setup tools (maybe even with a GUI if it would speed up the manual work) that could be cleanly/easily uninstalled after the system was set up. If major changes were required, just run a quick install over the net and then uninstall again. No security holes and no bloat.

It was fun watching the Linux guys develop containers, "hey look we have this brand new idea that's never been done before" - wait, PHK did jails a decade before... haha. I'm usually a bit disappointed at the real world application of jails/containers, because they generally involve putting a full OS install into the container. The much more technically challenging way is to build a jail that just contains the needed bits.

AMAP I like your idea of lean services without the extra bloat. IMHO it's a cost benefit -- time/cost to create vs ROI of less overhead (and possibly better security).

I believe that the reason Linux has gotten so much more play is that they have made things much easier/user friendly so that a new user can get going/be productive much faster. It was pretty easy for me to get LXD and Docker running to the point that I could begin to do some simple work with them.

If the desktop commuity could just get together and create more standardization, and in particular good install/tuning tools, I believe the time is right for them to take a lot of market share away from windows. If there was a winows installer that would make it easy to just replace windows, and it could be made to work reliably, I'll bet there would be a lot of people switching. I switche my 82 year old from Windows to Linux about 2 years ago. In the two years since I've done it, he has had 0 problems. Windows was f*#@!ng up every couple of months. For the large percentage of the population that just needs a web browser (and maybe a real email client), and to stream some audio or video Linux is way better.

Apple has always had it's fanboy(fanpersons-to be politically correct) that would take anything Steve Jobs wanted to push at them. To his credit, he help get Adobe Flash out of the desktop environment, but now that Steve is gone, a lot of the emphasis on quality is gone.

Now Microsoft is starting to act like Apple, and that is cultue change --something that I am personally not willing to tolerate, so I worked very hard to educate myself to move to linux. It was a bit of a slug for about a year, now I would only go back kicking and screaming. Unfortunately the masses of technosheep have to choke down whatever is thown at them since they don't know any better.

But however you do it, creating strong barriers between bits of your services is a good idea.

Yup!

 

[danb35]

To their credit, distributions like Linux Mint, Manjaro, Linux Lite, etc have made it easier to install linux than windows, and on old/lower power hardware.

True--and though there have been at least a few "desktop FreeBSD" projects, they don't seem to stick around very long (see, e.g., PC-BSD, TrueOS, Project Trident). Maybe GhostBSD will make it work... You can always install plain FreeBSD and put a GUI on it, but even if that solves the "ease of use" issue, it doesn't do much for "ease of installation."

Software availability has become less of an issue in recent years, but not for the right reason--you still can't run Quicken or TurboTax under Linux or FreeBSD, but you don't have to any more, because they're "in the cloud."

 

[NASbox]

True--and though there have been at least a few "desktop FreeBSD" projects, they don't seem to stick around very long (see, e.g., PC-BSD, TrueOS, Project Trident). Maybe GhostBSD will make it work... You can always install plain FreeBSD and put a GUI on it, but even if that solves the "ease of use" issue, it doesn't do much for "ease of installation."

Software availability has become less of an issue in recent years, but not for the right reason--you still can't run Quicken or TurboTax under Linux or FreeBSD, but you don't have to any more, because they're "in the cloud."

So is TrueOS dead? That's an IX initiative isn't it?

Is Ghost OS https://en.wikipedia.org/wiki/Ghost_(operating_system) FreeBSD? If this article is right, then it's a complete redesign since Ghost OS is a microkernel.