SOLVED Increase SWAP? How?

[s_falahati]

Hey,
I just started using FreeNAS since Saturday on a hardware I salvaged from my older PC with brand new hard drives. Following is the spec of the device:

Main Board: Asrock P67 Extreme 6
CPU: Intel Core i5 2500k
RAM: 16GB (4x4GB 2133)
GPU: Asus GTX 730 (Mainboard doesn't support Intel Graphics)
Boot Device: 60GB SP SSD on Intel SATA3 6G Slot 0
SSD Storage: 250GB Samsung 850Evo SSD (used for jails) on Intel SATA3 6G Slot 1
HDD Storage: ~16TiB (4x6TB WD RED in 'Media' configuration - RAIDZ1) on Intel SATA2 3G Slots 2 to 5

Now, on my first install, I had some problems with swap size and the famous `getswapspace` error. So after playing with it for a while, I tried reinstalling freeNAS to use it for real this time.

However, on an empty device, after selecting `ownCloud` plugin for installation I got this error again and therefore installation failed, leaving me with a lot of problems including but not limited to network interface disappearing and losing the ability to install any other plugin (or reinstall ownCloud).
Therefore I decided to reset the configuration via the terminal and give it another shot. I also skipped installing `ownCloud` as I suspected it might have some memory leak of a sort and used another similar plugin named `nextCloud`. I also have `Plex Server` installed but it has no library as of now as I am still in the process of moving my files to the new device and I am limited to 100Mbps connection of the Seagate Personal Cloud. So far so good.

However, I just double checked the device an hour ago and figured the RAM is almost full, here are the details of `top` sorted by size:

Code:

last pid: 84632;  load averages:  0.28,  0.25,  0.23														up 0+14:08:14  15:14:36
83 processes:  1 running, 80 sleeping, 2 zombie
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 88M Active, 248M Inact, 939M Laundry, 14G Wired, 303M Free
ARC: 12G Total, 117M MFU, 12G MRU, 1637K Anon, 36M Header, 435M Other
	 12G Compressed, 17G Uncompressed, 1.46:1 Ratio
Swap: 4096M Total, 22M Used, 4074M Free

  PID USERNAME	THR PRI NICE   SIZE	RES STATE   C   TIME	WCPU COMMAND
 6857 www		   1  21	0   460M 45992K lockf   2   0:00   0.00% httpd
63354 www		   1  20	0   456M 26928K lockf   0   0:00   0.00% httpd
 6859 www		   1  20	0   456M 26912K kqread  2   0:00   0.00% httpd
 6855 www		   1  52	0   456M 26912K lockf   2   0:00   0.00% httpd
 6856 www		   1  52	0   456M 26912K lockf   2   0:00   0.00% httpd
 6858 www		   1  52	0   456M 26912K lockf   2   0:00   0.00% httpd
 6802 root		  1  20	0   456M 26896K select  2   0:01   0.00% httpd
 6860 root		 15  52	0   281M   219M umtxn   2   1:52   0.00% uwsgi
 6742	 88	   16  20	0   235M 57540K select  1   0:12   0.02% mysqld
37307 root		  1  20	0   235M   191M select  0  20:58   0.00% smbd
 6756	972	   17  48	0   203M 55716K uwait   3   0:28   0.01% Plex Media Server
  213 root		 20  20	0   191M   139M kqread  3   4:16   0.03% python3.6
 6861	972	   13  52   15   173M 47484K piperd  2   0:42   0.06% Plex Script Host
27834 root		  1  20	0   170M   135M select  1   0:01   0.00% smbd
 7043	972	   17  52	0   157M 40176K usem	0   0:32   0.04% Plex DLNA Server
 2991 root		  1  20	0   147M   114M kqread  0   0:09   0.00% uwsgi
 5817 root		  6  20	0   132M 36576K select  0   0:07   0.00% python2.7
 4667 root		  6  20	0   132M 36972K select  2   0:07   0.01% python2.7
27850 root		  1  20	0   128M 96224K select  3   0:00   0.00% smbd
27848 root		  1  20	0   128M 96200K select  2   0:07   0.00% smbd
 2980 root		  1  52	0   101M 84028K select  3   0:45   0.00% python3.6

As you can see around 3GB of ram is used by the `httpd` processes and ultimately I have around 1.5GB (including cache) that can be free if needed, and already swap is filling up. So considering the fact that there is no VM, no active plugin (running, but not active in any sense) and the only thing the server did up until now was to transfer files via `smb`, is this normal? (Also what is the 14G used by `wired`? Is it arc? And if it is, should I be worry that it takes almost 80% of the ram?)
As you can see the swap size is 4GB and I have read somewhere on this forum that the swap size should be automatically set to 1x or 2x of memory size, is this somehow a bug? Can I increase the swap size just to be sure?
That being said, I already tried increasing the swap size with a swap file, however, `fstab` resets after each reboot maybe by the `freeNAS` itself, so unless I write a script to add it on each restart, I might be out of luck. But should I even try that and do I really need it?
This is also my first experience with FreeBSD or any other Linux distro for that matter except CentOS and RH.

 

[Chris Moore]

and already swap is filling up.

If your system is using swap, you don't have enough RAM. The solution is not more swap. The solution is to get more RAM.
I understand that may not be an option, for now, but you probably want to start thinking about buying better hardware for your server.
You don't have to go full on "new server" but you might pickup some decent second hand gear from eBay.
There is a button in my signature for useful links and the top couple (actually, the third and fourth) listings in there are hardware guides.

For the time being, there are some changes that can be made to this system to get a little more out of it. Did you copy data to it, or move data to it?
Some of the changes might require doing thing that would wipe out your data, so I don't want to tell you the wrong thing if this is the only copy of your data...

 

[s_falahati]

Well, I just moved like <1T of data to it but I can move it back. So please go on with the suggestions.

I know adding more RAM probably going to solve the problem, however, isn't 16G enough for 16T of disk space? I was thinking about adding more SWAP so in case some spikes happened, it won't crash. Unfortunately, all my rams are 4G and the Mainboard has only 4 slots of memory, therefore, I cant simply expand it. I need to buy another MB for this CPU and I failed to find a decent one on the market that supports 1155; or any good DDR3 ram.

That being said I investigated this more and it seems that ARC will free some memory if the need arises. I just started an empty VM and ARC already changed its size to 11G instead of 12.5G. How much can I expect ARC to shrink if something happens that needs more memory considering the fact that the server is mainly used as a Plex server and Windows backup destination? (I have also two network cameras that occasionally upload some footages to it via FTP)

 

[s_falahati]

I also just seen this thread on your useful links:
https://forums.freenas.org/index.php?resources/how-to-relocate-swap-to-an-ssd-or-other-partition.68/

Can I do this on the second SSD (250G Samsung)? It is almost empty (except the two plugins installed on it) and I can safely repartition it.

 

[Redcoat]

What version of FreeNAS are you using? 11.1-Release has a memory leak bug...

 

[Chris Moore]

I need to buy another MB for this CPU and I failed to find a decent one on the market that supports 1155; or any good DDR3 ram.

I am not saying that you must get this, but this is an option for a socket 1155 CPU:
https://www.ebay.com/itm/TESTED-Sup...w-E97379-001-Heatsink-I-O-Shield/232430611431
It is the type of system board I use in my home NAS and it works well for me. With 32GB of RAM, I almost never need to use any swap space, but the only plugin I use is Plex.

isn't 16G enough for 16T of disk space?

If all you were doing is sharing files, that is true, but if you want to run plugins, they need to have some additional memory because they use a type of shared (para) virtualization. It is not as resource intensive as running a full virtual machine, but it does take some additional resources.

I also just seen this thread on your useful links:
https://forums.freenas.org/index.php?resources/how-to-relocate-swap-to-an-ssd-or-other-partition.68/

Can I do this on the second SSD (250G Samsung)? It is almost empty (except the two plugins installed on it) and I can safely repartition it.

If you move the swap space using the directions in the resource, that could be a solution, but it takes the whole drive for swap space.
So, you might want to do this instead.
I was going to suggest that you look at this value in the System / Advanced tab:

This allows you to select the amount of space of each disk that is dedicated to swap space. You can change this value to a higher number if you want to do that. The trouble is, the actual swap partition is only changed if you destroy your storage pool and recreate it. Once the pool is created, the partition sizes are not able to change. This would allow more swap space and the remaining portion of the disk can still be used for other purposes.

 

[Chris Moore]

That being said I investigated this more and it seems that ARC will free some memory if the need arises. I just started an empty VM and ARC already changed its size to 11G instead of 12.5G. How much can I expect ARC to shrink if something happens that needs more memory considering the fact that the server is mainly used as a Plex server and Windows backup destination? (I have also two network cameras that occasionally upload some footages to it via FTP)

ARC will consume as much memory as you have, up to a point. I have a server at work with 128GB of memory and the ARC is using about 75% of that. The more memory the system can dedicate to ARC, the better the performance should be, but I would expect the minimum amount you could get along with to be around 8GB. You already know about the rule 1GB to 1TB and that is not a hard guide, but a general kind of "it would be better if" kind of rule. The impact will be on system responsiveness and it could cause the occasional crash if memory is not available when it is needed..

 

[Chris Moore]

I apologize, on further reflection, the system board I suggested doesn't really help you because you have a full load of 4GB modules. Memory prices right now are a bit outrageous. I was looking at upgrading the memory in one of my servers and the price to go from 16GB to 32GB was going to be over $350. There really are not many easy paths. If you are interested in a hardware suggestion for newer system board, processor and memory, please say and I will make some suggestions.

 

[s_falahati]

@Redcoat
WebUI reports it as FreeNAS-11.1-U2, however, I can't see any newer version on the stable channel. Can I ask where is the memory leak so I can avoid it for now? Or is it in the underlying services?

@Chris Moore
Thanks, Chris,
I have only one question, can I change this value and then destroy my SSD volume (Samsung, not the one booting the FreeNAS) and recreate it without touching the WD RED Raid volume? So maybe change it to 8, recreate the SSD volume and then I should end up with 10G of swap (8G on SSD and 2G on Reds). Is this possible so I can skip the `backing up data` part?

... but I would expect the minimum amount you could get along with to be around 8GB.

So that's the reason documentation specifically asks for the minimum of 8GB of ram. So I am right to consider I can push the server to like 12 or 10 GB of ram using VMs and other stuff at the expense of system performance but never lower (considering that the OS itself might need up to 2G of ram + 8G for ARC)?

 

[Redcoat]

WebUI reports it as FreeNAS-11.1-U2, however, I can't see any newer version on the stable channel. Can I ask where is the memory leak so I can avoid it for now? Or is it in the underlying services?

That's good - 11.1-U2 doesn't have the issue.

 

[Chris Moore]

I have only one question, can I change this value and then destroy my SSD volume (Samsung, not the one booting the FreeNAS) and recreate it without touching the WD RED Raid volume? So maybe change it to 8, recreate the SSD volume and then I should end up with 10G of swap (8G on SSD and 2G on Reds). Is this possible so I can skip the `backing up data` part?

This might work, but it might not and I would need to do some testing to be sure because it is unlikely that a developer will chime in here. The reason is this, they changed the way the swap system works recently. The way it used to work, each disk had a fully independent swap partition and all those partitions were pooled (not with ZFS) together to give the total swap space for the system. The problem with that was, if you had swap in use and a drive dropped out for any reason, like a failure, it would crash the system because the swap was damaged. This potential for crashing began to be more an issue when certain things changed that caused swap to be used more often. The change they recently made was to take the 1st swap partition and mirror it with the 2nd, then the 3rd swap partition would be mirrored with the 4th and so on. They don't use the boot drive for swap, so don't count it, and if you have a drive that doesn't have a partner, that swap partition isn't used. This means that the drive with no partner (the SSD) would likely not be used for swap at all. I am not entirely sure I like the new arrangement, but it is what we have right now.

So that's the reason documentation specifically asks for the minimum of 8GB of ram. So I am right to consider I can push the server to like 12 or 10 GB of ram using VMs and other stuff at the expense of system performance but never lower (considering that the OS itself might need up to 2G of ram + 8G for ARC)?

I think there is some flexibility in that but I caution you that the more you push the boundary, the more likely it is to crash. If the system needs more memory and it can't allocate it because of having used all of the swap space, it will not deal with it well. One of the guiding principles I use is that the server should always be over built so the operating system never runs out of resources.

 

[s_falahati]

That's good - 11.1-U2 doesn't have the issue.

Ok, that's a relief. :)
However, I am pretty sure that the time I got the error when installing `ownCloud` was some sort of memory leak. It filled both ram and swap in like a minute and then device became so unresponsive that I couldn't investigate the reason behind it.

... I was looking at upgrading the memory in one of my servers and the price to go from 16GB to 32GB was going to be over $350. ...

Well, that's on crypto mining guys. Can't find anything useful these days or prices are too high.
I also can't buy anything from the eBay anyway tho. It takes ages to get here.
Worst of all, local stores are not that much in the storage business here either. I can only find new stuff and even then, they are very pricey. That's why I was eager to increase the swap instead of buying new parts for the server.

This might work, but it might not and I would need to do some testing to be sure because it is unlikely that a developer will chime in here. The reason is this, they changed the way the swap system works recently. The way it used to work, each disk had a fully independent swap partition and all those partitions were pooled (not with ZFS) together to give the total swap space for the system. The problem with that was, if you had swap in use and a drive dropped out for any reason, like a failure, it would crash the system because the swap was damaged. This potential for crashing began to be more an issue when certain things changed that caused swap to be used more often. The change they recently made was to take the 1st swap partition and mirror it with the 2nd, then the 3rd swap partition would be mirrored with the 4th and so on. They don't use the boot drive for swap, so don't count it, and if you have a drive that doesn't have a partner, that swap partition isn't used. This means that the drive with no partner (the SSD) would likely not be used for swap at all. I am not entirely sure I like the new arrangement, but it is what we have right now.

Yeap, that was the case. I double checked the `gpart` and I can confirm there is a 2GB swap partition on all devices except the boot device. Therefore I have 5 swap partitions but only 4GB swap in total. Therefore FreeNAS sure tries to do as you said and as SSD is alone in the array, its swap partition is ignored.

However, I added a 10G partition to it and managed to add it as the third swap device by creating a `post-init` task:

Code:

swapon /dev/ada1p1

Considering the fact that the SSD is never going to get replaced, it should be safe. Right? :)


Thank you all guys. With the information you provided, hopefully, I now know more about FreeNAS, limitations, and specs needed for each use case.

I am going to mark the thread as solved.

 

[Chris Moore]

Considering the fact that the SSD is never going to get replaced, it should be safe. Right? :)

Thank you all guys. With the information you provided, hopefully, I now know more about FreeNAS, limitations, and specs needed for each use case.

I am going to mark the thread as solved.

If you did need to replace it, to put in a larger drive, when you offline the disk it is supposed to flush the swap. As long as it is done gracefully, it should not crash the system.

Happy we could help.

 

[Redcoat]

Good stuff! BTW, are you keeping track of Memory/Swap on the Reporting/Memory tab?

 

[s_falahati]

Good stuff! BTW, are you keeping track of Memory/Swap on the Reporting/Memory tab?

Yeah,
Even tho I panicked in the morning about the memory usage, it was and still is pretty stable through out the day. I just didn't know how ARC expected to work and use memory.

Currently, memory is almost full (300+MB free and some 500+MB dirty), but the swap is like 130M in use. It is the maximum, minimum and the average of swap usage in the past couple of hours. So nothing really happening in there.

But I figured the memory usage is probably because I am moving new files to the server from at least two locations, around 30MBps~50MBps in total (sometimes even peaking at 100MBps, I am not sure if it is real or Windows is making fun of my setup xD), therefore file system should be pretty busy and it is normal to have 12G~13G of ARC memory usage. However, as soon as I try to do something else, ARC frees some space for other processes. So everything is alright for now. I didn't even need the extra swap space apparently. But it is good to have it in place anyway. I feel better already.

 

[Chris Moore]

I didn't even need the extra swap space apparently. But it is good to have it in place anyway. I feel better already.

My server has 32GB of memory and will occasionally use some swap. When it does, it is usually less than 1GB.

 

[s_falahati]

Yeah, the swap usage is next to nothing; and I think it is normal.

I also like to thank FreeBSD and FreeNAS developers and contributors for this piece of software (even tho they are not going to see this post probably). I double checked, ZFS is actually capable of moving files with the incredible speed of around 120MBps from the network and it is the maximum BW of my LAN controller (1Gb/8), I am now thinking of aggregating the other LAN port too.
My Intel Raid on the main PC is only capable of 60~70MBps of write speed. However, to be honest, my Intel Raid setup only has 3 disks while this one has 4. But still incredible performance with little to no CPU usage (6~8%).