How to relocate swap to the boot drive

[tortue]

I noticed the release notes for 11.1 showed 'Support for mirrored swap partitions'. Does this occur automatically (and is there way to verify/determine) or is this something that has to be configured through the gui/cli?
I did not find a reference in the manual, and I'm curious what 'support' means in this case. Does it support it as in if we configure it it will work or does it configure it that way automatically. I'm a few days away from setting up my pools and was planning on moving the swap to the boot drive as I am not fond of the idea of the system crashing when a drive goes offline.

 

[SweetAndLow]

Swap is configured every time on boot. It happens automatically. I had to test it to figure out what happens. Didn't find any documentation on the topic.

 

[Stux]

With 11.1, I assume this is a bit moot? I have this set up, but I am thinking I may want to revert it.

Thoughts?

Perhaps. It certainly should remove some of the reason for doing it.

But I haven't had time to investigate 11.1 yet.

It should be a simple matter to disable the startup command now, as I believe they added enable/disable feature to the initi commands list :)

 

[MrToddsFriends]

Does this occur automatically (and is there way to verify/determine) or is this something that has to be configured through the gui/cli?

Mirrored swap is configured automatically in 11.1 as SweetAndLow already wrote (I'm using -U1) and can be checked for in the CLI (note the "mirror" as a part of the device names):

Code:

~ # swapinfo
Device		  1K-blocks	 Used	Avail Capacity
/dev/mirror/swap0.eli   2097152		0  2097152	 0%
/dev/mirror/swap1.eli   2097152		0  2097152	 0%
/dev/mirror/swap2.eli   2097152		0  2097152	 0%
Total			 6291456		0  6291456	 0%

Don't know a way to check that in the FreeNAS GUI, other than reviewing the total swap space size in Reporting -> Memory.

 

[svtkobra7]

Can anyone confirm if this is still relevant?

• To be honest, my knowledge is a bit lacking as to what FreeNAS (11.2 U1) is currently doing with swap.
• And I implemented the guidance in this resource "blindly" as @Stux really knows his stuff, but wonder if it is even needed anymore.
• I never see swap usage (maybe because 200 GB of RAM per host?) and don't encounter any issues if I perform a clean install and accidentally forget to implement the post init script.
• Of note, whether right or wrong, the guide I used to initially configure FreeNAS back in Apr '17 suggested System > Advanced > "Swap size on each drive in GiB" = 0, and I've held to applying that setting since, never using swap on my pools.

[As I'm trying to troubleshoot a couple of other items ATM, I figured I'd post the query to rule this out, and also since I've never really understood this, but would like to - thanks in advance for anyone who cares to educate me]

Code:

root@FreeNAS-01[~]# swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/md99         4194304        0  4194304     0%

 

[Ericloewe]

To be honest, my knowledge is a bit lacking as to what FreeNAS (11.2 U1) is currently doing with swap.

FreeNAS 11.1 and newer group swap partitions in GEOM mirror pairs for reliability. I'm not sure how disk replacements are handled in 11.2, but I think 11.1 ignored that part of the problem (until you rebooted).

So, a disk failure should no longer crash your system if it's under memory pressure.

Of note, whether right or wrong, the guide I used to initially configure FreeNAS back in Apr '17 suggested System > Advanced > "Swap size on each drive in GiB" = 0, and I've held to applying that setting since, never using swap on my pools.

That you probably don't want to do. Although 200 GB of RAM is a lot...

 

[svtkobra7]

Thanks so much for shedding some light on this for me (its kept me up at night ;)).

FreeNAS 11.1 and newer group swap partitions in GEOM mirror pairs for reliability. I'm not sure how disk replacements are handled in 11.2, but I think 11.1 ignored that part of the problem (until you rebooted).

• I'm only seeing 1 swap device, not the mirror you mentioned, for what reason? (again, just seeking to understand)

May I ask why the suggest for not leaving swap per drive @ 0? Not questioning your correctness, just curious. My uncalibrated suggestion would be I'm not sure it matters if I've been running so long without any issue @ 0 + I don't encounter memory pressure + you noted a remedy in 11.1 to alleviate a crash.

In summary, to recap your post, would your guidance be to revert Advanced > "Swap size on each drive in GiB" = 0 to the default amount of 2 GB? And leave the script "active" ??

Although 200 GB of RAM is a lot...

So yeah about that ... well a wise man published the following comments somewhere and I put some effort into ensuring other components of my two hosts were up to snuff, so figured I could spring for 16 x 16GB x 2 hosts (not sure I would be able to still afford to eat if DDR4 though).

• ZFS and, by extension, FreeNAS require a lot of RAM. A rule of thumb for RAM sizing is 1GB per 1TB of storage. This rule is left deliberately vague as it is only a rule of thumb.
• Basically, the community’s approach is “If you’re going to do it, do it right”.

(maybe I took the comments too literally?? ;))

Now if only I could figure out why an SEL entry of "ECC Correctable Error" makes ESXi unresponive. My layperson suggestion would be that based on the phrasing the error was correctable and the ECC should have saved from a crash. EDIT: Sorry for the tangent, and I swapped in a spare DIMM, just confused by what is really meant by the SEL log entry.

 

[Ericloewe]

I'm only seeing 1 swap device, not the mirror you mentioned, for what reason? (again, just seeking to understand)

The device is the mirror. GEOM works on a layer below.
Wait a minute, why is swap on a RAMdisk? What are you doing on that system?

 

[svtkobra7]

The device is the mirror. GEOM works on a layer below.
Wait a minute, why is swap on a RAMdisk?

• Is it? How can you tell.
• Shamefully I had no clue that was the case, I simply followed the instructions in this resource ...
• ... which I thought put it on the boot disk.

Code:

            dd if=/dev/zero of=/usr/swap0 bs=1m count=4096
            chmod 0600 /usr/swap0
           
swapoff -a ; grep -v -E 'none[[:blank:]]+swap[[:blank:]]' /etc/fstab > /etc/fstab.new && echo "md99 none swap sw,file=/usr/swap0,late 0 0" >> /etc/fstab.new && mv /etc/fstab.new /etc/fstab ; swapon -aL

What are you doing on that system?

• Its a node in an ESXi cluster. ATM, trying to figure out how to get replication speeds up to snuff and woefully underutilized until that is resolved.
• (is that what you are asking?)

 

[Ericloewe]

Is it? How can you tell.

The md driver is the RAMdisk driver.

Well, that series of commands turns off swap, adds an md device to fstab and turns swap on again. I guess the idea is to force it to stay in RAM, but this way it's going to drive halfway to China to end up back in the same spot.

At this point, I recommend you drop that command. If you end up with swap usage, limit the ARC to a little less than it typically is when memory pressure starts to happen.

 

[Spearfoot]

I've been using the same swap scheme as @svtkobra7 for going on four years now, with no problems. I've always set the disk swap size to zero, and run a post-init script to create a swap file on the FreeNAS boot drive per the FreeBSD instructions here:

https://www.freebsd.org/doc/handbook/adding-swap-space.html

I use 8GB or 16GB swap files, but I seldom see swap usage go above 1GB.

Note that users booting from a USB drive should most certainly not use this approach because of excessive wear on the flash drive. But it works fine if you're booting from an SSD or hard drive.

 

[Stux]

Well, that series of commands turns off swap, adds an md device to fstab and turns swap on again. I guess the idea is to force it to stay in RAM, but this way it's going to drive halfway to China to end up back in the same spot.

It’s actually the suggested way to have a file backed swap from the FreeBSD manual.

I have not confirmed anything about swap with 11.2.

 

[Stux]

I expect the technique is still valid, and if you want boot hosted swap rather than hd pool, this is still the best way to do it.

Leave 2GB swap enabled in the FreeNAS prefs means that 2GB of space is reserved on your HDs. Once upon a time this gave you flexibility if a replacement hd was slightly smaller than the original. I’m not so sure this happens these days though.

 

[svtkobra7]

I expect the technique is still valid, and if you want boot hosted swap rather than hd pool, this is still the best way to do it.

So I'm using two 40G vDisks partitioned to 20G as SLOGs [1] ... would the better play be to take some of that 2 x 20G "over provisioning" to use as swap? I suppose it is a question that isn't worth the time to even type considering my system's use of swap (none). I know you posted a how to with the P3700 in your thread, but for some reason I ran into an issue when attempting to use your approach (can't recall the issue atm - it was some time ago)..

[1] from Optane 900ps which I wasn't able to use the passthru.map workaround to hand to FreeNAS directly ... not sure why, possibly because that issue has something to do with PCI slot #s and I have 2 per host.

 

[Ender117]

I expect the technique is still valid, and if you want boot hosted swap rather than hd pool, this is still the best way to do it.

Leave 2GB swap enabled in the FreeNAS prefs means that 2GB of space is reserved on your HDs. Once upon a time this gave you flexibility if a replacement hd was slightly smaller than the original. I’m not so sure this happens these days though.

On 11.2 swapoff -a seems to no longer turn off the swap on HDD, and they do not show up in /etc/fstab either. Not sure why and how but this script now only add a file-backed md swap next to the default swap

Code:

root@freenas:~ # swapoff -a ; grep -v -E 'none[[:blank:]]+swap[[:blank:]]' /etc/fstab > /etc/fstab.new && echo "md99 none swap sw,file=/usr/swap0,late 0 0" >> /etc/fstab.new && mv /etc/fstab.new /etc/fstab ; swapon -aL
swapoff: md99 on /usr/swap0: Device not found
swapon: adding /dev/md99 as swap device
root@freenas:~ # swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/mirror/swap4.eli   2097152        0  2097152     0%
/dev/mirror/swap3.eli   2097152        0  2097152     0%
/dev/mirror/swap2.eli   2097152        0  2097152     0%
/dev/mirror/swap1.eli   2097152        0  2097152     0%
/dev/mirror/swap0.eli   2097152        0  2097152     0%
/dev/md99         4194304        0  4194304     0%
Total            14680064        0 14680064     0%
root@freenas:~ # cat /etc/fstab
fdescfs /dev/fd fdescfs rw      0 0
md99 none swap sw,file=/usr/swap0,late 0 0
root@freenas:~ # swapoff -a
swapoff: removing /dev/md99 as swap device
root@freenas:~ # swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/mirror/swap4.eli   2097152        0  2097152     0%
/dev/mirror/swap3.eli   2097152        0  2097152     0%
/dev/mirror/swap2.eli   2097152        0  2097152     0%
/dev/mirror/swap1.eli   2097152        0  2097152     0%
/dev/mirror/swap0.eli   2097152        0  2097152     0%
Total            10485760        0 10485760     0%
root@freenas:~ # cat /etc/fstab
fdescfs /dev/fd fdescfs rw      0 0
md99 none swap sw,file=/usr/swap0,late 0 0