SOLVED Limiting the scope of Swap Pool

NASbox · Jan 22, 2019

TL;DR SOLUTION:
The best way to prevent a device from being used as a swap device (and hence become difficult to unmount) is to delete the swap partitions from all the drives in the pool. For instructions on how to do this see:
https://forums.freenas.org/index.ph...g-swap-partition-is-it-safe-to-do-this.73197/

TL;DR
I am trying to control which drives are used for swap space.

Most importantly I want to prevent ada3 from being used for swap (and would prefer to exclude ada0/1 as well).

ada3 is a single disk removable pool that is mounted/unmounted for backups.
When it becomes part of the swap space it can cause problems (hard to unmount/system instability).

Additional Info:
My thinking is that the best situation is to restrict swap to da0-7 (Main pool always spinning-redundant)
or
QUESTION: Would it be a good idea to force the swap onto the boot drive ada2 (SSD)? (If so how?)

I would not normally expect much swapping, but if a small bit of swapping was necessary, then either of these choices would be best.

System Configuration:
FreeNAS 11.1-U6 / 32GB
ada2 - Boot Drive 120GB SSD
da0-da7: Main Pool - 8 WD Red Drives [Avago HBA]
ada0-ada1 - 2 Old Hitachi Drives
ada3 - Removable mount - used for single drive pools (different pool names for different drives - imported/exported by bash script(s).

Additional background:
At the moment only ada0/ada1 seem to be part of the swap space.

#>swapinfo
Device 1K-blocks Used Avail Capacity
/dev/mirror/swap0.eli 2097152 0 2097152 0%
/dev/mirror/swap1.eli 2097152 0 2097152 0%
/dev/mirror/swap2.eli 2097152 0 2097152 0%
/dev/mirror/swap3.eli 2097152 0 2097152 0%
/dev/mirror/swap4.eli 2097152 0 2097152 0%
Total 10485760 0 10485760 0%

#>gmirror status
Name Status Components
mirror/swap0 COMPLETE ada1p1 (ACTIVE)
ada0p1 (ACTIVE)
mirror/swap1 COMPLETE da7p1 (ACTIVE)
da6p1 (ACTIVE)
mirror/swap2 COMPLETE da5p1 (ACTIVE)
da4p1 (ACTIVE)
mirror/swap3 COMPLETE da3p1 (ACTIVE)
da2p1 (ACTIVE)
mirror/swap4 COMPLETE da1p1 (ACTIVE)
da0p1 (ACTIVE)

Some how when I was doing a backup a /dev/mirror/swap5.eli got created-I don't know what I did, but I suspect I may have had ada3 mounted over a reboot. I ended up with a weird configuration ata2 (boot) and ada3(removable) broke up the clean arrangement and scattered themselves though the mix. The result was the system became a bit unstable ( and for some reason-don't know why as there was nothing significant running) htop showed 1+GB of swap in use. Very weird.

I just mounted ada3 (Single disk zpool in a removable carrier used for backup), and so far it isn't part of the swap.

Chris Moore · Jan 22, 2019

Here is another thread that discusses the kind of thing you want to do:
https://forums.freenas.org/index.php?threads/cannot-eject-rdx-removable-device.68297/post-467696

I think it will answer your question, but if you still have questions, please post back here.

Chris Moore · Jan 22, 2019

PS. Here is some information on manually partitioning disks, but the simple solution is in the first thread I pointed you at:
https://forums.freenas.org/index.ph...s-partitioned-for-two-pools.62787/post-483761

NASbox · Jan 23, 2019

Hi Chris - Thanks for the reply - very helpful!

Chris Moore said:
PS. Here is some information on manually partitioning disks, but the simple solution is in the first thread I pointed you at:
https://forums.freenas.org/index.ph...s-partitioned-for-two-pools.62787/post-483761

Sorry, but this one went over my head - I didn't understand/make the connection to my situation.

Chris Moore said:
Here is another thread that discusses the kind of thing you want to do:
https://forums.freenas.org/index.php?threads/cannot-eject-rdx-removable-device.68297/post-467696

I think it will answer your question, but if you still have questions, please post back here.

IIUC the "short answer" is to nuke the swap partition. ???

Given this drive, is there a way I can do that without losing the data on ada3p2?

Code:

#>gpart show -p ada3
=>         40  19532873648    ada3  GPT  (9.1T)
           40           88          - free -  (44K)
          128      4194304  ada3p1  freebsd-swap  (2.0G)
      4194432  19528679248  ada3p2  freebsd-zfs  (9.1T)
  19532873680            8          - free -  (4.0K)

My situation is a bit different than the one you were dealing with. I assume an RDX manages the drive removal to make sure that a connected drive doesn't get removed incorrectly. In my case I have a manual system that would allow me to physically remove the drive even when it was being written (which would not be good), so I need to understand what I am doing and be mindful when I am doing it.

Just to make absolutely sure that I understand the implications of what I am doing I have a couple of follow up questions:

Question #1: Removable Drive/Pool (on ada3)

To remove my backup pool I currently do:

zpool export POOLNAME

when I am finished backing up. Assuming the export is successful, do I also need to do:

camcontrol eject ada3 -v

as well?

Currently I have just been removing the drive. It hasn't done any damage to anything, and I get an entry in dmesg about Periph destroyed:

Jan 22 15:17:02 freenas ada3: <WDC WD100EFAX-68LHPN0 83.H0A83> s/n XXXXXXXX detached
Jan 22 15:17:02 freenas (ada3:ahcich5:0:0:0): Periph destroyed

Question #2: Stopping a swap device

Given this current situation:

Code:

#>gmirror status -ags
     swap0  COMPLETE  ada1p1 (ACTIVE)
     swap0  COMPLETE  ada0p1 (ACTIVE)
swap0.sync       N/A  N/A
     swap1  COMPLETE  da7p1 (ACTIVE)
     swap1  COMPLETE  da6p1 (ACTIVE)
swap1.sync       N/A  N/A
     swap2  COMPLETE  da5p1 (ACTIVE)
     swap2  COMPLETE  da4p1 (ACTIVE)
swap2.sync       N/A  N/A
     swap3  COMPLETE  da3p1 (ACTIVE)
     swap3  COMPLETE  da2p1 (ACTIVE)
swap3.sync       N/A  N/A
     swap4  COMPLETE  da1p1 (ACTIVE)
     swap4  COMPLETE  da0p1 (ACTIVE)
swap4.sync       N/A  N/A

a) Am I correct that this command would remove ada0/ada1 from the swap pool?

gmirror stop -fv swap0

b) Is there anything I need to worry about if I do this? Is there any problem with just removing disks from the swap pool?

c) If there were swap 0-5 in operation, would it matter if I had to stop 2 or 3 of the mirrors? (I would still have 2 or 3 left)

d) If there is any swap in use, will the system automatically move the data or will the system crash?

Question #3 - Removal of ada0/ada1

To remove ada0/ada1 from the swap I assume I need to nuke the swap partitions - Correct?
If so, hopefully I can infer how to do that from the answer above.

Question #4 - Boot Drive

Given that the boot drive is an Odd drive should I consider removing the swap partition from it?

Thanks again Chris, very helpful.

hansch · Jan 23, 2019

Interesting thread. It would be nice if the interface provided options to include/exclude disks from being used as swap devices. I don't want swap on my boot USB disk for example. [Edit: It isn't, but 2 USB disks are.] They are connected for some tests and they are automatically being used as swap drives. I think it is bad for performance and stability to include those. But someting far worse happens:]

After 5 days I got loads of this error:

Jan 23 09:09:28 freenas swap_pager_getswapspace(32): failed

After rebooting, these start to appear after 15 minutes.
So something is going wrong.

How can I find out which device is causing this?
And how do I disable that disk's swap space (preferably without rebooting)?
So I am not talking about the initial setting up. The system is running, I want to reconfigure as little as possible.

EDIT:
I did that:

gmirror stop -fv swap0

(Oddly enough, swap0 is on disks 8 and 9 which are USB disksk.)

Jan 23 12:22:08 freenas GEOM_MIRROR: Device swap0: provider destroyed.
Jan 23 12:22:08 freenas GEOM_MIRROR: Device swap0 destroyed.
Jan 23 12:22:08 freenas GEOM_ELI: Device mirror/swap0.eli destroyed.
Jan 23 12:22:08 freenas GEOM_ELI: Detached mirror/swap0.eli on last close.
Jan 23 12:30:48 freenas swap_pager_getswapspace(32): failed
Jan 23 12:30:48 freenas swap_pager_getswapspace(32): failed

So... it did not help.
I do not think the other disks are bad because they are good quality, new SATA NAS disks. No SMART errors, volume is healthy.

So... how can I find out what is happening?

[If I have to start another thread please tell me.)

Update:
I removed the USB disks from the system after temporarily placing the systemdata on the boot drive (USB flash drive). Then I attached a 64 GB SATA SSD and put the systemdata there.

The error comes back very soon.

The system has 24 GB of RAM, and a pool of 8 x 8 TB disks (Z2).
I use it for storing backups which are written through the minio service.

When the error occurs, the web interface and minio are not responsive any more, but ssh is.

#top -w

last pid: 5654; load averages: 1.11, 1.14, 1.09 up 0+00:42:05 18:53:31
61 processes: 1 running, 54 sleeping, 1 zombie, 5 waiting
CPU: 0.2% user, 0.0% nice, 12.8% system, 0.0% interrupt, 86.9% idle
Mem: 20G Active, 68K Inact, 2611M Wired, 98M Free
ARC: 1622M Total, 1366M MFU, 215M MRU, 96K Anon, 21M Header, 18M Other
893M Compressed, 1589M Uncompressed, 1.78:1 Ratio
Swap: 8192M Total, 8192M Used, K Free, 100% Inuse

PID USERNAME THR PRI NICE SIZE RES SWAP STATE C TIME WCPU COMMAND
226 root 33 20 0 230M 105M 0K uwait 4 0:36 1.54% python3.6
3012 root 22 20 0 152M 24776K 0K umtxn 3 0:11 0.53% collectd
2786 minio 63 52 0 28458M 20820M 0K uwait 2 4:45 0.15% minio

Minio is using a whopping 28 G of memory!
BTW.... why is only 8 GB of swap configured? (It has 8 disks of each 8 TB.)

After killing the minio process the system becomes responsive again.

Chris Moore · Jan 23, 2019

Something to understand about FreeNAS. It will always use the first ten disks that come online, out of what ever number of disks that are connected, to be the swap mirrors. Which disks are used can change from one boot to the next.
I have a server at work that has 122 disks connected. By default, FreeNAS would take 2GB from each of those disks to be a swap partition, but it only used five mirrors (ten disks) so there were 112 x 2GB partitions that were not being used. I found that pretty wasteful, so I changed this setting (in the legacy UI):

The default is 2, but I changed it to 0. It does require re-creating the storage pool, which re-partitions the drives, but then there are no swap partitions for FreeNAS to randomly pick from. I then installed some SSD drives and manually created swap partitions on them and now FreeNAS has swap space on SSDs that don't affect my storage pool.

The point is, you need to not have swap partitions on drives that don't need swap partitions, but if you don't change this setting, FreeNAS will create swap by default on every drive you connect to the system.

Chris Moore · Jan 23, 2019

hansch said:
[If I have to start another thread please tell me.)

It is usually best to start a new thread.
Your problem is similar in nature, but the underlying cause is different. When you installed USB memory sticks and created a pool on them, FreeNAS created swap space on them because that is the default behavior of FreeNAS. New disks automatically get swap space. FreeNAS does not know or care what kind of disk you have connected. That is for you, the administrator, to know and handle accordingly.
The thing you should have done, if you didn't want the swap space on those drives, is disable swap space creation by the method in my previous post in this thread. Then install the disks and create the pool. There would have been no swap space for FreeNAS to use. That would have solved part of the problem.
The other part of the problem is, the system dataset is constantly being written to and that will wear out any USB memory stick in a short time, and a low quality stick will die even quicker. This problem is only slightly mitigated by using a mirrored pair because both sticks are being used the same amount, they can both fail at the same time. That is undoubtedly what happened to you. The SSD will last better.

hansch · Jan 23, 2019

Each time I have problems like this - and it has been a problematic thing to set up Freenas and minio - I just wonder why I get these problems, and not anybody else since the last beta testing (and before 11.2 (U1) being released.

I have seen the bug where FreeNAS does not write the S3 config files (how can you not com accross that? It happens every time.
I have seen this bug where Minio is using up all RAM. (I know, this is middleware, but it is bundled with FreeNAS. (Even the name is made generic ("S3) as if to hide the underlying technology.)

BTW it is reproduced here:
https://github.com/minio/minio/issues/6490
In that case, it was solved by sharing a disk (letter), not a directory through Minio.
I don't know if it will make a difference to give minio my data pool and not just a directory in it.

My conclusion can only be that with FreeNAS (and minio) you must be prepared to solve many, many problems if you want to get even the most basic system running well. All I want is a steady platform to store and host files. I trusted to have found it in FreeNAS and minio, but I was mistaken.

I have been using open source software since 2000 but I have never seen anything this buggy "released".
I'm really sorry to say this. That said, I have invested a lot of time and money in this.
Many people seem to be able to get their systems running (and running reliably) so I'm not giving up yet. I really appreciate all the work you guys do...
I will start a new thread.

Chris Moore · Jan 23, 2019

NASbox said:
Question #4 - Boot Drive

Given that the boot drive is an Odd drive should I consider removing the swap partition from it?

The boot drive should not have a swap partition.

NASbox said:
Question #3 - Removal of ada0/ada1

To remove ada0/ada1 from the swap I assume I need to nuke the swap partitions - Correct?
If so, hopefully I can infer how to do that from the answer above.

FreeNAS picks the first (up to ten) swap partitions that become available to make the swap mirrors. If you do not want a drive to participate in swap, you must not have a swap partition on the drive. I have not tried to delete a swap partition. In the testing I did, I completely cleared the drive, set the swap partition size to zero in the GUI and then allowed FreeNAS to partition the drive. This results in a drive with no swap partition.

NASbox said:
Question #1: Removable Drive/Pool (on ada3)

To remove my backup pool I currently do:

zpool export POOLNAME

when I am finished backing up. Assuming the export is successful, do I also need to do:

camcontrol eject ada3 -v

as well?

You should not need to do the eject command. The reason for that is the RDX media that was being used in the other thread. It has a powered (motorized) eject mechanism. The eject command tells the drive to run the ejection motor, and it will only complete successfully if the drive is not being used, which is why the swap needed to be disabled.

NASbox said:
Currently I have just been removing the drive. It hasn't done any damage to anything, and I get an entry in dmesg about Periph destroyed:

Jan 22 15:17:02 freenas ada3: <WDC WD100EFAX-68LHPN0 83.H0A83> s/n XXXXXXXX detached
Jan 22 15:17:02 freenas (ada3:ahcich5:0:0:0): Periph destroyed

The way you are doing it should be fine, as long as the drive either has no swap partition or the swap mirror has been stopped.

NASbox said:

Yes. That command looks to be correctly formatted.

NASbox said:
b) Is there anything I need to worry about if I do this? Is there any problem with just removing disks from the swap pool?

I don't think it will cause a problem. It will reduce the amount of swap space available.

NASbox said:
c) If there were swap 0-5 in operation, would it matter if I had to stop 2 or 3 of the mirrors? (I would still have 2 or 3 left)

I actually ran a FreeNAS server at work for almost two months with no swap space at all and it didn't give me any trouble, but that server had 256GB of RAM. The odd thing, even with that much RAM, once I gave it swap space, it started using it. I can't make promises. These are just my observations from my systems at home and the ones I manage for work.

NASbox said:
d) If there is any swap in use, will the system automatically move the data or will the system crash?

It is probably best to flush the swap before you remove it, as a precaution against creating a crash.

You might find some additional useful information in this thread:

How to relocate swap to an SSD or other partition - [not needed in v 11.1 and newer systems]
https://forums.freenas.org/index.php?resources/how-to-relocate-swap-to-an-ssd-or-other-partition.68/

Chris Moore · Jan 23, 2019

hansch said:
My conclusion can only be that with FreeNAS (and minio) you must be prepared to solve many, many problems if you want to get even the most basic system running well. All I want is a steady platform to store and host files. I trusted to have found it in FreeNAS and minio, but I was mistaken.

Please do not blame the faults of minio on FreeNAS. The minio software is a separate project and there is no guarantee that any additional software will work without significant configuration.

hansch · Jan 23, 2019

Chris Moore said:
Please do not blame the faults of minio on FreeNAS. The minio software is a separate project and there is no guarantee that any additional software will work without significant configuration.

I disagree. It is reasonable to expect that any software that is released is for the most part free of bugs.
But then, I am a user (and involuntary beta tester). I expect things to work if they are released. Minio is bundled, so it should work at least reasonably well with defaults. In the 11.2 U1 release, it cannot be installed, and it will crash when some data is actually stored in it. Even worse, it will almost entirely crash the system. It should not even be allowed to do that imho, how buggy it might be.

It seems that no one in the developers team even tried to test installing it, and putting some data in it.
But I know I am out of line, I'm just pissed. I am glad my zpool is (still) error free.

Chris Moore · Jan 23, 2019

hansch said:
I disagree. It is reasonable to expect that any software that is released is for the most part free of bugs.

Bugs in minio are not the fault of FreeNAS.

hansch · Jan 23, 2019

As it is bundled, and because of the Web GUI cannot easily be replaced {not by me at least} with a self compiled version, they are.
But... can we continue in a new thread? :)

Chris Moore · Jan 23, 2019

Probably best to start your own thread. This is getting completely out of the arena of the original post here.

hansch said:
It seems that no one in the developers team even tried to test installing it, and putting some data in it.
But I know I am out of line, I'm just pissed. I am glad my zpool is (still) error free.

Have you looked on redmine to see if there is a bug report. It might be that one needs to be submitted. They can't fix it if they don't know it is broken.
https://redmine.ixsystems.com/projects/freenas

hansch · Jan 23, 2019

The other bug (minio config files not being written by the gui) was found and reported by someone else just one day before I reported the same bug... I did not find it using their search and Google. They say it is solved in the next release/update.

Bug report here:
https://redmine.ixsystems.com/issues/71875
Seems not to be high priority ;)

NASbox · Jan 26, 2019

For the benefit of anyone following:
Details on how to delete the swap partitions from a pool see:
https://forums.freenas.org/index.ph...g-swap-partition-is-it-safe-to-do-this.73197/

Important Announcement for the TrueNAS Community.

SOLVED Limiting the scope of Swap Pool

NASbox

Guru

Chris Moore

Hall of Famer

Chris Moore

Hall of Famer

NASbox

Guru

hansch

Explorer

Chris Moore

Hall of Famer

Chris Moore

Hall of Famer

hansch

Explorer

Chris Moore

Hall of Famer

Chris Moore

Hall of Famer

hansch

Explorer

Chris Moore

Hall of Famer

hansch

Explorer

Chris Moore

Hall of Famer

hansch

Explorer

NASbox

Guru

Similar threads