Hard Drive Error?

[kspare]

This came up in our logs today. I've been using freenas for quite some time but never seen this error before!


Checking status of gmirror(8) devices:

Name Status Components

mirror/swap0 COMPLETE da23p1 (ACTIVE)

da22p1 (ACTIVE)

mirror/swap2 COMPLETE da19p1 (ACTIVE)

da18p1 (ACTIVE)

mirror/swap3 COMPLETE da17p1 (ACTIVE)

da16p1 (ACTIVE)

mirror/swap4 DEGRADED da14p1 (ACTIVE)

mirror/swap1 COMPLETE da21p1 (ACTIVE)

da20p1 (ACTIVE)



-- End of daily output --

 

[Chris Moore]

It would appear that one of your disks went offline long enough to break one of the swap mirrors. It usually only happens when there is some sort of hardware fault. I would guess, because of the other numbers, that it might be da15. If you give some more details about your hardware, we might be able to offer additional guidance, but I would start looking at the health of all my disks.

Here is a guide about what information is good to provide when asking questions:
https://forums.freenas.org/index.php?threads/updated-forum-rules-8-3-16.45124/

 

[Chris Moore]

PS. when sharing text like this, it is a good idea to put it inside code tags:

Code:

Checking status of gmirror(8) devices:

		Name	Status  Components

mirror/swap0  COMPLETE  da23p1 (ACTIVE)

						da22p1 (ACTIVE)

mirror/swap2  COMPLETE  da19p1 (ACTIVE)

						da18p1 (ACTIVE)

mirror/swap3  COMPLETE  da17p1 (ACTIVE)

						da16p1 (ACTIVE)

mirror/swap4  DEGRADED  da14p1 (ACTIVE)

mirror/swap1  COMPLETE  da21p1 (ACTIVE)

						da20p1 (ACTIVE)



-- End of daily output --

That way it looks much better and is easier to read, especially for text that is aligned into columns.

 

[kspare]

DA15 was a bad disk, and had been recently replaced, so it's now sitting as a cold spare in the server. We have 24 disks and do 11 mirrors, keeping 2 drives for cold spares. Everything shows healthy right now so i'm not really sure what this means?

 

[kspare]

I checked /var/log/messages and there is nothing about that disk having any errors. it's really very odd. I'm not even sure how to generate that log to make sure it's fixed!

 

[Chris Moore]

Everything shows healthy right now so i'm not really sure what this means?

If you remove a disk, FreeNAS does not automatically reconstruct the swap mirror on the new disk, it takes a reboot. The swap mirrors are not always going to use the same disks either. I have observed that FreeNAS will choose the disks in the order they report ready, so it can change from boot to boot somewhat randomly.
All the disks in the pool have swap partitions by default, but FreeNAS will only use ten of them in the swap mirrors that it dynamically creates during each boot.
Does that explain it?

 

[kspare]

I kind of wondered that. We used the replace disk function to place a new disk into the mirror. So we did that process, removed the bad drive and put in a new drive, which now becomes the cold spare.

I was unaware of these swaps. So nothing is really at risk data wise then.

Will the swap repair itself or do I really need to evacuate the server and just give it a reboot? Or can I just leave things alone?

 

[kspare]

Better yet, how do I move the swap to my two boot disks? I have 256gb of ram on these servers, I don't really need a swap.

 

[Chris Moore]

PS. You can list current status of active swaps with this command: gmirror status -ags
It should give you output like this:

Code:

root@freenas:~ # gmirror status -ags
	swap0  COMPLETE  ada2p1 (ACTIVE)
	swap0  COMPLETE  ada1p1 (ACTIVE)
swap0.sync	   N/A  N/A
	swap1  COMPLETE  da0p1 (ACTIVE)
	swap1  COMPLETE  ada0p1 (ACTIVE)
swap1.sync	   N/A  N/A

 

[kspare]

heres what I get.

Is it possible to move that swap somewhere else without a reboot? I have mirrored 120gb ssd's on all of these servers for boot volumes, i'd rather have it there if possible.

Code:

# gmirror status -ags
	 swap0  COMPLETE  da23p1 (ACTIVE)
	 swap0  COMPLETE  da22p1 (ACTIVE)
swap0.sync	   N/A  N/A
	 swap2  COMPLETE  da19p1 (ACTIVE)
	 swap2  COMPLETE  da18p1 (ACTIVE)
swap2.sync	   N/A  N/A
	 swap3  COMPLETE  da17p1 (ACTIVE)
	 swap3  COMPLETE  da16p1 (ACTIVE)
swap3.sync	   N/A  N/A
	 swap4  DEGRADED  da14p1 (ACTIVE)
swap4.sync	   N/A  N/A
	 swap1  COMPLETE  da21p1 (ACTIVE)
	 swap1  COMPLETE  da20p1 (ACTIVE)
swap1.sync	   N/A  N/A

 

[Chris Moore]

I was unaware of these swaps. So nothing is really at risk data wise then.

I don't think it would cause a problem unless you are using swap, which you are likely not doing. Are you?

Will the swap repair itself

In my experience, the swap will not automatically repair itself.

do I really need to evacuate the server and just give it a reboot? Or can I just leave things alone?

Probably just leave it, especially if you are not using any swap space.

Better yet, how do I move the swap to my two boot disks?

I don't necessarily advocate that. What kind of boot disks are you using? The thing I did with my most recent build was set the swap partition size to zero, so there was no swap space on any of the data drives. I put a NVMe SSD in a PCIe slot and created a swap partition there, just in case, and it has been working great for me. Now I always know where my swap is and don't need to concern myself with swap when changing data drives. It is something that would need to be done when the pool is initially built though.

 

[Chris Moore]

Is it possible to move that swap somewhere else without a reboot?

I don't know if you can do it without a reboot, but here is a link to some directions that were written by @Stux a little while back, it might give you some pointers:
https://forums.freenas.org/index.php?resources/how-to-relocate-swap-to-an-ssd-or-other-partition.68/

 

[kspare]

Each server has 2 mirroed 120gb or 240gb ssd drives, 24 4tb drives, 2 400gb ssd for slog and 1 1.2tb for L2ARC. Is it possible to put the swap partition on the boot drives and just set it to 0?

 

[Chris Moore]

PS. You may not be able to put it on the boot drives, but you might, if you do it carefully. Still, it will probably take a reboot.
It is probably the kind of thing to do with a new config before it goes into production.

 

[Chris Moore]

The problem is, you need to remove the swap partitions from all the data drives or FreeNAS is going to keep trying to create swap mirrors there.
The easy thing for you to do would be to use part of that L2ARC as swap and the rest for L2ARC. I did something similar on my recent build, but I got FreeNAS to not build swap partitions on the data drives first.

 

[kspare]

With my Slog drives, they are 400gb drives over provisioned to 100gb. Could I just carve out 5gb for a swap volume and make the swap file 0?

I have 4 storage servers, so I can migrate the data off and blow away the volume with zero risk. Ultimately I think it would be best to have it on the boot volume and not on any other devices so I don't see ANY IO on them except from my vm's.

 

[kspare]

i'll get the server evacuated and start working on this. thanks for the help Chris!

 

[Chris Moore]

Ultimately I think it would be best to have it on the boot volume and not on any other devices so I don't see ANY IO on them except from my vm's.

There was a resource written for that also.
https://forums.freenas.org/index.php?resources/how-to-relocate-swap-to-the-boot-drive.69/

Please note, both of those resources were written before changes to the swap subsystem in FreeNAS 11.1-U2, so you may find some differences in the behavior.
This one is also by @Stux and he might drop in here and add some comments since I have called him out. He uses his FreeNAS systems to host virtualization also.