freenas-boot is degraded

[jobuel]

Hey guys,

my freenas-boot is degraded since today.

FreeNAS-11.2-u5
MB: Primez 370-P
RAM: 2x Kingston KVR26N19D8/16 (16GB 2Rx8 2G x 64-Bit PC4-2666 )
Root SSD: 1x WDC WDS120G2G0B-00EPW0 (WD Green 120GB SSD m.2)
HDD (Storage): 4x Seagate ST8000VX0022 SkyHawk 8 TB (8,89 cm (3,5 Zoll) 256 MB Cache, Sata 6 Gb/s)

The command zpool status -x fives me following answer (attached files fn1 & fn2):

There are even more corrupted files, but nothing special.
I visited https://illumos.org/msg/ZFS-8000-8A but i don't really understand what I have to do.

Unfortunately, the data cannot be repaired, and the only choice to repair the data is to restore the pool from backup. Applications attempting to access the corrupted data will get an error (EIO), and data may be permanently lost.

As I understand I have to restore a backup (i have a backup). But does this really solve the problem? Or are there another important things I have to watch out for?
Sorry I am a newbie who can't handle manuals.

I am very grateful for any help, if there are some questions about the issue: Please just ask me.

Kind regards,
jobu

 

[PhiloEpisteme]

Sorry I am a newbie who can't handle manuals.

The manuals can be a it overwhelming, but I find the FreeNAS User Guide to be a great place to look if for no other reason than what a lot of folks will advice you to do is exactly what the manual says in many cases.

In the case of a degraded root pool I imagine a reasonable solution would be to download a backup of your config (if you haven't already done so), get a new SSD and replace the one you have, reinstall FreeNAS, and upload your config. A couple of gotchas though; if you made use of encryption during your disk setup you need to take additional steps. Also, you may consider going with mirrored boot pools rather than a single pool.

I think it would be worth taking a look at the following section of the user guide System->General, Specifically the bits about backing up configs and the like.

 

[jobuel]

The manuals can be a it overwhelming, but I find the FreeNAS User Guide to be a great place to look if for no other reason than what a lot of folks will advice you to do is exactly what the manual says in many cases.

In the case of a degraded root pool I imagine a reasonable solution would be to download a backup of your config (if you haven't already done so), get a new SSD and replace the one you have, reinstall FreeNAS, and upload your config. A couple of gotchas though; if you made use of encryption during your disk setup you need to take additional steps. Also, you may consider going with mirrored boot pools rather than a single pool.

I think it would be worth taking a look at the following section of the user guide System->General, Specifically the bits about backing up configs and the like.

Did I understand you correctly: I just need to download a FreeNAS 11.2-u5 image, load it into a USB stick, replace the SSD with the new one and then fresh install? I already have a backup and I also saved my config.

 

[SweetAndLow]

Sorry I am a newbie who can't handle manuals

How did your install your os? When did you install it? It's setup incorrectly, should have gptid not the disk name. To fix your setup you just need to reinstall and upload your config.

Mod note: edited for tone -@jgreco

 

[jobuel]

That's just you being lazy not a newb. You need to read the manual. I bet after you try you will understand way more.

How did your install your os? When did you install it? It's setup incorrectly, should have gptid not the disk name. To fix your setup you just need to reinstall and upload your config.

I read the manual (https://www.ixsystems.com/documenta...tml?highlight=replace#replacing-a-failed-disk) a few times and took a look around in different threads but i found nothing I can work with.

Call me stupid, but sometimes I'm a little slow on the uptake.


But anyway, thanks for your help. I'll give it a try with your solution.

 

[PhiloEpisteme]

Reading the manual is a great step even for newbs. That being said, I can understand folks feeling overwhelmed or intimidated at first, especially if they are new enough to a field or area such that the terminology is dense and opaque.

 

[PhiloEpisteme]

Did I understand you correctly: I just need to download a FreeNAS 11.2-u5 image, load it into a USB stick, replace the SSD with the new one and then fresh install? I already have a backup and I also saved my config.

More or less, yes. The disk replacement procedures you posted above are generally for replacing disks in a vdev composed of more than 1 disk. In this case you have a single disk which is also your boot pool which requires replacing. For this reason you'll want to follow the steps outlined above.

Did you use encryption in your pool?

 

[HoneyBadger]

Root SSD: 1x WDC WDS120G2G0B-00EPW0 (WD Green 120GB SSD m.2)

This is related to an SSD controller bug. The only current fix is to disable TRIM prior to the install (via editing the GRUB line) and then ensuring that it stays disabled afterwards.

https://jira.ixsystems.com/browse/NAS-100276

 

[jgreco]

Folks, just a reminder to try to be polite to your fellow posters, especially people who may be trying to struggle through several different sets of unfamiliar all at once.

 

[jobuel]

1. Is there a risk to disable TRIM if it was enabled (picture enabled.PNG )?
The WD Green 120GB SSD m.2 would support TRIM (picture trim.PNG ).

2. I replaced the old SSD with the new one and did a fresh install. Then I imported the config file and the errors were gone. But just the next day they were back (picture freenas219.08.PNG ). Did I forgot something?

3. I read on the solution page for the error ZFS-8000-8A (https://illumos.org/msg/ZFS-8000-8A)
"If this is the case, the only option is to destroy the pool and re-create the datasets from backup. "
I'm a bit confused bacause I thought I just have to restore the config file? Or how can i destroy and re-create the datasets?

I'm sorry for my foolness but does anyone have a manual for dummies or some advice?

 

[HoneyBadger]

If you've already installed to a bugged device with TRIM enabled, the corruption will be there in the pool, and won't be detected until after a scrub or a read of the affected record.

The bug tracker hasn't quite nailed it down as to what the exact root cause is, but the current suspect is something wrong in the NCQ TRIM implementation.

 

[jobuel]

This is related to an SSD controller bug. The only current fix is to disable TRIM prior to the install (via editing the GRUB line) and then ensuring that it stays disabled afterwards.

https://jira.ixsystems.com/browse/NAS-100276

If you've already installed to a bugged device with TRIM enabled, the corruption will be there in the pool, and won't be detected until after a scrub or a read of the affected record.

The bug tracker hasn't quite nailed it down as to what the exact root cause is, but the current suspect is something wrong in the NCQ TRIM implementation.

Okay I'll do a fresh install in the next few days and deactivate trim via GRUB before booting.
I will give a feedback if it worked or not.

Thanks a lot for your help!

 

[sretalla]

Another alternative is to use a kingston or samsung SSD instead and re-purpose your WD green to another system.

 

[HoneyBadger]

use a kingston or samsung SSD instead

That's a funny way to spell "Intel" ;)

Joking aside, they have an extremely good track record in the datacenter space.

 

[sretalla]

That's a funny way to spell "Intel" ;)

I was recommending "cheap" SSDs for the boot pool here, not for data pools or SLOG. I get your point, but the collective wisdom is that boot drives can be whatever the cheapest SSDs are since they are mostly disposable in the FreeNAS setup. Clearly the ones with the TRIM problem need to be avoided. As far as I could see, it was either 3D NAND and/or a marvell controller in the chain somewhere there.

 

[jgreco]

That's a funny way to spell "Intel" ;)

Joking aside, they have an extremely good track record in the datacenter space.

If you buy the expensive datacenter grade ones, perhaps. I've managed to kill a bunch of Intel 535's and can tell you that they have a nasty failure mode of simply vanishing. Fortunately I do RAID1 almost everywhere.

 

[HoneyBadger]

If you buy the expensive datacenter grade ones, perhaps. I've managed to kill a bunch of Intel 535's and can tell you that they have a nasty failure mode of simply vanishing. Fortunately I do RAID1 almost everywhere.

Surplus Intel 320s and DC S3500s are pretty inexpensive, but I'll second your motion on the mirrors; if it's a mission-critical situation, you need redundancy in the boot device, regardless of what it is or how well it's worked. Any device can fail.

That's why we're all using redundant vdevs and have separate backups, right?

 

[jobuel]

This is related to an SSD controller bug. The only current fix is to disable TRIM prior to the install (via editing the GRUB line) and then ensuring that it stays disabled afterwards.

https://jira.ixsystems.com/browse/NAS-100276

Okay I'll do a fresh install in the next few days and deactivate trim via GRUB before booting.
I will give a feedback if it worked or not.

Thanks a lot for your help!

This was the solution which worked for me! It seems like it indeed was the trim function.
Now three days after the fresh install with disabled trim I don't get another errors!

I'm absolute happy! Thank you very much for your help guys! *thumb up*

 

[sretalla]

I'm absolute happy!

As long as you're not using other SSDs in your system, it should be fine, but that TRIM setting applies to all SSDs, so keep that in mind and make sure to revert back as soon as the fix arrives.

 

[jobuel]

Thanks for the information.
I just use a SSD for the boot environment. For storage I use HDDs.