Hello everyone!
My disks got degraded.
I'm running TrueNAS under Proxmox. I have 2 disks in the pool - zfs stripe. I know this setup is discouraged, but this is what I decided to go with before I have more drives to create proper redundancy.
Here is what happened. I had 2 instances of TrueNAS installed. The pool was detached from one of the instances and attached to the other. Also Only one was running at the end, and the first one was shut down.
All was running fine. At some point the whole Proxmox system got rebooted and both instances was running (which I didn't notice for some time).
One day I needed to access the SMB from my PC and I logged into Truenas.
It appeared that the pool wasn't attached (at that point I didn't know that I have 2 instances running and that I'm logged into the one that was supposed to be shutdown). So I attached the pool and enabled SMB.
All was fine for a bit. Later that day I noticed that I have 2 instances of TrueNAS running and tried shutdown the one I didn't need. At some point during this process 1 drive showed status DEGRADED. And later the 2nd one also got degraded.
I probably also need to mention that I detach the pool from one of the instances
Is there way to restore at least some of the data from those drives?
Any help or advise on what can be done is greatly appreciated!
Thank you!
This is what zpool status -v says:
My disks got degraded.
I'm running TrueNAS under Proxmox. I have 2 disks in the pool - zfs stripe. I know this setup is discouraged, but this is what I decided to go with before I have more drives to create proper redundancy.
Here is what happened. I had 2 instances of TrueNAS installed. The pool was detached from one of the instances and attached to the other. Also Only one was running at the end, and the first one was shut down.
All was running fine. At some point the whole Proxmox system got rebooted and both instances was running (which I didn't notice for some time).
One day I needed to access the SMB from my PC and I logged into Truenas.
It appeared that the pool wasn't attached (at that point I didn't know that I have 2 instances running and that I'm logged into the one that was supposed to be shutdown). So I attached the pool and enabled SMB.
All was fine for a bit. Later that day I noticed that I have 2 instances of TrueNAS running and tried shutdown the one I didn't need. At some point during this process 1 drive showed status DEGRADED. And later the 2nd one also got degraded.
I probably also need to mention that I detach the pool from one of the instances
Is there way to restore at least some of the data from those drives?
Any help or advise on what can be done is greatly appreciated!
Thank you!
This is what zpool status -v says:
Code:
zpool status -v
pool: NAS
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
scan: scrub repaired 276K in 04:06:02 with 213 errors on Mon May 8 05:18:25 2023
config:
NAME STATE READ WRITE CKSUM
NAS DEGRADED 0 0 0
a548546e-b11b-48ce-8683-6a1e7a626358 DEGRADED 0 0 132 too many errors
a10ae8c7-7298-40ce-9d41-a8a1d5dcf1db DEGRADED 0 0 132 too many errors
errors: Permanent errors have been detected in the following files:
NAS/ix-applications/releases/nextcloud/volumes/pvc-1e1b3281-f845-48dc-abe7-155e1dce36d5:<0x0>
NAS/ix-applications/releases/nextcloud/volumes/pvc-287f77af-4fb9-49a7-a9dc-f3511effe4d0:/data/global/pg_control
NAS/ix-applications/releases/nextcloud/volumes/pvc-287f77af-4fb9-49a7-a9dc-f3511effe4d0:/data/pg_wal/000000010000000000000087
NAS/ix-applications/releases/nextcloud/volumes/pvc-287f77af-4fb9-49a7-a9dc-f3511effe4d0:/data/postmaster.opts
/var/db/system/update/update.sqsh
/mnt/NAS/ix-applications/k3s/server/db/state.db
/mnt/NAS/ix-applications/k3s/agent/cri-dockerd/sandbox/88ebfb0be7779ccb335b3965fdaed77ff919a37d99727d477458e1e2b3ffbbf1
/mnt/NAS/ix-applications/k3s/agent/cri-dockerd/sandbox/c5347baf34de279423a87edd1fce00fc22b243b955425231cde80c6a0a767506
/mnt/NAS/ix-applications/k3s/agent/cri-dockerd/sandbox/22d1704b97827302ffe7ac549ea4c4f9e18eea3f8575a95f9e9e32ddad11e8c8
/mnt/NAS/ix-applications/k3s/agent/cri-dockerd/sandbox/f6721c0e5de1fe3459c42b69d1f7d79dbd4a2fd98eb38d201e922673cd604174
/mnt/NAS/ix-applications/k3s/agent/cri-dockerd/sandbox/d96cfeaa930ba9718513b48c33f345675871d18841504a467d8f103583eef6ab
/mnt/NAS/ix-applications/k3s/agent/cri-dockerd/sandbox/795b33ed1e7be38339b2e35103680b90593e7f82170409843c9a5517803f65ff
/mnt/NAS/ix-applications/k3s/agent/cri-dockerd/sandbox/d1b1b262dea487a33dcbe0d60cafcb35666d7b56c9a82b74324b300007bf5e71
/mnt/NAS/ix-applications/k3s/agent/cri-dockerd/sandbox/bb56285e58440339b017ea861184447a423c1207f8d03c8175dd4c2965070092
/mnt/NAS/ix-applications/k3s/agent/cri-dockerd/sandbox/9bc7ba3cf4e2c28982dcf328f948f96972d07d08643e05fe9a746258812fa38a
/mnt/NAS/ix-applications/k3s/agent/cri-dockerd/sandbox/6b2b0f820be74bec2e6f94d184fd0dd5d55d6d2b83dbacbf89da19f439afcf39
NAS/.system/syslog-5a0a2a47cd884dbcbe527966286bfc29:<0x0>
/var/db/system/syslog-5a0a2a47cd884dbcbe527966286bfc29/log/containerd.log.1
/var/db/system/syslog-5a0a2a47cd884dbcbe527966286bfc29/log/kube_router.log
NAS/.system/syslog-5a0a2a47cd884dbcbe527966286bfc29:<0xfb>
NAS/.system/syslog-5a0a2a47cd884dbcbe527966286bfc29:<0xff>
/var/db/system/configs-5a0a2a47cd884dbcbe527966286bfc29/TrueNAS-SCALE-22.12.0/20230421.db
NAS/ix-applications/k3s/kubelet:<0x0>
NAS/ix-applications/k3s/kubelet:<0x22c>
NAS/ix-applications/k3s/kubelet:/pods/8cb5f21c-e67d-4752-9112-643f1ab8c0e4/etc-hosts
NAS/ix-applications/k3s/kubelet:/pods/8cb5f21c-e67d-4752-9112-643f1ab8c0e4/volumes/kubernetes.io~csi/pvc-287f77af-4fb9-49a7-a9dc-f3511effe4d0/vol_data.json
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/zerotier/ci/basic-values.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/zerotier/templates/common.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/zerotier/upgrade_strategy
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/enterprise/minio/metadata.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/enterprise/minio/questions.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/enterprise/minio/templates/_minio.tpl
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/enterprise/minio/templates/common.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/enterprise/minio/values.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/common/docs/container/command.md
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/common/docs/container/envFrom.md
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/common/docs/container/termination.md
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/common/docs/imagePullSecret.md
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/sonarr/Chart.lock
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/common/docs/persistence/README.md
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/sonarr/item.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/sonarr/questions.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/common/docs/service/README.md
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/sonarr/upgrade_strategy
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/sonarr/values.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/common/docs/workload/deployment.md
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/tailscale/Chart.lock
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/tailscale/Chart.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/tailscale/ci/basic-values.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/common/templates/class/_job.tpl
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/common/templates/class/_service.tpl
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/tailscale/templates/_tailscale.tpl
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/common/templates/helpers/_getPortRange.tpl
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/common/templates/helpers/_getSelectedPod.tpl
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/common/templates/lib/certificate/_getData.tpl
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/common/templates/lib/certificate/_validation.tpl
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/charts/ix-chart/templates/_helpers.tpl
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/charts/ix-chart/templates/_volumes.tpl
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/charts/ix-chart/templates/_workload.tpl
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/common/templates/lib/configmap/_validation.tpl
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/charts/ix-chart/templates/service.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/tdarr/metadata.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/common/templates/lib/container/_command.tpl
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/common/templates/lib/container/_env.tpl
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/charts/netdata/.helmignore
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/tdarr/templates/_portal.tpl
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/charts/netdata/Chart.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/charts/netdata/README.md
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/common/templates/lib/container/_lifecycle.tpl
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/tdarr/upgrade_strategy
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/tdarr/values.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/terraria/Chart.lock
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/common/templates/lib/container/_securityContext.tpl
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/common/templates/lib/container/_volumeMounts.tpl
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/terraria/app-readme.md
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/charts/netdata/templates/clusterrolebinding.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/charts/netdata/templates/deployment.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/terraria/ci/vanilla-basic-values.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/terraria/item.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/terraria/metadata.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/charts/nextcloud/app-readme.md
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/charts/nextcloud/item.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/charts/nextcloud/questions.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/vaultwarden/ci/admin-values.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/vaultwarden/ci/other-user-values.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/charts/nextcloud/templates/backup-postgres-config.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/vaultwarden/ci/ws-disabled-values.yaml
/mnt/NAS/ix-applications/catalogs/github_com_truenas_charts_git_master/library/ix-dev/community/vaultwarden/templates/_portal.tpl
NAS/ix-applications/docker:<0x0>
/mnt/NAS/ix-applications/docker/containerd/daemon/io.containerd.metadata.v1.bolt/meta.db
/mnt/NAS/ix-applications/docker/containers/d10725d15b0226c8a32b6a2149b38e6ae65b00df73d49b3e04b491a74b89cc05/d10725d15b0226c8a32b6a2149b38e6ae65b00df73d49b3e04b491a74b89cc05-json.log
/mnt/NAS/ix-applications/docker/overlay2/c1292c95f10e1a89cf222194feb8ecc1aab90181cb9a65bddac61558974c92e5/lower
/mnt/NAS/ix-applications/docker/containers/f6721c0e5de1fe3459c42b69d1f7d79dbd4a2fd98eb38d201e922673cd604174/hosts
/mnt/NAS/ix-applications/docker/overlay2/e79484c7fcff49a94a5602b55ac3e8fc61fb4e640357d06a80a37151eb0a6d97-init/lower
/mnt/NAS/ix-applications/docker/containers/992bc0a7166a358447c23ebf53467a0ddd75278a521d695f95af659d4dbd8876/config.v2.json
/mnt/NAS/ix-applications/docker/overlay2/4585ec12179c93d24f150fce2eef637986fc541e9b63898737440cc6a06286f3-init/lower
/mnt/NAS/ix-applications/docker/containers/88ebfb0be7779ccb335b3965fdaed77ff919a37d99727d477458e1e2b3ffbbf1/88ebfb0be7779ccb335b3965fdaed77ff919a37d99727d477458e1e2b3ffbbf1-json.log
/mnt/NAS/ix-applications/docker/containers/f2e60a29184e82ead3bca29d03a990835e8d83b7036ef7b0e92b8217f0d73643/config.v2.json
/mnt/NAS/ix-applications/docker/containers/f6721c0e5de1fe3459c42b69d1f7d79dbd4a2fd98eb38d201e922673cd604174/f6721c0e5de1fe3459c42b69d1f7d79dbd4a2fd98eb38d201e922673cd604174-json.log
/mnt/NAS/ix-applications/docker/containers/72dbc601ca64e5ed4773376c81f6ab2d16721b0e5a3edbd69114637d58e50abb/config.v2.json
/mnt/NAS/ix-applications/docker/containers/37f9c11933e913559d9c3da57069e42cdd958172de2e24d4957e2fef06fb42a5/config.v2.json
/mnt/NAS/ix-applications/docker/containers/d10725d15b0226c8a32b6a2149b38e6ae65b00df73d49b3e04b491a74b89cc05/hosts
/mnt/NAS/ix-applications/docker/containers/ade96b7c054d298269fd9406d7f2420ab36c7f4e1131ce2d5a4b178e37d1da7c/ade96b7c054d298269fd9406d7f2420ab36c7f4e1131ce2d5a4b178e37d1da7c-json.log
/mnt/NAS/ix-applications/docker/containers/fb362ca4dce1397c1b99f6f56291b5a350c7c464794fdaa83d16883e174201bf/config.v2.json
/mnt/NAS/ix-applications/docker/containers/992bc0a7166a358447c23ebf53467a0ddd75278a521d695f95af659d4dbd8876/hosts
/mnt/NAS/ix-applications/docker/containers/f2703d7a6fa4563a22207bc3b0e310b5c2061f4b186cd337e59f45691f553a49/f2703d7a6fa4563a22207bc3b0e310b5c2061f4b186cd337e59f45691f553a49-json.log
/mnt/NAS/ix-applications/docker/containers/d10725d15b0226c8a32b6a2149b38e6ae65b00df73d49b3e04b491a74b89cc05/config.v2.json
/mnt/NAS/ix-applications/docker/containers/cbd84ccb9175663c3779d06c779c86fbab62dab08c0d89a5b53574206bdfa8a1/config.v2.json
/mnt/NAS/ix-applications/docker/containers/fb362ca4dce1397c1b99f6f56291b5a350c7c464794fdaa83d16883e174201bf/hostconfig.json
/mnt/NAS/ix-applications/docker/containers/37f9c11933e913559d9c3da57069e42cdd958172de2e24d4957e2fef06fb42a5/37f9c11933e913559d9c3da57069e42cdd958172de2e24d4957e2fef06fb42a5-json.log
/mnt/NAS/ix-applications/docker/containers/cbd84ccb9175663c3779d06c779c86fbab62dab08c0d89a5b53574206bdfa8a1/hostconfig.json
/mnt/NAS/ix-applications/docker/containers/1fcf259d342c2e3c5e72bd215553e7e64f69845280ac843e28d58518ed54a094/hosts
/mnt/NAS/ix-applications/docker/containers/49870cd8e86d9f5ad782240e5ff82b25ac861949cbdbc82833d6402e9cd5cbe5/.tmp-config.v2.json489740597
/mnt/NAS/ix-applications/docker/containers/88ebfb0be7779ccb335b3965fdaed77ff919a37d99727d477458e1e2b3ffbbf1/hostconfig.json
/mnt/NAS/ix-applications/docker/containers/d96cfeaa930ba9718513b48c33f345675871d18841504a467d8f103583eef6ab/hosts
/mnt/NAS/ix-applications/docker/containers/a7a969a38c71abe657c761c9c7695cabfebd74dc9954430776493221875387f0/a7a969a38c71abe657c761c9c7695cabfebd74dc9954430776493221875387f0-json.log
/mnt/NAS/ix-applications/docker/overlay2/21818f699cd13e21980d63826d69c1791d76066da607a95d57514fde12bedd6b/lower
/mnt/NAS/ix-applications/docker/containers/f6721c0e5de1fe3459c42b69d1f7d79dbd4a2fd98eb38d201e922673cd604174/hostconfig.json
/mnt/NAS/ix-applications/docker/overlay2/03d10c20593b4a310d155674aa73d437e62ab6b18f916f280319382b1b717f21-init
NAS/ix-applications/docker:<0x212f6>
/mnt/NAS/ix-applications/docker/overlay2/c1292c95f10e1a89cf222194feb8ecc1aab90181cb9a65bddac61558974c92e5-init/lower
pool: boot-pool
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 00:00:13 with 0 errors on Fri May 19 03:45:15 2023
config:
NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
sda3 ONLINE 0 0 0
errors: No known data errors
