Failing disk and lost to ZFS config

[Singman]

Hi,

I'm pretty noob to Freenas and ZFS (but not on storage, I'm at ease with Datacore or Data Domain for my work), using it for my personnal lab.
So, I build a little NAS and put 2x 2 Tb disks in it some times ago. I've read all the docs I could but it look like I missed all the part about raid :)

Now, I have a faulty disk (read errors and some files on my storage are corrupted). I bought two others disks (2 Tb also), put them in the NAS and... that's all ! I have two questions :
- with 2 disks, I'm pretty sure I dont have any raid by default. That's something I missed when I build the NAS (shame on me). I need one. Can I do it now ?
- with the 2 additionnal disks, I think I could make a new pool, duplicate data, but that will not give me a raid (again, 2x disks only). I should include the safe "old" disk without loosing data.

For informations :
(i've launched a scrub after powering down and up)

root@nas[~]# zpool status
pool: Pool_1
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: scrub in progress since Sat Dec 7 17:57:41 2019
681G scanned at 581M/s, 211G issued at 180M/s, 1.44T total
0 repaired, 14.30% done, 0 days 02:00:03 to go
config:

NAME STATE READ WRITE CKSUM
Pool_1 ONLINE 0 0 2
gptid/ae929bad-4ff0-11e9-b7bf-00012e23ba5f ONLINE 0 0 4
gptid/d7fc7ef3-570d-11e9-8567-00012e23ba5f ONLINE 0 0 0

errors: 9 data errors, use '-v' for a list

pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0 days 00:00:31 with 0 errors on Mon Dec 2 03:45:31 2019
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
ada0p2 ONLINE 0 0 0

errors: No known data errors

Messages I've received before powering down and up the NAS (to add my 2 disks) :

Checking status of zfs pools:
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
Pool_1 3.62T 1.44T 2.18T - - 10% 39% 1.00x DEGRADED /mnt
freenas-boot 55.5G 4.54G 51.0G - - - 8% 1.00x ONLINE -

pool: Pool_1
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: scrub repaired 0 in 0 days 03:45:22 with 1 errors on Sun Nov 24 03:45:30 2019
config:

NAME STATE READ WRITE CKSUM
Pool_1 DEGRADED 0 0 123
gptid/ae929bad-4ff0-11e9-b7bf-00012e23ba5f DEGRADED 0 0 210 too many errors
gptid/d7fc7ef3-570d-11e9-8567-00012e23ba5f ONLINE 0 0 36

errors: 9 data errors, use '-v' for a list

-- End of daily output --

FreeNAS @ nas.domain.lan

New alert:
* Pool Pool_1 state is DEGRADED: One or more devices has experienced an error resulting in data corruption. Applications may be affected.

The following alert has been cleared:
* Pool Pool_1 state is ONLINE: One or more devices has experienced an error resulting in data corruption. Applications may be affected.

Current alerts:
* Device: /dev/ada2, 16 Currently unreadable (pending) sectors
* Device: /dev/ada2, 16 Offline uncorrectable sectors
* Device: /dev/ada2, Self-Test Log error count increased from 5 to 6
* Pool Pool_1 state is DEGRADED: One or more devices has experienced an error resulting in data corruption. Applications may be affected.

 

[blueether]

it looks to be striped - no redundency, do you have a backup of the data?
How much data do you have on the pool? will it fit in 2TB? looks like 1.44TB
How many disks can the system hold?
Do you have snapshots of the data?


I think I would switch to mirrored pairs by:
adding the new 2 x 2TB as a mirror in a new pool
copy all the data to the new pool (replacing the corrpted data or deleting it)
get another 2 TB drive and add a second mirrored using the pair to the new pool giving 2 x 2 TB mirrored pairs

Others may have other/better solutions

 

[Chris Moore]

It looks like you have data errors on both drives in your pool, which is what might (outside of ZFS) be called a RAID-0 set. You can't do this from the GUI, but you could add a mirror to each of those individual disks using some steps from the command line, if you feel comfortable with doing that, I can give you instructions to follow. It would involve partitioning the new disks and some other incantations.

 

[Chris Moore]

I will go ahead and put the link here so you can look at it if I am not handy when you are ready:

Create zfs mirror by adding a drive?

Hi there, I run my own business and have been using FreeNAS on an HP Proliant Microserver for over a year but it just sits there. The factory HDD suddenly died and I had no solution. I use FreeNAS for Windows CIFS fileshares and to date there's no easy way to get Acronis/Crashplan or any other...

www.ixsystems.com

You just need to add a mirror disk to each of your existing storage disks. You will then have redundancy and you will still need to follow the user guide with regard to replacing any defective disks.

 

[Singman]

I think I could copy my data to one disk only. But can I transform the whole into a RAIDZ1 (aka RAID-5) ? It's some kind of goal (one disk could fail and when I change it, RAID will rebuild).

 

[blueether]

There is no way to go from a stripe (or mirrors) to raidz1 or 2 without rebuilding the whole pool.

 

[Singman]

Ok, so maybe I could build a new pool RAIDZ1 with the 2x 2 Tb disks, copy data to it, remove the old stripped pool. Could I add later some disks to a RAIDZ pool ?

 

[Chris Moore]

You can create a degraded pool. A pool with a missing disk, but you would need to do some command line work as the GUI has no functionality for this kind of thing. Here is a guide:

Creating a degraded pool

WARNING: HERE BE DRAGONS. IF YOU DON'T KNOW EXACTLY WHAT YOU'RE DOING, DO NOT FOLLOW THESE INSTRUCTIONS. IN FACT, IF YOU NEED THESE INSTRUCTIONS, YOU PROBABLY SHOULDN'T FOLLOW THEM. There are a few cases where it may make sense to create a...

www.ixsystems.com

 

[Singman]

Ok, so maybe I could build a new pool RAIDZ1 with the 2x 2 Tb disks, copy data to it, remove the old stripped pool. Could I add later some disks to a RAIDZ pool ?

Hum, only stripe or mirror available options with 2 disks :/

 

[Chris Moore]

PS. I would suggest getting a couple more disks. Personally, I like RAIDz2.

You might want to review some of these gides for better clarity:

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://www.ixsystems.com/community...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Overview of ZFS Pools in FreeNAS from the iXsystems blog:
https://www.ixsystems.com/blog/zfs-pools-in-freenas/

Terminology and Abbreviations Primer
https://www.ixsystems.com/community/threads/terminology-and-abbreviations-primer.28174/

 

[Singman]

I will go ahead and put the link here so you can look at it if I am not handy when you are ready:

Create zfs mirror by adding a drive?

Hi there, I run my own business and have been using FreeNAS on an HP Proliant Microserver for over a year but it just sits there. The factory HDD suddenly died and I had no solution. I use FreeNAS for Windows CIFS fileshares and to date there's no easy way to get Acronis/Crashplan or any other...

www.ixsystems.com

You just need to add a mirror disk to each of your existing storage disks. You will then have redundancy and you will still need to follow the user guide with regard to replacing any defective disks.

I will try that solution. Mirroring my existing disks with the two new, then I will exchange the faulty one. I also have a backup on cloud storage but it's very slow to restore :)

 

[Singman]

Ok, I can't do that. ZFS make everything too complicated compared to anything else, there are no clear message that show what the pool is, his configuration, raid | mirror | stripe or anything else. BTW, the documentation is horrible.

I still don't know what is my pool : mirror or stripe ?
2nd, the best solution is to mirror the existing (2 disks) pool with the 2 new disks, so I could swap the failing one. But the thread you gave me is about mirroring a pool with only one disk. So, how is working ZFS ? Mirror disk by disk ? vdev by vdevs ? Pool by pool ? I'm totally lost, and reading more and more documentation lead to nowhere.*

 

[Chris Moore]

Your two disks are each a vdev, using ZFS terminology. Each of the vdevs in a ZFS storage pool are striped with all the other vdevs in that pool. All reddundancy is done at the vdev level. The example is a single vdev pool where a mirror is being added to that vdev. Your pool contains two vdevs, so you would need to do that process with each of the vdevs in your existing pool. Once you become more familiar with the function of ZFS and the terms used, you will find that it is very simple and easy. It is just different.

 

[Singman]

Ok. I think I understand what you mean. I've draw a picture to show you if I'm right :

On left, what I have now : a pool with two vdevs, one disk each (risky setup, vdev are SPF) and disk 2 failing.
On right and in black, what I want : a pool with two vdevs, two disk each. At 1st, I was thinking my old pool was a single vdev with two disk but it's not, so in red, i've modified the picture : 1st vdev with disk 1 & 3, 2 vdev with disk 2 & 4.
In that setup, I will have a reddundancy of one disk failure in each vdev. That's perfect. But again 2 questions :
- how do you find with the "zpool status" command where are vdev ? I don't see anything.
- from the link you gave me (https://www.ixsystems.com/community/threads/create-zfs-mirror-by-adding-a-drive.14880/#post-81348), I dont fully understand the last command : zpool attach tank /dev/gptid/[gptid_of_the_existing_disk] /dev/gptid/[gptid_of_the_new_partition]. Look like it will attach the new vdev to the existing pool but not to the vdev #1 or #2.

Thanks for your help.

 

[danb35]

Look like it will attach the new vdev to the existing pool but not to the vdev #1 or #2.

It doesn't create a new vdev; it modifies an existing one (and this is, as yet, the only way that vdevs can be modified: disks can be added to create mirrors (or add to existing mirrors), or taken away from mirrors). It knows which one to modify because you gave it the gptid of the existing disk. So, when you run zpool attach tank /dev/disk1 /dev/disk3, the vdev that currently contains only disk1 will contain both disk1 and disk3 in a mirror. You can then do the same thing for disk2 and disk4, and you don't need to wait for the first rebuild (or "resilver", in ZFS parlance) to complete before beginning the second.

 

[Singman]

Ok. Look like everything is ok :

Code:

root@nas[~]# zpool status
  pool: Pool_1
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Dec  8 12:04:04 2019
        755G scanned at 616M/s, 20K issued at 1.82K/s, 1.42T total
        0 resilvered, 0.00% done, no estimated completion time
config:

        NAME                                            STATE     READ WRITE CKSUM
        Pool_1                                          ONLINE       1     0    19
          mirror-0                                      ONLINE       0     0     0
            gptid/ae929bad-4ff0-11e9-b7bf-00012e23ba5f  ONLINE       1     0    37
            gptid/19c0fdae-19a9-11ea-83f4-00012e23ba5f  ONLINE       0     0     0
          mirror-1                                      ONLINE       0     0     0
            gptid/d7fc7ef3-570d-11e9-8567-00012e23ba5f  ONLINE       0     0     2
            gptid/1805e595-19a9-11ea-83f4-00012e23ba5f  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:31 with 0 errors on Mon Dec  2 03:45:31 2019
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          ada0p2    ONLINE       0     0     0

errors: No known data errors

 

[Singman]

Thanks everyone for your help !

 

[Chris Moore]

Don't forget, the two old disks are both exhibiting errors and it would be best to replace them. That can be done through the GUI.