Convert a single-disk stripe on a zpool to a mirror stripe

[allegiance]

I have inherited a FreeNAS 8.3.2 system with a single zpool on it. Unfortunately, it seems to be an odd setup. As far as I can tell, it consists of two vdevs, a raidz2 and a single-drive stripe. Below is the output from zpool status:
pool: Shared
state: ONLINE
scan: resilvered 1.60T in 73h31m with 0 errors on Sun Aug 10 13:19:32 2014
config:

NAME STATE READ WRITE CKSUM
Shared ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/85ad335c-5c27-11e2-9d8a-f46d042cc584 ONLINE 0 0 0
gptid/85fb1270-5c27-11e2-9d8a-f46d042cc584 ONLINE 0 0 0
gptid/8638a789-5c27-11e2-9d8a-f46d042cc584 ONLINE 0 0 0
gptid/867d4b60-5c27-11e2-9d8a-f46d042cc584 ONLINE 0 0 0
gptid/86be2c3e-5c27-11e2-9d8a-f46d042cc584 ONLINE 0 0 0
gptid/870502bf-5c27-11e2-9d8a-f46d042cc584 ONLINE 0 0 0
gptid/8745e469-5c27-11e2-9d8a-f46d042cc584 ONLINE 0 0 0
gptid/2ba8062f-1e4a-11e4-b147-f46d042cc584 ONLINE 0 0 0
gptid/87c76f9a-5c27-11e2-9d8a-f46d042cc584 ONLINE 0 0 0
gptid/885f0b5f-5c27-11e2-9d8a-f46d042cc584 ONLINE 0 0 0
gptid/88ff45ca-5c27-11e2-9d8a-f46d042cc584 ONLINE 0 0 0
gptid/89a667ff-5c27-11e2-9d8a-f46d042cc584 ONLINE 0 0 0
gptid/8a446af1-5c27-11e2-9d8a-f46d042cc584 ONLINE 0 0 0
gptid/da39d3e5-c48b-11e2-bb6d-f46d042cc584 ONLINE 0 0 0

errors: No known data errors

The last drive is shown as a stripe in the GUI.

My fear is that if that single drive dies, my whole zpool is lost. As far as I know, I cannot simple remove it from the zpool, so I thought the next best thing would be to make it a 2-disk mirror. I've tried so by adding a new hard drive to that machine and trying the following:
zpool create Shared mirror gptid/da39d3e5-c48b-11e2-bb6d-f46d042cc584 gptid/85ad335c-5c27-11e2-9d8a-f46d042cc584
invalid vdev specification
use '-f' to override the following errors:
/dev/gptid/da39d3e5-c48b-11e2-bb6d-f46d042cc584 is part of active pool 'Shared'







Is there anyway to remedy this short of backing up all the data and creating the zpool from scratch? I'd be happy with either making the single-disk stripe a mirror stripe or removing it from the zpool altogether. Or are my fears of that "single disk" unfounded?

 

[allegiance]

 

[cyberjock]

That is definitely a stripe. There is a command line procedure somewhere that shows how to do it but I don't have a link handy.

But to put it short, if da9 fails you lose the entire pool. Your understanding is correct and you are at risk for being a statistic. You (err, someone) also decided to do a 13 disk RAIDZ2 which is very wide and not recommended.

 

[david kennedy]

To be honest, not sure you want to risk doing anything to the pool considering if da9 dies you lose everything.

Since you asked about creating a mirror, the syntax (under Solaris) would be something like:
zpool attach poolname existingdisk newdisk

It is a bit more tricky under freenas as i think you need to copy the disk format from one of the existing devices first (it contains slices used for swap and such). More a solaris guy trying to get into freenas then a freenas guy so take the advice with a huge grain of salt...

Personally I'd recommend against trying to do anything with the pool other then "zfs send"ing it to another storage device and recreating this, or restoring from backups to your newly created pool.
I don't have a freenas box in front of me at the moment, but you might want to check if zpool supports the "-n" option (dry run).

Lastly, freenas didn't intend for you to manipulate devices outside the GUI, so you may end up making things worse if you attempt to add another device on top of potentially damaging your pool.

This appears to be the classic case of someone trying to add another device to increase available storage in a pool without realizing that zfs can not "reshape" an array, unlike what can be done in linux under mdadm.



PS. Actually doing something similar myself on a Solaris 11.2 box due to a failed non-raid disk (I backed it up to a raidz2 pool a few days ago). Anyhow, replaced the disk an "zfs send"ing it back.

Goes pretty quickly:

sending full stream to z02/summer@backup
estimated stream size: 1.91T
571GB 0:52:52 [ 469MB/s]
575GB 0:53:13 [ 478MB/s]
...

 

[cyberjock]

The command given above is only a part of a multi-step procedure to go from a single disk vdev to a mirror. It is not complete and shouldn't be used alone with FreeNAS.

 

[allegiance]

Yes, I also thought the 13-disk raidz2 was most likely not the best scenario, but that is not my primary concern (I assume multiples of 6-disk zpool2 would be better?).

I had also tried the zpool attach poolname existingdisk newdisk, to get the error:
invalid vdev specification
use '-f' to override the following errors:
/dev/gptid/existingdisk is part of active pool 'Shared'

But I didn't want to "force" it.

My eventual plan is to backup and rebuild this whole pool, but for now I would really like to mirror that single disk if at all possible.

 

[cyberjock]

Yeah, you definitely need to get that disk mirrored. As I said above there is instructions somewhere on the forums. You're just going to have to find it. If I remember correctly it's out of date so if you find the instructions I'll let you know what needs updated. Sorry but I'm too busy to go looking for those instructions right now or I'd do it myself.

 

[allegiance]

I will keep looking. All I had found so far was the zpool attach poolname existingdisk newdisk which results in the error
invalid vdev specification
use '-f' to override the following errors:
/dev/gptid/existingdisk is part of active pool 'Shared'

Any help is much appreciated.

 

[gpsguy]

Search for a message titled "Create ZFS mirror by adding a drive" dated Sep 10, 2013.

Dusan and paleoN discuss ways of doing it.

 

[allegiance]

Search for a message titled "Create ZFS mirror by adding a drive" dated Sep 10, 2013.

Dusan and paleoN discuss ways of doing it.

Thank you.

The summary instructions seemed to be (where /dev/ada1 is the new disk and tank is the pool name):

1. gpart create -s gpt /dev/ada1
2. gpart add -b 128 -t freebsd-swap -s 2G /dev/ada1
3. gpart add -t freebsd-zfs /dev/ada1
4. Run zpool status and note the gptid of the existing disk
5. Run gpart list and find the gptid of the newly created partition. It is the rawuuid field. In this example it would be rawuuid of ada1p1
6. zpool attach tank /dev/gptid/[gptid_of_the_existing_disk] /dev/gptid/[gptid_of_the_new_partition]

I will give this a try. Cyberjock, you had mentioned that you thought the notes may need updating - the notes list this as for FreeNAS 9.1.1, and I am running 8.3.2. Do you recommend any changes to the above before I give it a try? Thanks again.

 

[david kennedy]

Thank you.

1. zpool attach tank /dev/gptid/[gptid_of_the_existing_disk] /dev/gptid/[gptid_of_the_new_partition]

Slightly off Off-topic comment but I like freenas, it has a lot of good things going for it (vs Solaris) but using disk slices vs the whole disk (as is recommended) makes some things more difficult.

Per the ZFS best practices guide:
For production systems, use whole disks rather than slices for storage pools for the following reasons:
...
The recovery process of replacing a failed disk is more complex when disks contain both ZFS and UFS file systems on slices.

I've created and split mirrors multiple times in Solaris and it is rather easy (even when mirroring a boot disk under solaris 11.1).

Perhaps Freenas could one day move to using whole disks and have swap in the pool? Would also prevent a system crash in the event of a disk failure as well (app can no longer read the paged data as the disk is dead.)

 

[cyberjock]

They aren't 100% correct anymore, but they will do the job.

 

[cyberjock]

Umm.. FreeNAS has deliberately not gone with whole disks for a number of reasons. FreeNAS actually used whole disks in the early stages of development and was later fixed because it is better this way for a bunch of reasons.

No, getting rid of the swapspace won't prevent system crashes as you:

1. Should *always* have swapspace in the event that things go wrong.
2. Should never be actually using swapspace unless something is wrong.
3. If given a choice between using swapspace when something goes wrong or just crashing the box, which would you chose? Exactly.

Sorry but the developers have already done the 'deep thinking' and made the smartest decisions they could in regards to protecting your data, maintaining uptime, etc.

 

[david kennedy]

Umm.. FreeNAS has deliberately not gone with whole disks for a number of reasons. FreeNAS actually used whole disks in the early stages of development and was later fixed because it is better this way for a bunch of reasons.

No, getting rid of the swapspace won't prevent system crashes as you:

1. Should *always* have swapspace in the event that things go wrong.
2. Should never be actually using swapspace unless something is wrong.
3. If given a choice between using swapspace when something goes wrong or just crashing the box, which would you chose? Exactly.

Sorry but the developers have already done the 'deep thinking' and made the smartest decisions they could in regards to protecting your data, maintaining uptime, etc.

Where did I say get rid of swap space? I had clearly stated "and have swap in the pool"

So if you have a mirrored pool you have a mirror for swap. In this situation if a disk fails your swap is still available. In the current freenas model if a disk fails where has your swap gone?

Put another way, under freenas if a disk fails and you have swapped out data why will your app not crash given that it is now unable to retrieve the data from disk?

Out of curiosity, why is this better (what are the actual reasons) considering this is contrary to the recommendations from SUN (who initially created ZFS)?

Again, per oracle:
Recommended ZFS Storage Pool Practices
Non-root pools – Create non-root pools with whole disks by using the d* identifier. Do not use the p* identifier.

 

[cyberjock]

I stand corrected on your comment about the swapspace. Hadn't had coffee yet. But, on the flipside it's still better to keep the swapspace outside the pool incase the pool is having problems. The reality is that since the swapspace shouldn't *ever* be used, where you choose to store it is really inconsequential. Also ZFS's CoW philosophy could cause all that nasty fragmentation and crap that you are supposed to be trying to avoid by using partitions or slices dedicated for swap. Hopefully you see how mirroring something you aren't going to use anyway doesn't gain you anything.

If a disk fails with FreeNAS and you have swapped out data your app does stand a chance of crashing. The MUCH MUCH MUCH bigger problem.. if you are using swapspace you have already screwed up badly. The swapspace is literally only meant as a spare tire. If you are using it for any extended period of time you've failed as the admin of your server. A crashed application because swap disappeared is the least of your worries and I'd also argue that it's a damn good thing that the app is crashing so that the admin can get a boot in the butt and do his job before the server ends up down for the long haul.

Sun used to do whole disks because of the way their caching system worked. That's been long deprecated and they recommend partitions as you have a way of validating what is on the disk when you are using an OS that doesn't support ZFS. There's a bunch more reasons but I honestly don't have time to go into it again. There is a thread where this is discussed in detail earlier this year if you want to look for it. The whole disk versus partition thing is very old, there's pluses and minuses for each choice and FreeNAS has decided to go with partitions. If this is unpalatable to you feel free to use FreeBSD and do it however makes you happy. :)

Oracle's documentation is not completely correct or accurate for the FreeBSD implementation of ZFS. So using that documentation is only useful if you already know about the differences. And that means you don't need to read the documentation.

 

[allegiance]

They aren't 100% correct anymore, but they will do the job.

Thank you. I will give it a shot once I finish backing up the most critical pieces on this pool (it is a backup repository itself, so most of the data does reside elsewhere).

By the way, now that I have taken ownership of this thing, I spotted your guide mentioned in your signature and figured it would be a good place to get some base knowledge on this system. I wanted to tell you thank you for compiling all that information - it was very, very informative and will be very helpful to me.

 

[cyberjock]

You're welcome. Thanks for reading it. It saves us a bunch of time by not having to answer those questions again. :D

 

[david kennedy]

Sun used to do whole disks because of the way their caching system worked. That's been long deprecated and they recommend partitions as you have a way of validating what is on the disk when you are using an OS that doesn't support ZFS

But, on the flipside
Oracle's documentation is not completely correct or accurate for the FreeBSD implementation of ZFS. So using that documentation is only useful if you already know about the differences. And that means you don't need to read the documentation.

Agreed that swap really shouldn't be used but since freenas supports zvol's (http://doc.freenas.org/index.php/Volumes#Creating_a_zvol) the 'fragmentation' issue wouldn't be a problem.

Also agree that there are differences between SUN and ZoL or freebsd's zfs implementations but the reason for using whole disks was to make replacement easier (think zpool replace).

Lastly, disagree on the statement with SUN recommending partitions.
From Solaris 11.2 docs (only releced recently) they still say use whole disks.

 

[cyberjock]

Oh, so Sun is back to whole disks. Notice how much I care about Sun's implementation? ;) Don't even try to keep up with their stuff because it may or may not apply for FreeBSD. :p

And why do you think zvols don't fragment. zvols absolutely will fragment... LOL. Ask the iscsi suckers that did zvols to 90% full on their pool and couldn't figure out why performance degraded more and more each day. ;)

 

[david kennedy]

Oh, so Sun is back to whole disks. Notice how much I care about Sun's implementation? ;) Don't even try to keep up with their stuff because it may or may not apply for FreeBSD. :p

And why do you think zvols don't fragment. zvols absolutely will fragment... LOL. Ask the iscsi suckers that did zvols to 90% full on their pool and couldn't figure out why performance degraded more and more each day. ;)

Please quote where i said zvols don't fragment?
Is that how you interpreted this: "the 'fragmentation' issue wouldn't be a problem."

If swap is only for an emergency and you create an 2 - 10 gb zvol how bad can this get? Why would you create a swap zvol to 90%
As you stated earlier swap shouldn't be used anyhow (at least in the context of freenas) so even if it is fragmented does it really matter since it is unused?

I thought it was well known that zfs changes its allocation method at around 80% to the "wow it is slow now" method (best fit/ first fit).

You might think this is a moot point but there are two reasons why i ask this:

1) As discussed it is harder to replace drives and such
2) I have 20 disks and the default is to create 2gb/disk. I can change this but the gui strongly recommends against setting swap to zero (disable).
Assuming i don't change the defaults I would end up with ~40GB of swap and the system only has 24gb of physical (Yes ECC).

Name an OS that recommends swap ~ 2x physical?
Most OSs recommend AGAINST this.