Add a third disk to encrypted mirror volume

[Meseira]

Hi,

I run FreeNAS 9.2.1.3 on my home storage server. To store all my family stuff, I have two 4TB disks in an encrypted mirror volume. For one of these disks, SMART began to report some errors and I have bought a new one with similar specs. Because the old disk is still working, I don't want to throw it now and keep it in the volume. Thus, my plan is to add the new disk to my encrypted mirror volume to get three mirrored disks ready for the annouced death of the old disk.

This plan sounds quite simple to me and I was surprised to see that it could not be as straigthforward as it seems... The GUI of FreeNAS does not provide a way to add a third disk to a mirror volume :-/ After looking around, I discover that I am not alone: in this thread, Chase tried to reach similar goal but cyberjock let him know that some tweaks have to be done.

I think that I understand what to do and I would appreciate to know if I am right:

1. Add swap and data partitions to the new disk with gpart add to look like the two other disks in the mirror,
2. Get the rawuuid of the data partition with gpart list,
3. Attach the GPTID of the new data partition to the GPTID of one of the old data partitions with zpool attach to preserve encryption,
4. Wait for complete resilvering process.

These steps are similar to the ones given in this link: http://startext.tomsk.ru/node/431

Is my plan correct? In particular for encryption, is it the right way? Will I have to regenerate my keys?

I am careful because not comfortable enough with encrypted ZFS volume to be sure to avoid data loss. Thanks for any advice!

 

[m0nkey_]

Any reason why you opted for an encrypted pool? From experience it just adds overhead and complications with such scenarios as you have presented. Encryption doesn't prevent access to your data in a running system, it only prevents access if the drives are physically stolen.

I am careful because not comfortable enough with encrypted ZFS volume to be sure to avoid data loss

In order to preserve your data and not completely destroying it, backup your data, rebuild your pool minus the encryption. This will make your life heck of a lot easier in the future should you experience a drive failure. If you're not experienced with FreeNAS encryption you shouldn't be using it.

 

[jdong]

My advice is, when in doubt, always click Re-key, change passphrase, and download recovery key in that order.

 

[Meseira]

Any reason why you opted for an encrypted pool?

As you mention, this is to prevent access if the drives are physically stolen. In such a case, I would not appreciate to think that an unknown person can see my family pictures among other stuff.

For now, I think that my initial plan does not really suit. Indeed, according to this link, I think that I have to initially encrypt the data parition with geli in order to attach a properly encrypted third partition to my mirror. I am going to do some tests in a VM, mainly to understand how FreeNAS will handle the keys after this step.

 

[jdong]

Any reason why you opted for an encrypted pool? From experience it just adds overhead and complications with such scenarios as you have presented. Encryption doesn't prevent access to your data in a running system, it only prevents access if the drives are physically stolen.



In order to preserve your data and not completely destroying it, backup your data, rebuild your pool minus the encryption. This will make your life heck of a lot easier in the future should you experience a drive failure. If you're not experienced with FreeNAS encryption you shouldn't be using it.

I think there's valid reasons to encrypt. Primarily, it protects you against privacy breaches if your servers are physically stolen or seized, and even more importantly, for RMA'ing your hard drives. Wiping disks takes a long time, and I'm afraid that a failed drive won't even be in a state where one can wipe it.

But you are right in that FreeNAS encryption is somewhat complicated and there's too many ways where slightly incorrectly performing these operations can lead to complete pool loss. I kind of wish for this to be more simple in future FreeNAS versions. For example, it ought to know that adding/replacing a drive requires rekeying, a new pass phrase, and a new recovery key...


Sent from my iPad using Tapatalk

 

[joeschmuck]

Encryption should be more simple to manage but as of right now it is not. I would recommend backing up your data, destroying the pool, add your new drive and recreate your pool without encryption. If you need encryption then use a product like TrueCrypt (example) to encrypt your data. This makes things much simpler all around so I'm told. Nope, I have only used TrueCrypt for USB flash drives and it worked fine when I needed that, never tried it on a FreeNAS share.

 

[m0nkey_]

As you mention, this is to prevent access if the drives are physically stolen. In such a case, I would not appreciate to think that an unknown person can see my family pictures among other stuff.

A would be thief would do is re-format the drives since they're ZFS and couldn't read them in a conventional box. If you're paranoid, as some have mentioned, use TrueCrypt/VeraCrypt for encrypted containers.

 

[Meseira]

Wow... Your messages are very pessimistic about data encryption with FreeNAS. For me, disk encryption is not a debatable option and I will definitively not follow your advices about recreating a pool without encryption or crafting encrypted volumes with bits of rope that won't make the maintenance easier.

I am currently doing what I wanted to avoid, namely full backup of not vital data (important ones are already backuped, of course). I will give a chance to FreeNAS and try the experiment of manually adding a third disk to my encrypted mirror volume through geli and zpool but without too much hope, according to your warnings :-/ Honestly, with all my stuff backuped, I am now really thinking about moving to a more usable and flexible solution like Debian system with btrfs raid1 based on disks encrypted with luks...

I find it sad that encryption is not as simple as it should be with FreeNAS.

 

[joeschmuck]

We all wish encryption was easier when it comes to dealing with the hard drives but that is just the way it is today. Apparently full drive encryption is required for your situation. Good luck on whichever path you go down.

 

[m0nkey_]

it protects you against privacy breaches if your servers are physically stolen or seized

Yes and no. It protects the drives, not the whole server. If the server was taken, the decryption key goes with it, so whoever boots it up next has access to your data.

Wow... Your messages are very pessimistic about data encryption with FreeNAS. For me, disk encryption is not a debatable option and I will definitively not follow your advices about recreating a pool without encryption or crafting encrypted volumes with bits of rope that won't make the maintenance easier.

Yes, it is a very pessimistic view. Encryption is not supposed to be easy, which in turn makes adding a drive to said encrypted pool difficult. Heck, even replacing an encrypted drive in FreeNAS isn't exactly straight forward. Make one false move and it's game over, you lose your pool. I'd personally rather not take the risk otherwise I could be recovering data from a backup for days. But hey, this is your pool, your data, you're free to do as you please.. however, you did state very clearly that you are not experienced with FreeNAS encryption and know that you run the risk of losing your pool. Good luck!

 

[Mr_N]

Yes and no. It protects the drives, not the whole server. If the server was taken, the decryption key goes with it, so whoever boots it up next has access to your data.

wait what? You have a passphrase lol

 

[m0nkey_]

You have a passphrase

Not by default. You have to set the passphrase manually.

 

[Mr_N]

ok please take your obviously biased views against encryption somewhere else

http://doc.freenas.org/9.3/freenas_storage.html#encryption

This design is only suitable for safe disposal of disks independent of the encryption key. As long as the key and the disks are intact, the system is vulnerable to being decrypted. The key should be protected by a strong passphrase and any backups of the key should be securely stored.

Do you really think a statistically relevant % of people using freenas encryption do so without setting a passphrase?

Everyone with even basic knowledge of technology knows you need to set a password on something if you don't want others to be able to access it, sure its part of the setup process when using things like truecrypt/veracrypt but why would everyone suddenly forget when using it in freenas?

*I am in no way supporting anyone using encryption without the basic levels of understanding of its use, they deserve any and all misfortune that may befall them... but assuming everyone falls into this category is insulting :P

 

[m0nkey_]

I'm hardly being bias. I see the value of having encrypted drives. I utilize full-disk encryption for my off-site backups, just not on FreeNAS.

why would everyone suddenly forget when using it in freenas?

Because FreeNAS does not prompt for a passphrase when configuring an encrypted pool. It does remind you to backup your recovery keys, and that's it. You also have to ask yourself how many people on the Forums or IRC actually read the documentation, the answer is not very many. You can figure that out by just reading the OP.

The documentation is very clear on how to configure an encrypted pool, how to replace a disk, etc. it also warns of the dangers of an encrypted pool. However, the OP asked to add another drive, something the documentation does not cover and going to a 3rd party site with unverified instructions will only add to the frustration of managing said pool. We also know that performing operations on a FreeNAS pool without informing the middleware can cause problems.

The OP is using an encrypted pool and wishes to add a third mirror, also admitting his limited knowledge with FreeNAS encryption. There is no way to do this officially. To add a 3rd drive, regardless if it's encrypted or not will require the pool to be rebuilt.

 

[Mr_N]

fair enough but i re-read the posts and still dont see where the OP says "i havent set a passphrase" so you still assumed that bit :)

 

[Meseira]

fair enough but i re-read the posts and still dont see where the OP says "i havent set a passphrase" so you still assumed that bit :)

Indeed, I have set a proper passphrase

In OP, I said that I am "not comfortable enough with encrypted ZFS volume", this does not mean ignorant ;-)

Anymay, thanks to all for spending time on my problem. I did some tries in a VM but I am still frustrating by the complexity of simply adding an encrypted disk to a mirror pool with FreeNAS and I prefer to avoid thinking about replacing such a disk in the future :-/ As I said, I was really thinking about moving to an other solution and I have finally taken this option. I leave FreeNAS for a Debian system with which I am definitely more comfortable. My plan is now to set up a BTRFS raid1 based on fully encrypted disks. By experience, I know that adding/replacing a disk in such a framework is so mush simpler.

 

[m0nkey_]

prefer to avoid thinking about replacing such a disk in the future :-/

We'd all prefer not to think about it, but it's a fact of life with multiple disks. You will end up having one fail and must replace it.

As I said, I was really thinking about moving to an other solution and I have finally taken this option. I leave FreeNAS for a Debian system with which I am definitely more comfortable. My plan is now to set up a BTRFS raid1 based on fully encrypted disks. By experience, I know that adding/replacing a disk in such a framework is so mush simpler.

It may appear simpler right now, but you must consider long term too. Is it really worth your time and effort now? What would you gain from the move? Will BTRFS give you the same level of filesystem protection? Will it provide you with notifications for a failing drive or pool? Either way you chose (staying with FreeNAS or going Debian), you have to rebuild. I have limited experience with BTRFS, so can't comment too much, but I am aware that anything more than a mirror is considered 'experimental'. Then again, I could be wrong, but I've not followed up for a while.

 

[Meseira]

We'd all prefer not to think about it, but it's a fact of life with multiple disks. You will end up having one fail and must replace it.

Yes, I was only speaking about replacing an encrypted disk in my ZFS mirror. Of course, I am aware that disks in multiple devices are supposed to be replaced...

It may appear simpler right now, but you must consider long term too. Is it really worth your time and effort now?

Definitely yes, I have already wasted too much time with these problems around encryption in FreeNAS.

What would you gain from the move?

Comfort of a system that I better know than FreeBSD/FreeNAS and with which I am able to deal with without tons of questions about things that seem simple to my humble opinion.

Will BTRFS give you the same level of filesystem protection?

Such a question is an endless source of trolls...

Will it provide you with notifications for a failing drive or pool?

I can script such features.

Either way you chose (staying with FreeNAS or going Debian), you have to rebuild.

Yes and I have already begun...

I have limited experience with BTRFS, so can't comment too much, but I am aware that anything more than a mirror is considered 'experimental'. Then again, I could be wrong, but I've not followed up for a while.

You are wrong, a lot of stuff with BTRFS can now be considered as stable since August 2014.

 

[m0nkey_]

I had a quick read of the official site for BTRFS, the FAQ has one question: Is BTRFS stable? The answer: Maybe. That alone doesn't give me confidence. I'm going to stick with the tried and tested ZFS filesystem for the foreseeable future.

However, since you're going down that avenue, it appears there is a similar appliance to FreeNAS for the BTRFS filesystem, known as Rockstor. You may have better luck with it: http://rockstor.com/.

I wish you well on your endeavor for a filesystem that works for you.

 

[cyberjock]

Do you really think a statistically relevant % of people using freenas encryption do so without setting a passphrase?

Yep. These people with 1000's of posts know this is true. :)

In fact, if you understand encryption, unless you've actually set a passphrase and downloaded the keys, your data is very easy to gain access to for someone that really wants access to your data.

The whole purpose of having a key + passphrase is that your encryption is MUCH harder to crack with a full key being required. passphrases are trivially easy to brute-force these days.

Now to switch gears...

As a user of an encrypted pool, the "pessimism" you claim that people have is totally justifiable. You'd be amazed how many people have lost all of their data because they didn't know what they were doing, hit some bug they didn't know about, etc. So yeah, I don't recommend it for anyone that it isn't required by law to have. The reality is that if someone stole your server and couldn't get a GUI on power-on they're gonna go pawn it or wipe the disks (and your data was protected only because FreeNAS and ZFS was too complicated to figure out).

If you plan to be very active and informed (and plan to *stay* informed) as to the status of encryption and the bugs involved, by all means use it. If you don't plan to do all of those things, you probably should look elsewhere for "encryption" like everyone has already said.

HTH.