Manually mount an encrypted ZFS pool

[David123]

Hello,

First of all, I'm new to FreeNAS which can probaly explain the following question... ^^

I'm working on a small script to manualy mount an encrypted ZFS pool.

Code:

#!/bin/bash
disk1='AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA'
disk2='BBBBBBBB-BBBB-BBBB-BBBB-BBBBBBBBBBBB'
geliKey='/data/geli/KKKKKKKK-KKKK-KKKK-KKKK-KKKKKKKKKKKK.key'
passphrase='/root/mykey.txt'
zpoolName='MYPOOL'

geli attach -j $passphrase -k $geliKey /dev/gptid/$disk1
geli attach -j $passphrase -k $geliKey /dev/gptid/$disk2
zpool import -R /mnt $zpoolName

If I unlock the disk using the GUI, the "df -h" result is :

Code:

[root@lab-nas] /var# df -h
Filesystem                                              Size    Used   Avail Capacity  Mounted on
freenas-boot/ROOT/default                               3.8G    933M    2.9G    24%    /
devfs                                                   1.0k    1.0k      0B   100%    /dev
tmpfs                                                    32M    5.3M     26M    17%    /etc
tmpfs                                                   4.0M    8.0k      4M     0%    /mnt
tmpfs                                                   332M     62M    270M    19%    /var
freenas-boot/grub                                       2.9G    7.8M    2.9G     0%    /boot/grub
DATA                                                    1.9G    108k    1.9G     0%    /mnt/DATA
DATA/.system                                            1.9G    752k    1.9G     0%    /var/db/system
DATA/.system/cores                                      1.9G    700k    1.9G     0%    /var/db/system/cores
DATA/.system/samba4                                     1.9G    300k    1.9G     0%    /var/db/system/samba4
DATA/.system/syslog-ffb84ccc300c4843b1352f93d2beb43e    1.9G    240k    1.9G     0%    /var/db/system/syslog-ffb84ccc300c4843b1352f93d2beb43e
DATA/.system/rrd-ffb84ccc300c4843b1352f93d2beb43e       1.9G     96k    1.9G     0%    /var/db/system/rrd-ffb84ccc300c4843b1352f93d2beb43e

If I unlock the disk using the script, the "df -h" result is :

Code:

root@lab-nas] /var# df -h
Filesystem                   Size    Used   Avail Capacity  Mounted on
freenas-boot/ROOT/default    3.8G    933M    2.9G    24%    /
devfs                        1.0k    1.0k      0B   100%    /dev
tmpfs                         32M    5.3M     26M    17%    /etc
tmpfs                        4.0M    8.0k      4M     0%    /mnt
tmpfs                        332M     62M    270M    19%    /var
freenas-boot/grub            2.9G    7.8M    2.9G     0%    /boot/grub
DATA                         1.9G    108k    1.9G     0%    /mnt/DATA

When I use the "zpool import" command, all the legacy (.system/...) are not mounted.

I can manually mount it (using for exemple : mount -t zfs DATA/.system /var/db/system) but is it the good way to proceed ?
What should I do to have it all mounted properly ?

Thanks for you help :)

 

[cyberjock]

The short answer, you don't do what you are doing. What you are doing is very bad news for FreeNAS. It's an appliance. As such, you should use it as designed or you can have major problems (even sudden data loss).

This is such a bad idea I was hesitant to even tell you its a bad idea. From experience I've learned that random forum users take my reply as endorsing an idea. I don't.

Even if you fixed your problems right now, you aren't going to know what has or hasn't changed with an update (and those come twice a week). So unless you plan to audit every change made to the FreeNAS code, this is a *very* bad idea. And I'm sure the answers are in the code, and since you are asking those questions I'd be willing to bet you aren't experienced enough to be able to audit the FreeNAS code.

 

[Tywin]

It's an appliance. As such, you should use it as designed or you can have major problems (even sudden data loss).

I'm sorry this is off-topic, I don't really know where better to put this than in response to one of your replies :(

I don't think it is your intent @cyberjock, but reading reply after reply of yours saying "you could have sudden data loss!" under nearly any circumstances is making me and probably others think that FreeNAS is really not very reliable at all. I know there are a tonne of success stories, but it just seems like everything -- from your view, anyway -- is terribly fragile, and if you poke at it at all, the whole tower might fall over. This is not the image of a robust system, and I see it several times a day from you and other frequent posters.

In my industry, we refer to this kind of stuff as "death modes" -- ways in which the operator can kill the system. A well-designed robust system should be free of death modes. In my industry, the spacecraft should be unkillable from the ground. I recognize that it is impossible to cover every scenario -- there's nothing FreeNAS can do against a user taking a hammer to their platters -- but it seems like it has a long way to go. Encryption is a great example. You say you don't think the risks are worth the benefits, but I have to ask myself why there are any risks at all. Key management is the user's responsibility (this is like the hammer scenario), but it should be impossible -- or at least extremely difficult -- to replace a disk and lose your pool using the GUI, even if it is encrypted.

It is unfortunate, because I think FreeNAS is great and I love what they are trying to do, but it has to be more robust to off-nominal situations. Another great example is memory -- I've read on these forums time and again that using insufficient memory can destroy your data. This is ridiculous. If the system doesn't have enough memory, it should fail gracefully: "help, I'm dyi--!", not eat your data on it's way down. This sounds like a nasty unresolved bug. Sure, the correct answer is to throw more memory at it, but the punishment for using insufficient memory should be failure to operate, not data loss.

Sorry, I'm ranting now and this is getting out of hand. I'm not trying to attack anyone, I just think that FreeNAS could benefit from a) users that are less tolerant of weird unexplained behaviour, and b) more robust system design (here the system is the FreeNAS stack, not the system (computer) in your closet).

 

[cyberjock]

I'm sorry this is off-topic, I don't really know where better to put this than in response to one of your replies :(

I don't think it is your intent @cyberjock, but reading reply after reply of yours saying "you could have sudden data loss!" under nearly any circumstances is making me and probably others think that FreeNAS is really not very reliable at all. I know there are a tonne of success stories, but it just seems like everything -- from your view, anyway -- is terribly fragile, and if you poke at it at all, the whole tower might fall over. This is not the image of a robust system, and I see it several times a day from you and other frequent posters.

In my industry, we refer to this kind of stuff as "death modes" -- ways in which the operator can kill the system. A well-designed robust system should be free of death modes. In my industry, the spacecraft should be unkillable from the ground. I recognize that it is impossible to cover every scenario -- there's nothing FreeNAS can do against a user taking a hammer to their platters -- but it seems like it has a long way to go. Encryption is a great example. You say you don't think the risks are worth the benefits, but I have to ask myself why there are any risks at all. Key management is the user's responsibility (this is like the hammer scenario), but it should be impossible -- or at least extremely difficult -- to replace a disk and lose your pool using the GUI, even if it is encrypted.
It's not fragile, as long as you use the OS as designed. The problem: Everyone has some dream of how they want it to be, and they think they can bend FreeNAS to their desires and still have a reliable system. The reality: That's just not how it works.

You are absolutely right that FreeNAS can't stop users from doing silly things. The CLI is there. The problem is that virtually anything you do from the CLI that isn't just observing the status of things (camcontrol, smartctl, zpool status) can create problem. FreeNAS is literally FreeBSD with iXsystem's custom UI, a sql database of settings and scripts that execute based on the sql database and the UI. Start deviating from that and the database ends up not matching what you want, and the UI ends up with inputs and outputs that don't relate to the real status of the server.

"Off nominal" is too vague to even try to discuss. FreeNAS is an appliance. Just like your dishwasher. I'm willing to bet most people don't get upset because the dishwasher can't do laundry. The dishwasher has limitations. It can only wash certain things well, and pretty much nothing else. FreeNAS is the same way. If you use it exactly how it is laid out in the manual, things work out great. If you aren't happy with the limitations, you can always "wash your own dishes". FreeBSD gives you incredible power and responsibility. You can literally do just about any weird discombobulated thing you want in FreeBSD and its on you to figure out how to make it all work. FreeNAS expects *everything* to be managed from the WebGUI and controlled by the SQL database. FreeNAS makes a crapload of assumptions for you. That's why its easy. If you are wanting something more complex than FreeNAS offers, FreeBSD is an option (and something I'd *strongly* recommend if you want to go this path). The last thing you want is to have some WebGUI script/command go sideways and eat a pool, files, datasets, etc because it had no clue that what it thought the status of the server was and what the actual status was weren't the same.

It is unfortunate, because I think FreeNAS is great and I love what they are trying to do, but it has to be more robust to off-nominal situations. Another great example is memory -- I've read on these forums time and again that using insufficient memory can destroy your data. This is ridiculous. If the system doesn't have enough memory, it should fail gracefully: "help, I'm dyi--!", not eat your data on it's way down. This sounds like a nasty unresolved bug. Sure, the correct answer is to throw more memory at it, but the punishment for using insufficient memory should be failure to operate, not data loss.

I agree. It would be great if it failed gracefully. The problem is two fold:

1. We don't want to hard code it so that if you don't have 8GB of RAM the system starts booting, then give you the middle finger and tells you where to go. Windows does this during the installer if you have insufficient RAM and it sucks.
2. Because so many tasks already run in RAM and can't really be swapped out to disk, there isn't much leeway between "i'm on the cusp of being unstable" and "I can't even tell which way is up".

In today's world, 8GB of RAM really isn't that much. It's also a problem because even if we got it stable with 4GB, then people wouldn't be complaining because it's unstable. They'd be complaining because its too slow. And there's no way on God's green earth that can be fixed without more RAM. So at some point you have to decide "eh, this is just not worth working on as the number of users that would be okay with those circumstances anyway are too small to be concerned with". You won't even find TrueNAS boxes with less than 48GB of RAM, which is what brings in the money to fund development of FreeNAS. So I bet you can imagine how much interest there is with trying to make FreeNAS 100% stable and fast with "just" 8GB of RAM.

Sorry, I'm ranting now and this is getting out of hand. I'm not trying to attack anyone, I just think that FreeNAS could benefit from a) users that are less tolerant of weird unexplained behaviour, and b) more robust system design (here the system is the FreeNAS stack, not the system (computer) in your closet).

This isn't a user problem and its not because we aren't as tolerant as we could be. This is strictly a technical problem. The technical aspect is that FreeNAS is not designed to let you control every little thing from the CLI and be able to handle that in some graceful situation. We'd need MANY more developers for that to even try to tackle that problem, and unless you want to start paying those developer salaries, you kind of "get what you get".

You are welcome to do what you want with the OS. As one of the devs say, if you are unhappy with the OS you can request a 100% refund of the purchase price.

For the simpletons that want a WebGUI for basic functions and use it in that capacity, FreeNAS is amazing. I know I couldn't have jumped into ZFS when I did without some simplifying down thanks to the WebGUI. For those that want a more controllable OS, FreeBSD is there and is the same price to download and install.

TrueNAS is literally FreeNAS with a few bells and whistles. Companies all over the world use TrueNASes without data loss. Why? Because they use the product exactly as designed. Home users have these crazy alternative ideas for what they want to do. They want it dirt cheap (or free) and they expect the world. Well, that's just not reality. It takes money to get performance (and reliability).

 

[David123]

The short answer, you don't do what you are doing. What you are doing is very bad news for FreeNAS. It's an appliance. As such, you should use it as designed or you can have major problems (even sudden data loss).

This is such a bad idea I was hesitant to even tell you its a bad idea. From experience I've learned that random forum users take my reply as endorsing an idea. I don't.

Even if you fixed your problems right now, you aren't going to know what has or hasn't changed with an update (and those come twice a week). So unless you plan to audit every change made to the FreeNAS code, this is a *very* bad idea. And I'm sure the answers are in the code, and since you are asking those questions I'd be willing to bet you aren't experienced enough to be able to audit the FreeNAS code.

Dear Cyberjock,

Thanks for your answer which make me realize that I was totally wrong in my way to proceed.
You're right, a script like this will be a nightmare to keep updated and the risk to break things is high !
The best way to proceed would be to use an API... and I realized there is a FreeNAS API !!!

I just tried it on my lab and it's working like a charm !

I post my Python test code hoping it will help someone oneday :)

Code:

import requests
import json

req = requests.post(
        'http://MYNAS.MYDOMAIN/api/v1.0/storage/volume/MY-ZPOOL-NAME/unlock/',
        auth=('MYLOGIN', 'MYPASSWORD'),
        headers={'Content-Type': 'application/json'},
        verify=False,
        data=json.dumps({'passphrase' : 'MY-PASSPHRASE'}),
)
print (req.text)

 

[cyberjock]

Haha. Right after I clicked submit I went looking for the API stuff because I was going to provide you with the answer... but you found it yourself.

API is definitely endorsed by me. :)

 

[Tywin]

"Off nominal" is too vague to even try to discuss. FreeNAS is an appliance. Just like your dishwasher. I'm willing to bet most people don't get upset because the dishwasher can't do laundry. The dishwasher has limitations. It can only wash certain things well, and pretty much nothing else. FreeNAS is the same way. If you use it exactly how it is laid out in the manual, things work out great. If you aren't happy with the limitations, you can always "wash your own dishes". FreeBSD gives you incredible power and responsibility. You can literally do just about any weird discombobulated thing you want in FreeBSD and its on you to figure out how to make it all work. FreeNAS expects *everything* to be managed from the WebGUI and controlled by the SQL database. FreeNAS makes a crapload of assumptions for you. That's why its easy. If you are wanting something more complex than FreeNAS offers, FreeBSD is an option (and something I'd *strongly* recommend if you want to go this path). The last thing you want is to have some WebGUI script/command go sideways and eat a pool, files, datasets, etc because it had no clue that what it thought the status of the server was and what the actual status was weren't the same.

Sure, my dishwasher can't do my laundry. But if I forgot to put dish detergent in before running a load, I would be pretty upset if my dishwasher shredded all my dishes. Worst case scenario should be (and for all dishwashers I've seen, is) that dishes come out still dirty, if slightly rinsed. I've also never had to use the manual for my dishwasher after installation, or for the odd piece of maintenance here and there. The interface is intuitive enough to use on it's own, and robust enough to keep me from shredding my dishes when I'm groggy in the morning.

I agree. It would be great if it failed gracefully. The problem is two fold:

1. We don't want to hard code it so that if you don't have 8GB of RAM the system starts booting, then give you the middle finger and tells you where to go. Windows does this during the installer if you have insufficient RAM and it sucks.
2. Because so many tasks already run in RAM and can't really be swapped out to disk, there isn't much leeway between "i'm on the cusp of being unstable" and "I can't even tell which way is up".

In today's world, 8GB of RAM really isn't that much. It's also a problem because even if we got it stable with 4GB, then people wouldn't be complaining because it's unstable. They'd be complaining because its too slow. And there's no way on God's green earth that can be fixed without more RAM. So at some point you have to decide "eh, this is just not worth working on as the number of users that would be okay with those circumstances anyway are too small to be concerned with". You won't even find TrueNAS boxes with less than 48GB of RAM, which is what brings in the money to fund development of FreeNAS. So I bet you can imagine how much interest there is with trying to make FreeNAS 100% stable and fast with "just" 8GB of RAM.

I think hard-coded limits of any kind are counter-productive. If your system is designed to handle the things you actually care about -- like running out of memory -- gracefully, then there's no reason not to let it do it's best effort on what is available. If that turns out to be insufficient for a user's use-case, they deal with it and move on, without having lost their data. That is is able to go completely tits-up suggests a fundamental defect in the design, especially given that most of the RAM is dedicated to cache. The ZFS subsystem is directly managing that, I don't see why it has to assume anything about its state -- it should know it. Perhaps I'm simply insufficiently educated on the subject -- not beyond the realm of the possible ;) -- but from a high-level, the worst-case for insufficient memory should be some equivalent of thrashing, resulting in terrible performance and setting off alarms in the GUI saying "hey bro, I need more RAM :(". That said, I do understand that memory management is an incredibly difficult task and that very few systems handle it well. But in the case of ZFS, the priority should be "don't drop the pool! don't drop the pool!", not "shit guys, because the OOM is coming, BLOW UP OUR SHIT!" (sorry, I'm tired today and feeling a little under the weather and giddy ;).

This isn't a user problem and its not because we aren't as tolerant as we could be. This is strictly a technical problem. The technical aspect is that FreeNAS is not designed to let you control every little thing from the CLI and be able to handle that in some graceful situation. We'd need MANY more developers for that to even try to tackle that problem, and unless you want to start paying those developer salaries, you kind of "get what you get".

You are welcome to do what you want with the OS. As one of the devs say, if you are unhappy with the OS you can request a 100% refund of the purchase price.

For the simpletons that want a WebGUI for basic functions and use it in that capacity, FreeNAS is amazing. I know I couldn't have jumped into ZFS when I did without some simplifying down thanks to the WebGUI. For those that want a more controllable OS, FreeBSD is there and is the same price to download and install.

TrueNAS is literally FreeNAS with a few bells and whistles. Companies all over the world use TrueNASes without data loss. Why? Because they use the product exactly as designed. Home users have these crazy alternative ideas for what they want to do. They want it dirt cheap (or free) and they expect the world. Well, that's just not reality. It takes money to get performance (and reliability).

Totally hear you here. If anything, I think FreeNAS could do with a little less controllability (I think they've been going that direction, no? aren't tunables gone now?). I can't adjust the water temperature or the individual cycle times on my dishwasher beyond the defaults, but I can cram as many dishes as I want into it. Some might not get clean, but the worst case then is I wash them manually or re-do them, they don't get shredded by my dishwasher (I'm really trying to drive this point, because it's my main one -- data loss should be so rare it's almost never heard of, and when it is, it's a Big Deal in the community; the "negative side effects" of poor operational choices by users should be "that just straight up won't work" or "you won't get halfway decent performance out of that".

As for the OS, your cautionary tales are one of the main reasons I haven't switched to FreeNAS already. My current solution is no better, in fact objectively worse (Intel RAID5 with NTFS on top), but it is more reliable insofar as I know what the failure modes are. I worry about silent bit-rot, but I keep checksums of the stuff I really care about locally and in backups; annoying manual process, but it works. On the other hand, I don't worry about it completely swallowing itself one day for no apparent reason. I know most of the ways I can kill it, and they all require some hoop-jumping to get there.

 

[cyberjock]

I agree on the gracefully recovering from out of memory conditions. I tried to push to have this problem identified when the issue was a MUCH bigger and more frequent problem (happened almost every other day). The problem was that nobody could provide a reproduction case, and the cost of trying to find it for a problem that TrueNAS users (the paying customers use) will never see makes it really hard to justify. It really was more of a business decision than anything else. It also seems to only happen on low RAM conditions, and not every time either. We had users that had 1/2 the minimum required and they'd just reboot the box and things kept going. They might reboot the box 100+ times without a problem over the course of a year or more. Then that one day the server decides its had enough and munches on a zpool for breakfast. Trust me, I *loathe* the fact that I say "yep, insufficient RAM eats data for breakfast and you'd better believe me" and not have an actual technical reason. I come from a industry (nuclear power) where you do NOT say "x happens just because, now accept it". You'll quickly end up unemployed if you are saying that kind of thing on engineering documentation.

As for your dishwasher analogy, I think instead of dish detergent it's more like not hooking up the water. The dishwasher won't do much of anything except ruin itself.

For me, every time I see someone talk about "I lost my pool, how do I recover" I take it very seriously. I want to know what went wrong because *I* don't want to be the guy that see's that bug next. It's *extremely* rare that I see people use the right hardware and do everything else right and still lose their data. You will see me in every one of those threads. I'll often stick in them until I see they broke one of the known causes of serious problem (below).

iXsystems take it pretty seriously too. If even 5% of our customers called us with MIA pools iXsystems wouldn't be able to handle the workload.

For those that lose their data, it's almost always:

- Really crappy choice of hardware (especially with hardware RAID controllers, USB drives, and non-ECC RAM).
- Virtualizing FreeNAS
- Running RAIDZ1
- Insufficient RAM
- Not turning on SMART monitoring and setting up the email alert system
- Not doing regular scrubs
- Not taking warnings about impeding disk failure seriously

I think 99% of all of the people I've seen that lost data broke at least 2 of those, probably 75% broke at least 3, and probably 60% have broken at least 4.

The last person I did recovery for had a hardware RAID controller, 6GB of RAM, and it was an AMD desktop system. No scrubs were setup (he actually deleted the one that was created by default because it slowed down the server too much), he had no SMART monitoring or SMART testing, and he had no emailing setup. He did have the sense to have a RAIDZ2, but he had 4 disks failing. Poof, he just met the requirements of 4 of those. People don't think ahead to how to be proactive and identify failures ASAP. No joke, if you were to pull out one of my drives in my server right now, I'd have a text message on my phone within 30 minutes. Within an hour I'd have an automatically generated email to every one of my email boxes (and my cell phone as a text message), and every 24 hours my custom script would give me a full analysis of my server, so I can look back historically to see what was going on when.

Do you know how often I log into my server to check its status? NEVER. I don't feel like I have to. My server has *plenty of ways to contact me, and it does when it needs to. My friends that have FreeNAS servers built by me... how often do you think they log into their servers? Yep.. never. Why? They get emails too (and I get cc'd them automatically when their server so much as sneezes).

If you take all of the recommendations that the more experienced users give as requirements to build your server and "do this and nothing else" you are SO much better off. In all seriousness, if you buy everything that we recommend in the forum and set it up like we recommend and don't try to play games with FreeNAS, you're just a step or two below the quality TrueNAS provides. Seriously. The problem is that when people seek out a file server, they want something that is an all-in-one package. They want to virtualize a bunch of different tasks, they want a file server that is rock solid, they want to build it with the cheapest parts they can find on Newegg, they want it to be infinitely fast. They expect the world, but that's not what FreeNAS is about.

If you want to go with some thing that is an all-in-one, it's pretty expensive. I have a two-in-one. I run a FreeNAS box and an ESXi host. Between those two I can do everything my heart dreams up. But 99% of people that want the all-in-one are NOT going to drop the cash for two systems, they aren't even going to drop the cash for anything but the crappiest hardware they can find in Newegg, they're going to ignore every single recommendation we provide that involves spending more than they decided on before they found FreeNAS, and that's that. You just can't fix stupid. ;)