[Bounty] FUSE Mount within a Jail (encfs / sshfs)

[xioustic]

Hello,

I've spent a few days on this and not too sure how to proceed, so reaching out for help. I'll put a $75 bounty on this issue payable in Bitcoin.

I want to use EncFS (Encrypted Filesystem) within a a jail. EncFS requires the usage of FUSE (Filesystem in Userspace) which is native to FreeBSD 10+ now; related is kernel module fuse.ko and device /dev/fuse.

My use case is running Gogs (basically a Github clone writen in Go) within a jail (this works fine) but I want to keep the application data encrypted at rest when the jail is not being used by using encfs prior to running the Gogs service.

A similar use case which might be more common is the usage of sshfs within a jail, which is a common utility today and also requires fuse. I believe similar steps to getting sshfs working within a jail would parallel the steps needed for getting encfs working within a jail. See manpages here: https://www.freebsd.org/cgi/man.cgi?query=sshfs&manpath=SuSE+Linux/i386+11.3

Progress:
- The fuse kernel module loads at boot without issue per kldstat listing fuse.ko:

Code:

[root@freenas] ~# kldstat | grep fuse.ko
 2	1 0xffffffff81de6000 18210	fuse.ko

- I have fuse ( /dev/fuse) available. Permissions as follows within host:

Code:

[root@freenas] ~# ls -alh /dev/fuse
crw-rw----  1 root  operator   0x56 May  1 15:12 /dev/fuse

- I am able to see /dev/fuse from within the jail:

Code:

[root@freenas] ~# jexec 2 csh
root@gogs:/ # ls -alh /dev/fuse
crw-rw----  1 root  operator   0x56 May  1 15:12 /dev/fuse

- I have EncFS installed within the jail via pkg install fusefs-encfs:

Code:

root@gogs:/ # encfs --version
encfs version 1.9.1

Roadblock:

Code:

root@gogs:~ # encfs ~/.crypt ~/crypt
EncFS Password:
mount_fusefs: /dev/fuse on /root/crypt: Operation not permitted
fuse: failed to mount file system: Operation not permitted
fuse failed.  Common problems:
 - fuse kernel module not installed (modprobe fuse)
 - invalid options -- see usage message

My only additional lead at this point is to use sysctl vfs.usermount=1, but with this enabled or disabled and restarting the jail I still hit the same roadblock above.

I have read conflicting information on adding devfs.rules for 'fuse' and its permissions, and also revolving around the 'operator' group, but no definitive steps on how to proceed. So now I'm reaching out for help.

I'd see if I could use encfs fine on the host machine first but I do not want to go that route anyway: I want the jail to have autonomy over when/how encfs mounts its directory(s).

Thanks for reading and your time.

 

[joeschmuck]

I'm sure someone will help you out without the offer of payment if they can find an answer. Some of us like a challenge, but for me, I'm just too busy right now to setup a test rig to look into this right now.

And thanks for providing a well written posting. The only thing you are missing is what version of FreeNAS you are running.

Also, I took a brief look into the problem (Google) and it looks like it may be a permissions issue, or that is what the search engine comes up with. A few folks have had this type of error but as you said, no definative steps.

Good Luck!

 

[xioustic]

I'm sure someone will help you out without the offer of payment if they can find an answer.

Right, I'm just hoping it might inspire someone to tackle the issue even if it isn't immediately relevant to them. SSHFS and EncFS are very powerful but probably a bit niche, especially inside a Jail.

I've "looked" for an answer all over, I just think there's still some dots that need to be connected that I am just not experienced enough to figure out. I suspect there's a magic chgrp, chmod, mknod, sysctl, etc set of commands that make this work without an issue. Maybe an edit to devfs. I just don't know enough about those systems combined with FreeBSD jails to do it myself.

The only thing you are missing is what version of FreeNAS you are running.

Whoops! FreeNAS GUI lists build as FreeNAS-9.10.2-U3 (e1497f269)

This was originally a no-go issue until FreeNAS became based on FreeBSD 10+ which I believe automatically includes FUSE. As a result I believe that means anyone that wants to work on this needs 9.10 or better; 9.3 is based on FreeBSD 9.x.

This probably also belongs on a FreeBSD forum but I am not sure which would be best to post on.

Thanks!

 

[joeschmuck]

You could manually setup fuse for earlier versions, I know I did a few times for some testing, but that doesn't matter now.

Cheers!

 

[xioustic]

I'll bump bounty to $250 payable in Bitcoin. Just looking for instructions to have SSHFS and/or EncFS working within a jail so that the jail user has full autonomy over the mounting process.

 

[Philippe1987]

I'll bump bounty to $250 payable in Bitcoin. Just looking for instructions to have SSHFS and/or EncFS working within a jail so that the jail user has full autonomy over the mounting process.

Hello,

I'm currently having the exact same issue.
I've been looking for a solution for months but I'm still not able to install fuse within a jail. :(

I really hope someone can help us out...

 

[joeschmuck]

Out of curiosity could you do what you needed in a Linux VM? Does it have to be a freebsd jail or could it be a linux jail?

 

[kaphka]

Out of curiosity could you do what you needed in a Linux VM? Does it have to be a freebsd jail or could it be a linux jail?

I can't speak for the original poster, but I would also like to get FUSE working in a jail. I could use a Linux VM instead, but it seems like a waste of resources (especially dedicated RAM) to run an entire VM for a simple task. I know there probably isn't much difference in practice, but a jail would be more elegant.

Also, as far as I can tell, the current VM solution for FreeNAS is not fully baked yet. For one thing, it seems to be completely undocumented. I'm running an important VirtualBox VM on 9.10.1, and I'm not going to upgrade until I can properly replace that VM.

For what it's worth, I messed around with FUSE for an hour or two today, and I didn't get any further than xioustic did. I've seen no evidence that anyone has gotten FUSE to work in a jail since the upgrade to 9.10.

 

[BobGeorge]

It is problem with file system flags, see this post

https://serverfault.com/questions/849490/permission-denied-fusefs-inside-freebsd-freenas-jail

 

[j0achim]

This issue is driving me insane, I am trying to achieve what OP is doing as well, going through a VM does not work either as you get all sorts of different errors when you try to work with files that are mounted from your VM through NFS to Host, and then mounted back to Jail from Host

 

[paulbaptista]

No updates on this topic? D:

 

[asomers]

Ok, I've got it working now. Is anybody still interested?

 

[kaphka]

Kinda. If only because I had concluded that it was impossible, because of the flag limitation mentioned above. (It's not clear to me if you could just add the jail flag and recompile, or if you'd need to make other changes to the driver, but either way that's enough of a hassle to outweigh the convenience of FUSE for me.)

 

[Gab]

Hello,

If anybody still interested, I have found a way to have EncFS running on FreeNAS 11.

Here is the basic idea:
- you have to recompile EncFS from sources in a jail, and then move the compiled binary outside of the jail to the host system (in /root/bin/encfs for example) and run it from there.

However for the compilation/run to work, you need to update FreeNAS of the Jail to another version due to mismatch issues between the FreeNAS host version and the Jail's FreeBSD version (the issue is about the missing "libdl.so", that some users are already discussing elsewhere on the web).

The limitations are:
- EncFS need to be run outside of the jail, due to the encfs filesystem not being mountable in a jail.
- It need to be run as root, or at least, as a user which is member of the wheel group. This is probably due to limitations of Fuse on FreeBSD (user mounting does not seem to work correctly).
- For some reason, the Encfs process need to be run in foreground (-f option) and not as daemon. Otherwise this is not working, and I have no clues why. But it is easy to trick this limitation: start in the foreground and then put the process in a subshell.

In the end this is still working: I am now able to use encfs coupled w/rsync on my FreeNAS server, and this have been deployed in production

Here it is !

EDIT: typo

 

[asomers]

BTW, everything "just works" on FreeBSD 12. The FUSE daemon can run directly in the jail. I don't know when FreeNAS is planning to rebase to FreeBSD 12, however.

 

[Gab]

By "just works", do you mean that you are able to successfully run encfs from within the jail (= the encrypted folder is successfully mounted in the jail) ?

If so, then that's good news then! Just have to wait FreeNAS will rebase on FreeBSD 12, which will take some time I suppose. In the meantime, on FreeNAS 11 it is still possible use the solution I explained

 

[asomers]

Yes @Gab that's true. The encfs daemon runs within the jail and the source and dest directories are both within the jail too. FreeNAS will probably also need to update its jail manager. Do they use iocage? The next version of iocage will support mounting FUSE within the jail.