Bluefin NFS can't create more than one NFS share and some apps can't connect

[RavenNAS]

Hey all,

Like many after the recent update to 22.12.1, I have been changing over apps to use NFS inside of my SMB share.

Firstly, for some reason if I try to follow the guides and make multiple NFS shares for each folder that I need, it wont work. it seems like I can only create one NFS share per dataset, no matter the directory. So I created one back a file and set each app into its place. (see the jellyfin example)

Second, even though I got it to work for other apps like Jellyfin, I can't get qBittorrent to save to the directory, it will "error" with no data saved.

any ideas?
Thanks

 

[FrostyCat]

Can confirm, cannot add a second NFS share on the same dataset even if it's not part of the same subtree.

If my dataset is for example documents, it seems I cannot share these 2 folders separately:

• /mnt/data/documents/folder1
• /mnt/data/documents/folder2

Bug?

 

[sretalla]

Bug?

No. This is how NFS works. One "share" per filesystem.

But your clients can easily mount at multiple points of that filesystem if you just share the root of it out.

On the other hand... if you want to keep your apps from having access to the same data for some reason... 2 child datasets would be fine.

• /mnt/data/documents/dataset1
• /mnt/data/documents/dataset2

for example.

 

[RavenNAS]

On the other hand... if you want to keep your apps from having access to the same data for some reason... 2 child datasets would be fine.

• /mnt/data/documents/dataset1
• /mnt/data/documents/dataset2

okay that makes sense, I was following this truecharts guide (Link) and they didn't seemnto mention child datasets, looking at the screenshots it looks as easy as adding more NFS shares.

Great, so thats one issue down and one to go, if anyone has an idea for why qBit cant access the share aswell.

Could it be that maybe another app has the entire share locked and I really should just use child datasets?

 

[sretalla]

okay that makes sense, I was following this truecharts guide (Link) and they didn't seemnto mention child datasets, looking at the screenshots it looks as easy as adding more NFS shares.

I'm not sure that's fully clear for you.

A single NFS share at /mnt/data/documents/ can allow different apps to connect to the different paths within it... assuming the share specifies "all dirs" (which is an option in CORE, but I can't see it in SCALE, so would hope it's on by default).

That share can't allow the crossing of filesystem boundaries though, so you want to have subdirectories, not child datasets to make it work like that.

 

[FrostyCat]

No. This is how NFS works. One "share" per filesystem.

But your clients can easily mount at multiple points of that filesystem if you just share the root of it out.

On the other hand... if you want to keep your apps from having access to the same data for some reason... 2 child datasets would be fine.

• /mnt/data/documents/dataset1
• /mnt/data/documents/dataset2

for example.

No, I don't think that's how NFS works and I can prove it. It's probably a a bug or a limitation of how Truenas implements NFS, not NFS itself.

Let's say I have a location with a folder structure like this:

/mnt/data/nvr <- dataset
/mnt/data/nvr/today <- folder (not dataset)
/mnt/data/nvr/month <- folder
/mnt/data/nvr/archives <- folder

Let's say I only want to share today and month and not the whole nvr dataset.

The UI doesn't let me do that, but I can do it by hand very easily (I could even share in-tree if I pass no_subtree_check if I wanted to, which the UI doesn't seem to support).

/etc/exports

Code:

"/mnt/data/nvr/today"\
        127.0.0.1(sec=sys,rw,subtree_check)
"/mnt/data/nvr/month"\
        127.0.0.1(sec=sys,rw,subtree_check)

Mounts

Code:

localhost:/mnt/data/nvr/today on /root/today type nfs4 (rw,relatime,vers=4.2,... snip ...)
localhost:/mnt/data/nvr/month on /root/month type nfs4 (rw,relatime,vers=4.2,... snip ....)

All checks out:

Code:

root@nas ~ # ls -l /root/today
total 1
-rwxrwx--- 1 root root 0 Mar  8 18:45 file1
root@nas ~ # ls -l /root/month
total 1
-rwxrwx--- 1 root root 0 Mar  8 18:45 file-month

 

[sretalla]

No, I don't think that's how NFS works and I can prove it. It's probably a a bug or a limitation of how Truenas implements NFS, not NFS itself.

OK, so what I meant to say was, "That's not how NFS works on TrueNAS"... but I always assume that we're talking about TrueNAS, since it's the forum for TrueNAS here...

I don't do a lot with NFS on other systems (and don't particularly do a lot on TrueNAS either), so I'll take your word for how it works in the somewhat irrelevant context (of non-TrueNAS NFS) you mention.

 

[FrostyCat]

Sorry, I thought you were talking about NFS in itself and not how Truenas does NFS. It makes sense now in the correct context, I still wonder why it’s done this way, probably to encourage use of child datasets maybe?

 

[RavenNAS]

Would ether of you have any idea why some of my apps and use this nfs share and others cannot?

 

[FrostyCat]

What's the error?

Also, if you use these shares for apps only, restrict the shares to 127.0.0.1 only.

 

[RavenNAS]

What's the error?

Also, if you use these shares for apps only, restrict the shares to 127.0.0.1 only.

In the logs it states the path to what is should be saving or accessing. The path looks right but says "permission denied".

Apps is added to the group for the share, so Im not sure what the issue is with qbittorrent. It worked fine before updating and having to use NFS share

 

[FrostyCat]

When you added the apps group, what permissions did you give it? In case of qbittorrent, depending on the configuration, you may have to give it Full Control at the dataset level and apply it recursively (at least if the dataset it not empty) as it may need to move stuff around or create sub-folders and delete old ones.

Something like this (dataset created with SMB type ACLs):

The NFS share itself doesn't have anything special, it's a good idea to limit access to localhost only if the share is there just for apps:

User 127.0.0.1 as server in the app configuration in this case. NFS is mounted by kubelet (part of k3s in this case) which is running on the host, so localhost is always accessible.

 

[RavenNAS]

When you added the apps group

apps has full control, just like you have it.

Also I tired limiting Host to that IP that you listed and every app got stuck in a deploying loop...

Anyways without that, most apps work but qbit still wont. (error: Permission denied)

I may have to bring this to the truecharts discord, since we have found out my issue with NFS shares is normal behavior.

Thank you, and if you have any other ideas tha ks in advance

 

[FrostyCat]

Ok so I just gave it a shot myself, with a setup like this and it seems to work just fine..

Share is: /mnt/data/downloads

Ownership of the qbit folder is root:apps
The /mnt/data/downloads/qbit folder is mounted from 127.0.0.1 under /downloads

The group apps has full control over the /mnt/data/downloads dataset.
Qbittorrent is running with this pod security context:

Everything seems to check out just fine. I did change settings inside Qbittorrent's configuration to download to /downloads so make sure you set the path for you case too.


Edit: I have NFS v4 checked under the NFS server options btw, bit sure if it matters or not, probably not.

 

[RavenNAS]

I appreciate how mich detail you put into your replys. Thank you

Everything seems the same with my setup although I wasn't using NFS v4. I enabled that with no luck. Its still not working.

So it still wont mount or have permission to

 

[RavenNAS]

I'm not sure why if I try to use 127.0.0.1 as the mount address instead of the server IP, none of my apps can connect to NFS

 

[RavenNAS]

with help of truecharts support we found that the mount has to be /downloads. I had changed it and although it worked before the switch to NFS, it seems to be a requirement. now it works