Plex plugin using mount point with symlinks questions.

[Andrew Ostrom]

I don't have a strong Unix background, and I've had my FreeNAS running for about a month, so I'm unsure about how to best configure lots of things. What I'm struggling with now is how to best set up the file system to do what I need it to do for my Plex(pass) server. So, I'm looking for advice on how to make this work - I'll recount what I've tried and why it's not really what I want.

I have a variety of datasets on my FreeNAS, and want to create a Plex library that combines directories from multiple datasets. As a somewhat simplified example, let's say that I have three datasets:

Code:

/mnt/pool1/a
/mnt/pool1/b
/mnt/pool1/c

Each one of these has dozens or hundreds of sub-folders and hundreds to thousands of files in each folder. My requirement is to have a Plex library that contains the contents of a few of these sub-folders. For this example I'll say I want to have the following directories combined into a Plex library:

Code:

/mnt/pool1/a/foo
/mnt/pool1/a/bar
/mnt/pool1/b/fubar
/mnt/pool1/c/afu

I have created mount points in my Plex Jail for all three datasets, "a", "b", "c". Adding each of the directories to a plex library works fine; I then have all the files I want to be able to see available inside the library. Unfortunately, what happens is that all the folders are effectively combined into one superfolder, and I can't browse the contents by folder inside the Plex player. I can't add the dataset root directories to Plex, that would be orders of magnitude the number of files and many, many more TB of data. I know I can create collections, but then every time the contents changed I'd have to manually add the new files to the collection; it seems like a maintenance nightmare.

Is there a way to use Unix symlinks (or some other mechanism) to make this work? I have created another dataset, let's call it "/mnt/pool1/plex". If I use "cp" to copy the directories I want into the new dataset everything works great. BUT - of course I am now consuming lots of extra pool space, and I have the need to sync the directories on an ongoing basis. It seems to me that there should be some mechanism to have the "plex" dataset populated with directories that are links to the "real" data in the other datasets. After many hours of searching this site, and Google, and reading a lot of man pages and other material I think I'm just more confused than when I began. I'm sure this is a lot easier than I'm finding it, it's just my lack of Unix experience that's making it hard to figure out. A few experiments I've made with symlinks have been abject failures.

 

[garm]

Why? Why do you think anyone at the FreeNAS/iX forum would give you a better answer then someone on say, the plex forum?! https://forums.plex.tv/

But here goes..
I don’t use plex so I don’t know what or what not it can do, but I work with CAD enterprise solutions that are not totally dissimilar and I can read. It looks like it can create “libraries” of content indexed from several locations, just like you describe.
https://support.plex.tv/articles/200288916-overview/

Steve Jobs waged a war on the file system as the “face of computers”, trying to shift focus from “files” to applications. https://oleb.net/blog/2012/06/steve-jobs-on-the-file-system/

I have been doing the same personally for the last two decade or so. Managing files in folder structures is ineffective and complex. You are limited to very basic sorting mechanisms and basically no version control.

On my FreeNAS I do have NFS file shares because some applications I use is for sure file based, like https://hledger.org/, but then I use git to manage the folder.

But the builk of my content is either in software like Lightroom or stored/shared in Nextcloud. Working on raw files is simply to dangerous, even on ZFS, you need a database of some kind.

So what I’m saying is let go of the damaged view Microsoft have given you of computers and use Plex as the indexer of files, the information you tried to build in the folder structure should be built in plex..

 

[Andrew Ostrom]

Why? Why do you think anyone at the FreeNAS/iX forum would give you a better answer then someone on say, the plex forum?! https://forums.plex.tv/

But here goes..
I don’t use plex so I don’t know what or what not it can do, but I work with CAD enterprise solutions that are not totally dissimilar and I can read. It looks like it can create “libraries” of content indexed from several locations, just like you describe.
https://support.plex.tv/articles/200288916-overview/

Steve Jobs waged a war on the file system as the “face of computers”, trying to shift focus from “files” to applications. https://oleb.net/blog/2012/06/steve-jobs-on-the-file-system/

I have been doing the same personally for the last two decade or so. Managing files in folder structures is ineffective and complex. You are limited to very basic sorting mechanisms and basically no version control.

On my FreeNAS I do have NFS file shares because some applications I use is for sure file based, like https://hledger.org/, but then I use git to manage the folder.

But the builk of my content is either in software like Lightroom or stored/shared in Nextcloud. Working on raw files is simply to dangerous, even on ZFS, you need a database of some kind.

So what I’m saying is let go of the damaged view Microsoft have given you of computers and use Plex as the indexer of files, the information you tried to build in the folder structure should be built in plex..

Wow! Just a little supercilious (you can look it up) and condescending aren't you...

But - to address your issues:

1) This really has nothing to do with Plex, so why should I use their forums? This is really about the ability to construct/pass symbolic links from the FreeNAS environment into the Jail housing Plex. This whole concept of "jails" is new to me, and although I sort of understand the thinking behind them I find their capabilities, and configuration nuances beyond my experience.

You say you've been doing this for two decades or so -- I probably started doing this professionally before you were born. I've been in the computer industry since 1979 as a programmer, system manager (now folks call it sysadmin), data center manager, network manager, marketeer, product manager, and more. I've managed lines of business worth several hundred million dollars a year and I've seen technologies, products, and companies come and go, but disks, directories, and files have been there all along. I don't see that changing any time soon. I have files that originated on RSTS/E in 1980, or so, that were transferred to VMS, then to Windows-NT, and onward. I can still use those files today, and they'll be useable long after git, plex, FreeBSD and Unix are consigned to the computer museums.

2) I don't see what Steve Jobs (or even Microsoft) has to do with this at all. OK, you hate Jobs/Microsoft - that's your problem. AND -- by the way -- virtually none of the underlying technologies or concepts were developed either by Microsoft or Apple, or whoever else you hate. Most of the way we interact with filesystems, in both Windows and the various Unix/Linux environments, is almost completely based on the operating systems developed at Digital Equipment in the 1960s, 70s, and 80s. I know -- I was there, managing the PDP-11 and VAX-11 systems we used to run a $13Billion company in the late 70s, through the mid 90s. The technologies have been improved and extended, but the basic structure of disks/volumes/directories/files is still the basis.

3) Why on earth would I want Plex to manage the 1 million (round numbers) of files I have on my server when I have spent decades organizing them in a way that makes sense to me? And, when I want a few hundred to a few thousand to be available through Plex? You say that filesystems are archaic and vulnerable, but I've seen plenty of corrupt databases in my day, and the performance impact is rediculous. Why would I want all my organization structure to be maintained in some proprietary, non-portable form? This is just insane. You do realize that filesystems are just another form of database, right? Why is the git or Lightroom (which I use) or Plex database superior?

Since you 1) don't use Plex, and 2) had nothing useful, or positive, to contribute, why did you feel compelled to jump in here, just to be an ass?

 

[anmnz]

I have created mount points in my Plex Jail for all three datasets, "a", "b", "c". Adding each of the directories to a plex library works fine; I then have all the files I want to be able to see available inside the library. Unfortunately, what happens is that all the folders are effectively combined into one superfolder, and I can't browse the contents by folder inside the Plex player.

Does it help if you add a parent of the datasets' mount points to the Plex library instead? E.g. mount the datasets at /mnt/media/a, /mnt/media/b, etc., and add /mnt/media to the Plex library? (Don't know how this would work, I believe there's been noise in recent times about Plex "folder view" changing, moving, disappearing, reappearing and generally not making its users all that happy.)

Failing that, I would agree with @garm that the Plex forums may be a better bet than these forums for questions about how to organize files / directories / symlinks / filesystems / mount points / etc. so that Plex will present them as you want.

 

[garm]

Since you 1) don't use Plex, and 2) had nothing useful, or positive, to contribute, why did you feel compelled to jump in here, just to be an ass?

Because all those reasons you just spelled out for us, you need to break out of your set ways and embrace the new millennium. It’s not tape decks and analog modems any more.

I never said files where archaic, users direct interaction with files are. It’s simply too limited. I work at a 100k+ multinational company and we have a looong tradition of data management. We also have a culture war between your generation any my own. We have a management layer who probably read books written in the 1930s who view the world in terms of word documents (typewriters) and excel spread sheets (ledgers). Then we have a generation who where breed from when we started using databases back in the 1970s and subsequently started to change the meta from documents stored physically on microfiche to data managed in databases (and backed up on microfiche). We managed a switch from corporate wide file storage on an elaborate folder structure spanning a multitude of servers in several countries based on Unix back in the 90s to database driven solutions. And we still struggle with this kind of mindset, that the files are somehow superior when they sit on a file system and users interact with them directly through an editor contrary to users having their editor hooked up to a database where changes are tracked and questions can be asked.

Sure, plex is a very light version of this war. But nevertheless you come here and ask how to map your inferior folder structure to a database and I will tell you your premise is wrong. You don’t map the folder structure to the application, you let the application determine the appropriate folder structure. Personally i view “folders” as close to meaningless entities in regards of user experience, at best a virtual abstraction layer in an application. I don’t create folders on my servers, I create datasets based on access rights, backup meta and application content. I let Lightroom create folders for dates pictures are taken for manual disaster recovery, but i never touch the files "manually". I use clear text files as far as it’s possible for the same reason, but I still use applications and databases for managing their content and I set up storage “vaults” for those applications as datasets.

If I want to know how much I spent on fuel on my vacation in 2009 I don’t open an insanely complex and brittle file such as an excel workbook stored in a folder called "2009", I type the question hledger balance expenses -p “2009” —pivot vacation and hledger reads a text file and tells me the result. Likewise if I write minutes for our housing association I run Libreoffice in Nextcloud and save the finished document as pdf and keep the editable document in odt, nextcloud manages versions of this file along with other metadata. All this is of course stored on ZFS datasets and volumes, but I don't manage folders and files other then for system administration. And even then I tend to use scripts and databases (git) for tracking changes and (re)building systems.

 

[Andrew Ostrom]

Does it help if you add a parent of the datasets' mount points to the Plex library instead? E.g. mount the datasets at /mnt/media/a, /mnt/media/b, etc., and add /mnt/media to the Plex library? (Don't know how this would work, I believe there's been noise in recent times about Plex "folder view" changing, moving, disappearing, reappearing and generally not making its users all that happy.)

Failing that, I would agree with @garm that the Plex forums may be a better bet than these forums for questions about how to organize files / directories / symlinks / filesystems / mount points / etc. so that Plex will present them as you want.

Thanks for being constructive. So far the Plex folder view has been completely stable for me; to paraphrase Tom Lehrer - "It's like a sewer - what I get out of it depends on what I put into it."

I have added the the "parent" mount points to the jail. But, if I then add the actual folders to the library all the files are lumped together into one mess when I access that library in Plex. Naturally, if I add the "parents" to the Plex library I get 1000x the number of files in the library that I don't want.

The goal was to find a simple way to have a subset of my personal media available inside Plex without having multiple copies on the NAS. Since these are all personal files, this is all stuff for which Plex would be unable to create metadata unless I added it myself, or hand-built collections. I'm trying to minimize the manual work - I still have tens of thousands of slides and negatives to scan and just organizing then into folders as I scan them is time consuming enough. Some slides have the month/year on them, but the ones I developed myself, and all the negative's don't. So, when I scan I look at the first slide/negative in a roll of film, create a folder with the date/subject embedded, and let it rip. (I got my first SLR in 1969 - I got my first DSLR in 2005. I'm guessing I shot 2 to 3 36 frame rolls of film a week, minimum, so 35 years at 72 to 108 images a week = 135K to 200K images. That feels about right - MANY binders of negatives and trays and cartons of boxes of slides.)

What I'm trying to understand is where/how I try doing this with symbolic links, and even if that will work. Do I just log into root via ssh, create the links, and then links added outside the jail appear inside? Do I have to add them inside the jail, and if so, how do i do that? Are they persistent across reboots (like in Windows), or are they just kept in memory and have to be created at boot time (like VMS)? What commands do i use to create the links (ln -s, I assume). All these are things I'm hoping some FreeNAS person has done before and can just say, "here's the best way to do this."

Thanks.

 

[Andrew Ostrom]

Because all those reasons you just spelled out for us, you need to break out of your set ways and embrace the new millennium. It’s not tape decks and analog modems any more.

I never said files where archaic, users direct interaction with files are. It’s simply too limited. I work at a 100k+ multinational company and we have a looong tradition of data management. We also have a culture war between your generation any my own. We have a management layer who probably read books written in the 1930s who view the world in terms of word documents (typewriters) and excel spread sheets (ledgers). Then we have a generation who where breed from when we started using databases back in the 1970s and subsequently started to change the meta from documents stored physically on microfiche to data managed in databases (and backed up on microfiche). We managed a switch from corporate wide file storage on an elaborate folder structure spanning a multitude of servers in several countries based on Unix back in the 90s to database driven solutions. And we still struggle with this kind of mindset, that the files are somehow superior when they sit on a file system and users interact with them directly through an editor contrary to users having their editor hooked up to a database where changes are tracked and questions can be asked.

Sure, plex is a very light version of this war. But nevertheless you come here and ask how to map your inferior folder structure to a database and I will tell you your premise is wrong. You don’t map the folder structure to the application, you let the application determine the appropriate folder structure. Personally i view “folders” as close to meaningless entities in regards of user experience, at best a virtual abstraction layer in an application. I don’t create folders on my servers, I create datasets based on access rights, backup meta and application content. I let Lightroom create folders for dates pictures are taken for manual disaster recovery, but i never touch the files "manually". I use clear text files as far as it’s possible for the same reason, but I still use applications and databases for managing their content and I set up storage “vaults” for those applications as datasets.

If I want to know how much I spent on fuel on my vacation in 2009 I don’t open an insanely complex and brittle file such as an excel workbook stored in a folder called "2009", I type the question hledger balance expenses -p “2009” —pivot vacation and hledger reads a text file and tells me the result. Likewise if I write minutes for our housing association I run Libreoffice in Nextcloud and save the finished document as pdf and keep the editable document in odt, nextcloud manages versions of this file along with other metadata. All this is of course stored on ZFS datasets and volumes, but I don't manage folders and files other then for system administration. And even then I tend to use scripts and databases (git) for tracking changes and (re)building systems.

Somehow I doubt that you manage every file you have by lumping them into one single directory and access EVERYTHING through git. And, if you do I think you'll be sorry when some git bug corrupts that database and you lose the ability to access anything. Why does EVERY operating system organize their files into (Horrors!) directories? Couldn't Windows or BSD or Linux just put everything into one big directory? Why does EVERY software product in the world use a directory structure to organize its contents? Why not just have your system with one directory at the root of the boot drive and just dump everything into it?

I hate to break your stereotyped (and sad) view of the world, but I was a leader in adopting database technology when it was first created; I managed engineering, product management & product marketing of AI technologies at one of the TRUE pioneers in the space (DEC), way back in 1991, and have pushed many organizations to stop building web sites with files and instead to move to database-oriented systems systems like Drupal (which still stores images, etc. in directories). But I also know the limitations of technology (look at the archaic UI of Unix/Linux, for example), and don't feel the need to use a a technology that's not needed or intend to become the slave to them. (And I certainly NEVER add up what I spend on vacation!)

 

[Andrew Ostrom]

I can't get symlinks to function at all. I read: https://www.ixsystems.com/community/threads/questions-about-symbolic-links.56939/ but that doesn't seem to help.

To try it I logged into root via SSH and did this:

Code:

root@freenas[~]# cd /mnt/BigPool
root@freenas[/mnt/BigPool]# ls
backups docs    email   home    iocage  kits    music   photos  stuff   tv      video   vids    work    xfr
root@freenas[/mnt/BigPool]# ls /mnt/BigPool/stuff/Photos
TOPGUN sign at entrance.JPG
root@freenas[/mnt/BigPool]# ls /mnt/BigPool/photos/Photographs/2018
01 - January                            05 - May                                Alex & Lucy's House
02 - February                           06 - June                               August & September (TOPGUN, Yosemite)
03 - March                              07 - July
04 - April                              11 - November
root@freenas[/mnt/BigPool]# ln -sv /mnt/BigPool/stuff/Photos/2018 /mnt/BigPool/photos/Photographs/2018
/mnt/BigPool/photos/Photographs/2018/2018 -> /mnt/BigPool/stuff/Photos/2018
root@freenas[/mnt/BigPool]# ls /mnt/BigPool/stuff/Photos/2018
ls: /mnt/BigPool/stuff/Photos/2018: No such file or directory
root@freenas[/mnt/BigPool]# ls /mnt/BigPool/stuff/Photos
TOPGUN sign at entrance.JPG

It SAYS it's creating the link, but then it's not there...

 

[anmnz]

What I'm trying to understand is where/how I try doing this with symbolic links, and even if that will work. Do I just log into root via ssh, create the links, and then links added outside the jail appear inside? Do I have to add them inside the jail, and if so, how do i do that? Are they persistent across reboots (like in Windows), or are they just kept in memory and have to be created at boot time (like VMS)? What commands do i use to create the links (ln -s, I assume).

This might help, if you're unfamiliar with Unix. A symbolic link is just the "simplest thing that could possibly work" -- it's essentially a file in the filesystem whose content is a string of text, which is interpreted as the path that the link points to. (The file is specially flagged as a symlink and the kernel handles resolving the link when the application tries to open it.)

You can create a symlink from inside or outside the jail, it makes no difference as long as you're creating the link in what is actually the same place in the filesystem. (For example running "ln -s foo /path/to/bar" inside the jail and something like "ln -s foo /mnt/iocage/jails/myjail/root/path/to/bar" outside the jail would have exactly the same effect,)

The key thing to understand w.r.t symlinks and jails is that the content of the link is really just a string, it doesn't really contain a reference to any other file, and the target of the symlink will be resolved according to the view of the filesystem that the opening process has. So a simple relative link ("ln -s foo bar") will likely be resolved to the same file by processes in or out of the jail, while an absolute link ("ln -s /path/to/foo bar") will almost certainly be resolved differently. If that makes sense?

"ln -s" is the usual way to create symlinks but a nice alternative can be "install -l sr", which instead of making you specify the content of the symlink, figures out what the content of a (relative) symlink needs to be in order to make it point to a given file. I find this useful for making symlinks that resolve to the same place in and out of jails. Hope that helps you figure it out a bit...

 

[anmnz]

It SAYS it's creating the link, but then it's not there...

I think you have the arguments to "ln -s" backwards... "ln -s foo bar" creates a link named bar pointing at file foo, not vice versa. (Also note the target of the link does not need to exist.)

 

[sretalla]

The problem with all this is that nullfs mounts don't follow symlinks, nor can they traverse dataset boundaries.

You need to mount all the datasets directly to the places you want them in the jails you use them in.

 

[Andrew Ostrom]

This might help, if you're unfamiliar with Unix. A symbolic link is just the "simplest thing that could possibly work" -- it's essentially a file in the filesystem whose content is a string of text, which is interpreted as the path that the link points to. (The file is specially flagged as a symlink and the kernel handles resolving the link when the application tries to open it.)

You can create a symlink from inside or outside the jail, it makes no difference as long as you're creating the link in what is actually the same place in the filesystem. (For example running "ln -s foo /path/to/bar" inside the jail and something like "ln -s foo /mnt/iocage/jails/myjail/root/path/to/bar" outside the jail would have exactly the same effect,)

The key thing to understand w.r.t symlinks and jails is that the content of the link is really just a string, it doesn't really contain a reference to any other file, and the target of the symlink will be resolved according to the view of the filesystem that the opening process has. So a simple relative link ("ln -s foo bar") will likely be resolved to the same file by processes in or out of the jail, while an absolute link ("ln -s /path/to/foo bar") will almost certainly be resolved differently. If that makes sense?

"ln -s" is the usual way to create symlinks but a nice alternative can be "install -l sr", which instead of making you specify the content of the symlink, figures out what the content of a (relative) symlink needs to be in order to make it point to a given file. I find this useful for making symlinks that resolve to the same place in and out of jails. Hope that helps you figure it out a bit...

That helps, thanks. It's pretty much what I thought, though it had not occurred to me that an absolute link wouldn't work inside the jail because the mount point is different than the absolute reference. I'll experiment both with "ln -s" and "install -l sr" to see if I can get it to work for me.

 

[Andrew Ostrom]

The problem with all this is that nullfs mounts don't follow symlinks, nor can they traverse dataset boundaries.

You need to mount all the datasets directly to the places you want them in the jails you use them in.

Thanks. I have all the datasets mounted in the jail. I'm sure your comments about nullfs mounts not following symlinks is the issue, but even after reading a bunch of stuff on the web, like:
https://forums.freebsd.org/threads/how-can-i-mount-a-nullfs-inside-a-jail-iocage.67891/
https://www.ixsystems.com/community...t-fstab-store-not-an-absolute-pathname.61520/
https://www.ixsystems.com/community/threads/nullfs-freenas-and-right-approach.54113/
but honestly it makes me more confused, not less.

 

[garm]

Nullfs is what makes it possible to “mount” a dataset to a jail. It essentially mounts an arbitrary folder to another, making traversing the jails isolation possible without having to mount the dataset directly in the jail. If you want a subset of the content of a folder to be accessible to a user (every service runs by its own user, “plex” in this case) you have few options.. to automate anything you need to think about the abstraction layer you could manage. For instance, you could have different groups owning different folders in your dataset. This is a nightmare to manage. A large part of my profession is managing content access and their underlying policy. We never set permissions on folder level, instead we have a group schema where each equivalent of a dataset gets groups assigned. So each group needing access to a specific set of content is assigned to that contents dataset.

This is why I said that the folder structure shouldn’t be governed by anything other then application needs. You’ll spend your life tinkering with folders..

Nullfs makes links hard to deal with as it is isolated from the rest of the filesystem.

My own Lightroom workflow is that I have the raw files in a date based folder tree managed by Lightroom. I then export “developed” photos to their own dataset (or in some cases Nextcloud) where each export resides in a folder named after the collection being developed (event, album, project etc). By doing this I sever any dependencies between developed photos and raw files. I can still recreate developed photos so I don’t need the same backup schema as raw files and the Lightroom database. And I can copy, share or make developed photos part of what ever service I want without exposing raw files or the database to any unessesary risk. But yes, this depends on there being two files, the raw camera data and the “developed” jpeg.

 

[Andrew Ostrom]

All my attempts at setting up symbolic links have failed to produce the desired end result. Right now I have more available pool space (16TB) than I have available time for experimentation, so I have just set up DirSyncPro (on the Windows side, where I am much more comfortable) to just copy the directories I want to share via Plex to a directory tree inside the jail mount point. It takes me 1 minute (really less) to add or remove a directory to the DirSyncPro job, which runs daily, or on demand. Very much brute force, and not elegant, but it works. Right now I need to spend my time trying to figure out why the Chelsio 10GbE driver keeps crashing my PC! :-(