Graphical file manager application/plugin?

[alpha754293]

I was doing a search to see if there was a graphical file manager that, for example, Qnap offers with their NAS units/in their NAS operating system and so far, I haven't really been able to find one following this thread: https://www.truenas.com/community/threads/would-like-a-graphical-file-manager.85672/

In there, one of the other community members mentioned using either cpio -p or rsync -a to copy and/or move data around.

The problem that I am facing with that proposed strategy/solution is that it is more difficult to do a multi-select like I can with a graphical file manager interface where using combinations of CTRL and SHIFT, I can quickly select multiple files and then either move and/or copy them to a destination.

I am surprised that TrueNAS 12 doesn't have this yet/still.

A cursory search for the same shows that there was a topic or feature request on this forum dating as far back as 2011.

Is there a reason why there isn't a host managed graphical file manager?

How do people migrate select data/files between TrueNAS servers then? (As I think that going from TrueNAS to TrueNAS using CTRL and/or SHIFT for multi-select would be faster than going from TrueNAS <-> some other management computer with a graphical file manager <-> other TrueNAS server over network.)

 

[winnielinnie]

There was a feature request which received enough votes and support, and supposedly it's on the table now as "Suggestion Accepted".

Not sure if it'll ever make it into CORE, or if it will only be available for SCALE.

FEATURE TICKET: Web GUI File Management as part of the new UI

STATUS TICKET: TrueNAS Local File Browser

 

[winnielinnie]

How do people migrate select data/files between TrueNAS servers then?

They use replications, ZFS to ZFS.

 

[alpha754293]

There was a feature request which received enough votes and support, and supposedly it's on the table now as "Suggestion Accepted".

Not sure if it'll ever make it into CORE, or if it will only be available for SCALE.

FEATURE TICKET: Web GUI File Management as part of the new UI

STATUS TICKET: TrueNAS Local File Browser

Yeah, I saw that.

I'm somewhat kind of suprised that there hasn't been like a community developed plugin for this already whereas QNAP et. al. all have a host-managed, web GUI based GUI file manager available.

They use replications, ZFS to ZFS.

My understanding is that you can't do/use unstructued multi-selects for this, correct?

Please educate me if what I am saying is inaccurate.

But my understanding is that if you want to do something like an unstructured multi-select, you'd pretty much have to do it in some other system that HAS a graphical file manager, and then find/figure out a way to write your selections into like a text file or something, and the pass that text file as arguments to the command line tools, no?

This also means that if you selected the wrong thing, and you wanted to cancel the move and/or copy task, make the changes you need to your unstructured multi-select, regenerate or update said text file, and then pass said text file back into the command line tool as input arguments to run the move/copy task/job again.

Would I be correct in regards to that?

(vs. if I do it wrong with a graphical file manager, I can cancel/abort the task, use a combination of CTRL and SHIFT to change my selections, and then at minimum "drag & drop" to rerun the either move and/or copy task/job.)

Like I can see how ZFS replication and/or cpio -p and/or rsync -a would be useful if the data on the system that's being moved/copied has a structured filename pattern to it so that it can be easily scripted.

But if the file names doesn't have said structured filename pattern to it, then it's a LOT of typing by using the command line tools, no?

 

[winnielinnie]

If you want to leverage ZFS's efficiency ("block-based", not "file-based") and "like for like" copy of a dataset/snapshot, then ZFS-to-ZFS is what to use.

In your case, you want to copy and move files around like a traditional file manager ("file-based"), so your options are to use the command-line, or your file browser, and move/copy files from one share to another. Akin to local file operations, but in your case these would be network folders, not local folders.

As for the built-in GUI file manager for TrueNAS, it's likely only going to be available for SCALE, and possibly only supports local file management (not server-to-server.) It appears to be backlogged, and not sure what iXsystems' priority is.

 

[alpha754293]

Akin to local file operations, but in your case these would be network folders, not local folders.

Right. But this is where and why I was referencing how, for example, QNAP has a web based, graphical file manager, where I can manage the files, on the host/server itself and also (and some/perhaps especially) between servers (because it's much faster to transfer files server <-> server vs server -> client -> server.

It takes the client out from the middle.

As for the built-in GUI file manager for TrueNAS, it's likely only going to be available for SCALE, and possibly only supports local file management (not server-to-server.) It appears to be backlogged, and not sure what iXsystems' priority is.

Yeah...that's also another difference when compared to QNAP's file browser because if you have a network mount (NFS/AFP/SMB/CIFS, etc.), you can manage the files on the local system and also server-to-server in the same graphical file manager that TrueNAS lacks.

I don't have any experience with FreeBSD and I'm relatively new to TrueNAS, so I am not sure if FreeBSD has ever had a graphical desktop environment that you could install as a part of the base OS installation process (e.g. I think that MOST Linux distro installations gives you at least the option to install a graphical desktop environment if you want one, when you are installing the operation system). I just don't have any experience with FreeBSD to be able to say whether that option is also, likewise available, for FreeBSD installs.

It certainly does not appear to be an option for TrueNAS installs.

Thanks.

 

[winnielinnie]

I don't have any experience with FreeBSD and I'm relatively new to TrueNAS, so I am not sure if FreeBSD has ever had a graphical desktop environment that you could install as a part of the base OS installation process

It's different for TrueNAS, as it would have to be web-based, python-based, and work seamlessly with the middleware, be respectful of ZFS's differences (vs other filesystems), and smoothly handle user accounts, permissions, and ownerships.

It's one of the most requested and popular feature requests, yet appears to be too massive of a project to incorporate any time soon. (It might also carry more risks and require much deeper QA testing.)

I'm only guessing here based from what I've gathered.

 

[Ericloewe]

In the meantime, there's always mc.

 

[danb35]

It certainly does not appear to be an option for TrueNAS installs.

It is not. Yes, FreeBSD has graphical desktop environments available (and it has had for some time), but TrueNAS (and FreeNAS before it) have never been designed for console usage; they've always intended to be managed over the network.

if you have a network mount (NFS/AFP/SMB/CIFS, etc.),

...which Free/TrueNAS doesn't. It's a file server, not a client.

as it would have to be web-based, python-based, and work seamlessly with the middleware, be respectful of ZFS's differences (vs other filesystems), and smoothly handle user accounts, permissions, and ownerships.

Other than being web-based (which would be the point of it--it's always been possible to manage files using a client via a file-sharing protocol), there's no inherent reason that any of these would be necessary. If you limit the file manager to only handling the data pools, nothing that you'd be doing there would really affect the middleware, and so long as it was using normal filesystem commands in the background, there's no reason the GUI system would need to be ZFS-aware. But security is really the issue--the file manager is almost certainly going to run as root, and that increases the risk significantly.

 

[winnielinnie]

there's no inherent reason that any of these would be necessary

For example, off the top of my head:

A large copy operation (or bulk click-and-drag of many files in one sweep), to run in the background with its progress displayed in the GUI's Task Manager (upper-right corner), which will not be interrupted if you close (intentionally or by accident) the web page or browser tab. (Seamlessly integrated with the GUI/middleware).

(Consider "Run Now" in the Replication Tasks to manually start a replication. You don't need to leave the web page open during its operation.)

I'd hope and imagine iXsystems wouldn't settle for a "good enough" rudimentary file manager that could risk data loss for move operations (that handle many files simultaneously) with an advertised user-friendly built-in GUI tool.

 

[danb35]

I'd hope and imagine iXsystems wouldn't settle for a "good enough" rudimentary file manager

They've done such things many times before (see, e.g., their short-lived ZeroTier "support", which consisted of "we bundled the zerotier binary, what more could you want?"). I don't anticipate that they're going to write their own web-based file manager from scratch; I rather expect that they're going to integrate--to some degree--an existing web-based file manager like filebrowser. And I'd frankly be very surprised if the "some degree" went any further than using the web GUI's authentication to authenticate to whatever project they went with--and I'd be kind of surprised if it went even that far. Integration of the sort you're describing sounds like it would be a massive project.

 

[Redcoat]

Right. But this is where and why I was referencing how, for example, QNAP has a web based, graphical file manager, where I can manage the files, on the host/server itself and also (and some/perhaps especially) between servers (because it's much faster to transfer files server <-> server vs server -> client -> server.

It takes the client out from the middle.

In the meantime, there's always mc.

@alpha754293 , at 122 years old according to your profile, you are certainly old enough to have been exposed to Norton Commander, which I used for years until enticed on to Windows. So, please heed @Ericloewe 's suggestion - try out mc, which'll look a lot like Norton Commander to you, for direct server-to-server file transfer and on-screen file selection, the benefits of which I also understand and support.

If you find can't hack that program's presentation. then I suggest that you use WinSCP and live with the "client-in-the-middle" network transfer until such time as there is a graphical file manager built for/in TrueNAS.

Based on your experience with those utilities, and any others that you come across, I'm sure you'll be able to offer valuable first-hand guidance to the iXsystems developers via comment to the Jira thread earlier referenced.

 

[anodos]

SMB and I believe NFS 4.2 (on SCALE) support server-side-copy, and so both of those are probably preferable to using SFTP. I haven't looked at kernel NFS server in FreeBSD 13 to see status of server-side-copy there. NFS of course is limited in this case to same dataset.

 

[winnielinnie]

SMB and I believe NFS 4.2 (on SCALE) support server-side-copy, and so both of those are probably preferable to using SFTP.

Server-side copy works across servers (bypassing the client)? I thought it was only applicable on the same server.

For example, say you have two network folders open on Windows (a SMB share from server1 and a SMB share from server2), if you click and drag to copy files from one folder to the other, it's not server-to-server connection, correct? It's still server1 -> client -> server2 from what I understand.

From the original post:

How do people migrate select data/files between TrueNAS servers then? (As I think that going from TrueNAS to TrueNAS using CTRL and/or SHIFT for multi-select would be faster than going from TrueNAS <-> some other management computer with a graphical file manager <-> other TrueNAS server over network.)

 

[anodos]

Server-side copy works across servers (bypassing the client)? I thought it was only applicable on the same server.

For example, say you have two network folders open on Windows (a SMB share from server1 and a SMB share from server2), if you click and drag to copy files from one folder to the other, it's not server-to-server connection, correct? It's still server1 -> client -> server2 from what I understand.

From the original post:

Ah yes, I glossed over that. Yes, you perform connections from server1 -> client -> server2. The best option in this case is ZFS replication of datasets. On a file level between shares, then this should be done through the relevant file sharing protocol since configuration options may impact how metadata is written. For example, if NAS1 is using POSIX1E ACLs and NAS2 is using NFSv4 ACLs, then simple rsync won't be able to preserve permissions, but going through SMB protocol from NAS1 to NAS2 will basically do the magic of converting relevant ACL info. Ditto for NFSv4.

It seems like a niche administrative action though (unless this is for doing a migration from server 1 to server 2).

 

[alpha754293]

So, please heed @Ericloewe 's suggestion - try out mc, which'll look a lot like Norton Commander to you, for direct server-to-server file transfer and on-screen file selection, the benefits of which I also understand and support.

Thank you.

I'll take a look at it to see if I can run it as a graphical file manager that runs directly on the TrueNAS server.

In the meantime, there's always mc.

Stupid question - does mc run in a GUI on the TrueNAS server?

but TrueNAS (and FreeNAS before it) have never been designed for console usage; they've always intended to be managed over the network.

I understand that.

But even if I log in to say a QNAP NAS, I AM managing it over the network, but operations can still be performed directly on/from/with the server itself., otherwise, if, for example, all of the data processing is handled by the client, then really; I wouldn't need anything more powerful (in terms of a CPU on the TrueNAS server side) than an Intel Atom or an ARM processor if that were to be the case (unless you need PCIe lanes for NVMe SSDs, then you might need Threadripper and/or Epyc).

Another example is that two of my QNAP NAS units have built-in 10 GbE connections. My desktop client does not.

And if I am managing the server from a slightly older Intel NUC (8th generation), you can't add 10 GbE SFP+ NIC to it to perform that kind of a management, whereas, again, in a comparative analysis between TrueNAS and QNAP (for example, as that's what I have and therefore; that's what I have experience with, but I'm sure other NAS vendors might have similar capabilities as well), I can completely bypass the client by executing inter-server data transfers vs. having to route through a slow client.

Other than being web-based (which would be the point of it--it's always been possible to manage files using a client via a file-sharing protocol), there's no inherent reason that any of these would be necessary.

My apologies, but I would have to disagree with that.

I'm about to decommission one of my QNAP NAS units because I want to move it onto TrueNAS actually.

But what this also means is that I am currently in the process of evacuating all of the data from said QNAP NAS unit onto my other QNAP NAS units and some of the files that's on the second, bigger QNAP NAS unit - I have to partially evacuate some of the data from that onto a temporary CentOS install that I've got on the same physical hardware that's intended to run TrueNAS.

From the older, smaller QNAP NAS unit that's about to be decommissioned, I'm moving something like 5.5 million files off of it, and if I were to set said data evacuation task as one giant task, it would take forever and then some.

Conversely, with the QNAP File Manager (that I access via the Web GUI), I am able to select, for example, 100 folders at a time, and move them in batches, therefore; leveraging parallelisation in the data transfer itself so that when you're moving so many smaller files, the transfers can be parallelised vs. running it as a single, monolithic task.

This is where having a graphical file manager can come in very handy.

Suppose you have folders numbered from 0-200, but you have random numbered folders that are missing in the middle of that range somewhere, and possibly, relatively about 10% of them are missing; if you script it using a text based/CLI shell command, the system is going to report back errors. If there were errors or problems during the transfer, now you have to sift through the log to figure out which errors were because the folders were missing vs. which errors were actual problems with the data evacuation task.

With the graphical file manager, I don't have to worry about that.

In fact, with QNAPs web based, graphical file manager, because I am moving 100 folders at a time, if there was a problem with one of those batches of 100 files (which, if there are missing folders in the middle of that range, because you're clicking on a "check all" box, it doesn't know and doesn't care that there are missing folders in that numbered sequence. It will just move what it sees.

So, there are a LOT of use case where something like this can be handy.

My desktop client only has a single GbE NIC in it.

My QNAP NAS units have both dual GbE AND dual 10 GbE NICs in them (4 NICs total for a total of 22 GbE worth of bandwidth available).

Think about how much faster it is to move about 5.5 TB of data over 22 GbE worth of bandwidth (another server) vs. 1 GbE worth of bandwidth (client).

But security is really the issue--the file manager is almost certainly going to run as root, and that increases the risk significantly.

Yeah....I'm not sure if this is necessarily a true statement either.

Again, drawing from my experiences with QNAP NAS units, as long as I have created the same user on all of the NAS units (because I am not using a centralised user/login authentication server/ADS), I can log in NOT as admin, but rather as myself on the individual NAS units and remotely mount each other using the same user-level credentials to pass data back-and-forth.

So, I'm not sure why there is a requirement for said web based graphical file manager to be run as root if you have your user accounts already set up across any and all TrueNAS servers.

I'd be interested in finding out what your rationale that's behind this statement.

For example, if NAS1 is using POSIX1E ACLs and NAS2 is using NFSv4 ACLs, then simple rsync won't be able to preserve permissions, but going through SMB protocol from NAS1 to NAS2 will basically do the magic of converting relevant ACL info. Ditto for NFSv4.

Per my original question though, it's TrueNAS to TrueNAS migration, but it's SELECT data migration rather than a pure ZFS replication/"migrate everything" mindset.

With that being said, given that it's a TrueNAS to TrueNAS migration, ACL and permissions methodology/protocol would be identical between the two unless iXSystems changes that.

Sidebar:
I LITERALLY installed CentOS 7.7.1908 on the same physical hardware as what I was going to use as a TrueNAS server BECAUSE I am able to install the GNOME graphical desktop environment on the system so that I can ensure that I can manage the data migration using a graphical file manager interface.

A part of this is also because with dual GbE NICs on the Supermicro X7DBE motherboard, I don't have those NIC bonded together (either in the OS nor at the L2 managed switch level) because in GNOME, it is LITERALLY way easier and faster for me to just mount the QNAP NAS that I am evacuating the data of off, directly via a SMB mount that's tied/directly connected to GbE NIC2.

That way, I can ensure that I can leverage the bandwidth that's available from BOTH GbE NICs rather than it being less-than-intelligent, and will try to route all of the traffic through the primary GbE NIC.

So on said CentOS system, I just installed tigervnc-server on it, and now I am managing the system (even the graphical desktop) remotely. On the console itself, it is running in init 3.

Sidebar #2 (as to why I want to do this and/or am looking for a graphical file manager that runs natively on the TrueNAS server):
If I am going from server1 -> client -> server2, I am putting that much traffic on the client's (in my case, single) GbE NIC to the extent that it literally eats up the full GbE NIC's worth of bandwidth, just shuffling data around.

The downside to this is that insodoing, it has also made it mostly unusable for anything else that I need or want to do that requires the network (browse the web, watch YouTube videos, etc.) because almost 100% of the available bandwidth is dedicated to the data evacuation and data migration.

If I take the client out from the middle of the equation, I've now have all four systems (three QNAP NAS units plus the server that I was going to put TrueNAS on again) all connected together with it's own switch, and so, it can shuffle data between each other and my client just only needs MAYBE 1 MB/s (10 Mbps) worth of bandwidth to manage the servers' server-to-server data transfers/data evacuation tasks.

That frees up the GbE NIC on my client for it to do other things whilst the servers are doing their things. It seems unintuitive to me why the solution that people appear to be recommending is to inject a client in between a server-to-server data transfer/selective migration, which means, really, that if you don't want your single GbE NIC to have all of its bandwidth sucked up for said server-to-server data transfer/selective migration, you're going to need to deploy at least a dual 10 GbE NIC plus a 10 GbE switch solely for a purpose like this.

This seems like it's such a massive oversight and as a result, such a bummer.

Solaris can create and manage ZFS pools and it has a graphical file manager, but Solaris has its own set of problems.

TrueNAS helps to manage the ZFS, but the absence of a graphical file manager, that just seems like it's a massive oversight to me.

CentOS has the graphical file manager, but trying to deploy ZFS on Linux is not trivial.

It's no wonder why QNAP charges almost $4000 for their 12-bay QuTS hero NAS (Source: https://qnapdirect.com/products/qna...diskless-rackmount-nas?variant=31989134098483) because it is literally the combination of everything that I used talked about.

Hmmm.....pity/shame.

Thank you, everybody, for your input.

 

[alpha754293]

at 122 years old according to your profile, you are certainly old enough to have been exposed to Norton Commander, which I used for years until enticed on to Windows

Yup. I actually do remember Norton Commander.

In fact, I remember Windows 1.0. :)

(functionally and largely/effectively "same-diff", right?) ;)

mc looks very much like that.

 

[danb35]

mc looks very much like that.

That isn't an accident.

Per my original question though, it's TrueNAS to TrueNAS migration, but it's SELECT data migration rather than a pure ZFS replication/"migrate everything" mindset.

Replication works on the dataset level, not on the pool level, so it could still easily be select data migration rather than "migrate everything"--as long as the "select" data were in different datasets from the "non-select" data.

So, I'm not sure why there is a requirement for said web based graphical file manager to be run as root if you have your user accounts already set up across any and all TrueNAS servers.

I guess it depends on what all you want it to do, but if you want it to be able to, say, create or destroy datasets, or really to do anything ZFS-aware, it's going to need to run as root, regardless of the user login that's used. And while you say you log into your QNAP as a non-root user, that doesn't mean that the file manager process(es) isn't/aren't running as root.

And again, perhaps it's just me being jaded, but iX seem to have a very low threshold of what they consider an included feature, at least if it's something they really aren't that interested in. Maybe the situation with ZeroTier was an anomaly, but it left me with no confidence that, assuming any sort of web-GUI file manager ever does get implemented, it will be integrated well at all.

 

[alpha754293]

as long as the "select" data were in different datasets from the "non-select" data.

But if the "select" data is in its own dataset, that still means that you are migrating everything from that said dataset, no?

In other words, you would have needed to configure the initial server when you were setting up the datasets for this, correct?

but if you want it to be able to, say, create or destroy datasets, or really to do anything ZFS-aware, it's going to need to run as root, regardless of the user login that's used.

My understanding is that the creation of the pools/datasets would happen BEFORE you attempt to manage the files that are on the pool/datasets, correct?

i.e. Using Windows Explorer as an analogy, you can't create "pools"/"datasets" (directly) from Windows Explorer so I am still not sure where the rationale for this statement/assumption is coming from and/or what's the background behind this statement.

(e.g. in Windows, you have the Disk Management to manage "pools" and if "formatting" the resulting software RAID array (the closest analogy that I can get to "pools" is also the closest analogy that I can get to creating "datasets"), neither of those operations are performed with Windows Explorer. That's done with Disk Management.)

In QNAP, there is the Storage & Snapshots app that manages the creation of the pools and formatting it for use.

I think that if you are using Linux software RAID, there's mdadm for that (along with mkfs for formatting the RAID array after said RAID array has been created) and that graphical file managers like Nautilus doesn't have anything to do with mdadm and/or mkfs.

And again, perhaps it's just me being jaded, but iX seem to have a very low threshold of what they consider an included feature, at least if it's something they really aren't that interested in. Maybe the situation with ZeroTier was an anomaly, but it left me with no confidence that, assuming any sort of web-GUI file manager ever does get implemented, it will be integrated well at all.

Yeah.

But this is why I was thinking that if FreeBSD already has a graphical file manager and that TrueNAS is treated as middleware that stacks on top of said FreeBSD, I don't really see why iXSystems can't deploy said graphical file manager from FreeBSD in at least a jail.

It would certainly facilitate an unstructured and selective data migration amongst other things.

I'm surprised that it doesn't/hasn't had one yet so far already and I'm also surprised that there hasn't been a community developed plugin/jail for the same either as well.

Thank you, everybody, for your input.

 

[Patrick M. Hausen]

But this is why I was thinking that if FreeBSD already has a graphical file manager and that TrueNAS is treated as middleware that stacks on top of said FreeBSD, I don't really see why iXSystems can't deploy said graphical file manager from FreeBSD in at least a jail.

Because FreeBSD does not contain a graphical environment. The common desktop environments are all third party products. And they don't make any sense on a server. None of my TrueNAS systems has a display connected, let alone a mouse.

How do you suggest I would access a graphical file manager in a jail?

As for a terminal based tiled interactive file manager people repeatedly hinted at mc, which is included. What's wrong with that one?

Kind regards,
Patrick