Graphical file manager application/plugin?

[alpha754293]

Because FreeBSD does not contain a graphical environment. The common desktop environments are all third party products.

I'm trying to think. Isn't this almost entirely true for all or nearly all Linux and Unix OSes?

CDE was (not sure about now, but I think it is still true today) released by The Open Group.

GNOME is 3rd party.

KDE is 3rd party.

JDS, which is based on GNOME, is therefore; 3rd party (despite being originally authored by Sun for Solaris).

Off the top of my head, I can't think of a single Unix or Linux distro that has their own desktop environment (vs. using someone else's).

And they don't make any sense on a server

HP-UX has CDE. Solaris/SunOS had CDE. AIX had CDE. Irix had CDE.

The data and the evidence would suggest otherwise. (Yes, you can run them WITHOUT the desktop environment on the operating systems that are used to (still) run mainframes, but you CAN also run the same OSes WITH a desktop environment as well, if you want and/or need to.)

Yes, FreeBSD has graphical desktop environments available (and it has had for some time)

Unless my understanding is incorrect, but according to this^, it should be entirely possible.

None of my TrueNAS systems has a display connected, let alone a mouse.

Not the point of the question though. This wasn't an inquiry into what you are doing personally and the reasons and rationale for doing things the way that you're doing them.

None of my QNAP NAS servers has a display nor mouse attached to them and yet I can STILL graphically manage files, on the server, remotely, via a web based GUI, which HAS a graphical file manager.

The absence of a display connected to it and/or an input device is wholly irrelevant here.

(And really, even for my CentOS system, I only really needed the monitor, mouse, and keyboard JUST for the install (because I'm not doing an unintended, scripted deployment. If I were doing an unintended, scripted deployment over PXE, I wouldn't need any of those things either). And yet, the moment that I install tigervnc-server and then set the vncpasswd, I can connect to the graphical desktop environment remotely in order to administer the system. And of course, I can do that without having a monitor, mouse, and keyboard connected to the system as well.

In fact, none of my CentOS cluster nodes, whether it is the headnode, or the compute nodes in my micro HPC cluster has a monitor, mouse, and keyboard, and yet, I can adminster ALL of them, graphically, remotely.

This statement has no bearing on the question.

How do you suggest I would access a graphical file manager in a jail?

In order to administer the TrueNAS system, does the TrueNAS host already tell you to go to the IP address of your server from a web browser from another system to do just about everything else, remotely, via the web based GUI?

Therefore; if you're already doing that, why CAN'T you access a graphical file manager in a jail?

Why CAN'T the jail, for example, install and set up a X server, and then you can run any desktop environment that runs on X to then run the graphical file manager?

That would be one idea and/or proposal.

Alternatively, I don't know much about the "guts" from the QTS OS that QNAP uses, but why can't you just do that?

What's wrong with that one?

It will depend on how well unstructured multi-selections work either with a combination of CTRL and SHIFT keys and also whether you can perform said unstructued multi-select with a mouse and/or keyboard.

If, by relative comparison, being able to perform an unstructured multi-select in mc works just as well as it does in Windows Explorer as it does in Nautilus, then there is no reason why I can't use it.

But, if it can't do that as well, then that would be a part of the answer to your question in regards to "what's wrong with that one?"

(Also depends on whether you'd be able to resize the columns as well so that, for example, you'd be able to see the full name of a file or if it is fixed width, which means all that you see is all that you are going to get.)

Because of this, I currently, no longer have a TrueNAS server running.

*edit*
So I just installed TrueNAS Core 12.0-U1.1 in a VM just to test out mc and to answer your question "what's wrong with it?", here are my findings below:

1) No mouse integration. Even when remotely administered. (Which is not that surprising given that it runs in terminal/console/is text-based.)

2) If you created 100 files with random file names, and you were to select the 5th, 6th, and then the 13th through the 17th file, etc., when you press "+" (which, by the way, the plus sign on the keypad does NOT work for this) (for a group select), you would have to type in the select pattern in order for it to be able to select the files.

In order words, I can't just be on the left panel, and keep hitting the down arrow and the spacebar for example, to be able to either select or tag the files that I want to move and/or copy. My understanding is that in order for the select to work, you would have to be able to give it enough information so that it would be able to properly select those files and if you can't even see the full name of the file, that can make that a whole lot more tricky/more difficult (because you can't just click on the column to resize it and then have a scroll bar on the bottom to scroll left/right due to the size of the column now).

In other words, I can use the down arrow to scroll down to the file and/or folder that I want to manipulate, but then to select it, I have to use a pattern to actually select it (again, I'm still reading tutorials online on how I can select files/folders with the mc interface). If you uncheck the "using shell patterns" checkbox, then "*" doesn't select all.

This means that if you have the aforementioned 100 randomly named file and you want to select random files out of that list of 100, you would have to type in the select shell pattern for mc to be able to select said files before you can manipulate it, which also means that you have to either know or you have to learn how to write shell patterns so that it would be able to select what you want it to select vs. just being able to click a checkbox (in a GUI) and/or using a combination of CTRL and/or SHIFT to perform the same kind of randomised multi-select.

So, what's wrong with it?

Try doing that when you are trying to move and/or manipulate 5.5 million files.

If there are errors with any of the move/copy operations that you have sent to the background, how will it tell you what kind of an error it experienced whilst trying to perform those file/folder manipulation operations?

Don't think about the problem as something that you can script in a structured manner.

Think about the problem statement as something that is incredibly difficult to script due to its highly unstructured manner.

(Imagine 5.5 million randomly named files and the number of characters in the file name ranges from 1 to 132 chars long, (I am trying to present the problem statement to you so that you can understand the highly UNstructured nature of it) which is what would make scripting shell patterns so darn difficult. Based on your response, I'm not sure that you quite fully grasp the nature of the problem statement that's been presented to you/that's behind this question/request for a graphical file manager.)

(And I would be willing to bet you that even if you knew shell expressions extremely well, I can probably out-click you with a graphical file manager than you would be able to enter the shell expressions for the multi-select and start moving/copying the files over before you would have finished typing in said shell expressions for said multi-select.)

Shell expressions for file selection is extremely user unfriendly.

If you are learning how to use it, you will end up selecting something that you DON'T want it to select, and in order to deselect it, because it's by shell expression, either you are going to deselect something that you didn't want to deselect, or that you would spend quite a bit of time, messing with it in order be able to craft the perfect shell expression for said selection vs. just clicking on the file(s) that you are wanting.

 

[Patrick M. Hausen]

In order to administer the TrueNAS system, does the TrueNAS host already tell you to go to the IP address of your server from a web browser from another system to do just about everything else, remotely, via the web based GUI?

Therefore; if you're already doing that, why CAN'T you access a graphical file manager in a jail?

For me "graphical" and "web based" are two entirely different things. Sorry, I misunderstood. Of course a web based file manager is a valid feature request. As for the other alternative - while technically possible I don't see many people use an X11 server on their Windows or Mac nowadays to run a "real" file manager like Nautilus via SSH and X11 forwarding.

 

[danb35]

if FreeBSD already has a graphical file manager

...it makes no difference at all, because you're not asking for a GUI file manager at the local console (which FreeBSD has, but Free/TrueNAS doesn't and almost certainly never will); you're asking for a web-based GUI file manager. Completely different animal.

It would certainly facilitate an unstructured and selective data migration

No, it really wouldn't, unless the capability to mount remote storage to the TrueNAS box is also added. That's a completely different ask.

I'm also surprised that there hasn't been a community developed plugin/jail for the same either as well.

Apparently there's a disconnect between the energy people are willing to put into asking for it, and the energy they're willing to put into making it happen. Strange how that happens. Edit: But really, this isn't hard. Create a jail, enter the jail, and run these commands:

Code:

pkg install curl bash nano
curl -fsSL https://raw.githubusercontent.com/filebrowser/get/master/get.sh | bash
mkdir -p /usr/local/etc/filebrowser/
filebrowser config init -d /usr/local/etc/filebrowser/filebrowser.db -a "0.0.0.0"
filebrowser users add admin admin -d /usr/local/etc/filebrowser/filebrowser.db
sysrc filebrowser_enable=YES
mkdir -p /var/run/filebrowser
mkdir -p /usr/local/etc/rc.d
touch /usr/local/etc/rc.d/filebrowser
chmod +x /usr/local/etc/rc.d/filebrowser
nano /usr/local/etc/rc.d/filebrowser

You're in a text editor; copy and paste these contents:

Code:

#!/bin/sh
#
# PROVIDE: filebrowser
# REQUIRE: networking
# KEYWORD:

. /etc/rc.subr

name="filebrowser"
rcvar="filebrowser_enable"
filebrowser_user="root"
filebrowser_command="/usr/local/bin/filebrowser -d /usr/local/etc/filebrowser/filebrowser.db"
pidfile="/var/run/filebrowser/${name}.pid"
command="/usr/sbin/daemon"
command_args="-P ${pidfile} -r -f ${filebrowser_command}"

load_rc_config $name
: ${filebrowser_enable:=no}

run_rc_command "$1"

Ctrl-X to exit, yes to save. Then start it with service filebrowser start. It will automatically start whenever the jail starts.

You've just set up a GUI file manager for the contents of that jail. Adding storage mountpoints to the jail and securing with SSL are left as an exercise for the reader.

How do you suggest I would access a graphical file manager in a jail?

Well, I guess you could install X, a regular desktop environment, and something like noVNC in the jail. But why?

 

[Ericloewe]

1) No mouse integration. Even when remotely administered. (Which is not that surprising given that it runs in terminal/console/is text-based.)

It does work with a mouse. Try mc -x. If that doesn't work, definitely read up on the subject because it works just fine with a mouse, even inside a tmux session over SSH. It could also be a problem with your shell on the client...

2) If you created 100 files with random file names, and you were to select the 5th, 6th, and then the 13th through the 17th file, etc., when you press "+" (which, by the way, the plus sign on the keypad does NOT work for this) (for a group select), you would have to type in the select pattern in order for it to be able to select the files.

Right click would select, once you have the mouse working.

 

[alpha754293]

As for the other alternative - while technically possible I don't see many people use an X11 server on their Windows or Mac nowadays to run a "real" file manager like Nautilus via SSH and X11 forwarding.

No, but as I mentioned, I use tigervnc (or really, you can use just about ANY other VNC server, on the server side) and then you just need a VNC viewer like "VNC Viewer" on the Windows client to then remotely log in to either a graphical desktop environment (if you want to use one), or if you aren't using a graphical DE, you can use a web based graphical file manager (and that won't require a VNC server).

I mention this because, as I mentioned, the hardware that I was to run TrueNAS is now running CentOS instead, and in order to get to/get at a graphical file manager (Nautilus on GNOME), I just install tigervnc-server on it and connect to it remotely on my Windows client using "VNC Viewer" (that's literally the name of the app that I use) and that gives me the full, graphical DE view without the server itself needing to run in init 5. (I mean, it can, but it doesn't have to because the session that would be on console is separate from the remote VNC session; i.e. NOT like Teamviewer where it kinda just "mirrors" what you see on console, remotely.)

I've tried X11 forwarding via SSH via xterm via Cygwin. It sorta works, but it isn't really all that stable nor is it really all that great. It's a "yeah, you CAN do that in theory", but in my attempts of trying to use it, it really didn't work. Tigervnc-server + VNC Viewer worked better and even then, that sometimes, still can have issues as well, but I digress.

...it makes no difference at all, because you're not asking for a GUI file manager at the local console (which FreeBSD has, but Free/TrueNAS doesn't and almost certainly never will); you're asking for a web-based GUI file manager. Completely different animal.

I'm looking for either and/or both options.

If FreeBSD has a graphical DE and I can setup a VNC server where then I can connect to it remotely to get access to said graphical file manager that runs on said graphical DE - that's one option/solution.

But if FreeBSD/TrueNAS does NOT have a graphical DE and is unlikely ever going to get one, then a web based graphical file manager will also work for me as well.

The availability or lack of availability of one option does not automatically preclude nor exclude the availability or lack of availability of the other option.

In other words, I'm agnostic to its implementation. If someone tells me that TrueNAS is going to adopt the FreeBSD graphical DE, and I can google how to install a VNC server on TrueNAS, then it meets the requirement of having a graphical file manager.

If, on the other hand, a graphical DE is an automatically terminal point of discussion, then make the graphical file manager web based.

I'm solution agnostic. All I care about is getting/having one. I don't care about what form/flavour it comes in.

No, it really wouldn't, unless the capability to mount remote storage to the TrueNAS box is also added. That's a completely different ask.

I would suggest to re-read the use case and the rationale behind asking for said graphical file manager in the first place (which is to facilitate an unstructured multi-select manipulation of data/files/folders).

Apparently there's a disconnect between the energy people are willing to put into asking for it, and the energy they're willing to put into making it happen. Strange how that happens. Edit: But really, this isn't hard. Create a jail, enter the jail, and run these commands:

This is true if you already HAVE the know-how.

If you don't, then it falls under the bin of "you don't know what you don't know".

(Sidebar: I'm a mechanical engineer, not a sysadmin. I wouldn't expect a sysadmin to be able to do a mechanical engineering job anymore than I would otherwise expect a mechanical engineer to do the job of a sysadmin.)

You've just set up a GUI file manager for the contents of that jail. Adding storage mountpoints to the jail and securing with SSL are left as an exercise for the reader.

If this is the case then, doesn't this mean that you can package this up as a community developed plugin/jail that people can then download, use, and install for anybody and everybody who's been asking for one if this has been one of the most requested features?

To your point above re: "Apparently there's a disconnect between the energy people are willing to put into asking for it, and the energy they're willing to put into making it happen. Strange how that happens. Edit: But really, this isn't hard. Create a jail, enter the jail, and run these commands:"

Your solution code then begs the question - if it wasn't that hard to do, then why it is, that when I try to search the forum for one of the lines from your solution code, this is the only thread where it shows up despite the fact that as stated, it has been one of the most requested features to be added to TrueNAS.

If I understand this correctly, this would be the answer that @anodos can roll into the main TrueNAS code, and closes out these tickets:
FEATURE TICKET: Web GUI File Management as part of the new UI

STATUS TICKET: TrueNAS Local File Browser

with the solution code that you had just written/posted.

(Or at least it would be A form of a graphical file manager, if it might not be web based.)

(More stupid question from my end. I did look at it and their installation instructions page basically centers around Docker and doesn't say anything about how to pull it down from github (which I am assuming that they are assuming that if you are wanting to do that, that you already know what you're doing, which I most certainly do not). Also, their documentation also doesn't show how do you connect to it after it has been installed as well because the README.md file on Github shows "filebrowser.web" as the address and I don't know if that's the example that they've used or if that's the read address that you need to go to to be able to use it. Their github repo doesn't even have a "docs" folder so for someone who doesn't know what they're doing, it's a "great. I can install it. I have no idea how to connect to it because there is no obvious documentation for that.")

(Thank you for the code, by the way. That is greatly appreciated. Now I'll have to google how to mount the local ZFS pools/datasets/shares so that it doesn't have to go through the loopback device.)

It could also be a problem with your shell on the client...

I wasn't using a shell on a client.

Being that it was in a VM, I was manipulating it directly on the console of said VM.

The lack of mouse integration might have something to do with it running in a VM (VirtualBox to be more specific).

Right click would select, once you have the mouse working.

That would be AWESOME!!!

*edit*
I did try to connect to it remotely using Cygwin. If you are not using the Cygwin Terminal, then no, mouse integration does not work.

(And if you have also enabled right-click = paste in Cygwin Terminal, then trying to right-click to select in mc -x would just paste whatever text you had copied last, into the single line that's shown at the bottom for commands.)

But if you disable that, then it works like a charm!

(The other problems with the size of the columns, presumably a lack of a horizontal scroll bar, and not being able to see the full names of files might still be issues, but at least the unstructured multi-select is working now.)

Very cool!

Thank you.

 

[Patrick M. Hausen]

You can install a VNC server and a graphical environment of your choice inside a jail.

To get at all your share datasets from within a jail - if it is only a small number - you can mount them inside the jail via the "mountpoints" feature.

That leaves the issue of migrating data from one system to a second one, because there is no remote filesystem mount capability in TrueNAS.

 

[danb35]

If FreeBSD has a graphical DE and I can setup a VNC server where then I can connect to it remotely to get access to said graphical file manager that runs on said graphical DE - that's one option/solution.

GUIDE: Setup secure GUI control on a jail using either VNC or X11

It is not easy to find guides for FreeBSD. In this guide, I explain with a near step by step how to: - install a KDE5 or GENOME 3 GUI in a jail - remote control the jail using either VNC or the X11 forwarding - install VNC as a service started on jail boot - properly setup a secure SSH...

www.truenas.com

I would suggest to re-read the use case and the rationale behind asking for said graphical file manager in the first place (which is to facilitate an unstructured multi-select manipulation of data/files/folders).

Yes, I'm familiar with your idiosyncratic use case. I don't see how you think a graphical file manager is going to help you with it. TrueNAS does not support mounting remote filesystems, and any such requested capability has nothing to do with how you titled your thread, nor with the tickets you reference, nor with any of the other requests for a GUI file manager.

I did look at it and their installation instructions page basically centers around Docker and doesn't say anything about how to pull it down from github

Really? You mean this page?

Your solution code then begs the question - if it wasn't that hard to do, then why it is, that when I try to search the forum for one of the lines from your solution code, this is the only thread where it shows up despite the fact that as stated, it has been one of the most requested features to be added to TrueNAS.

Because none of the whiners who have been asking for it have bothered to put in the least effort to figure out how to do it, apparently, preferring instead to complain about why nobody else has done the work for them.

 

[anodos]

To clarify, my primary areas of responsibility are file sharing protocol (AFP, NFS, SMB). I won't comment much on the needs / usage of a graphical file manager.

My gut feeling is that for a local system the particular sharing protocols with a modern client do sufficient server-side offloading that there's not a lot to be gained by doing things locally vs through a client for simple tasks like "move this directory to this other place on the same server" or "make a copy of this thing in this other share".

For the more specific task of "I bought this new server and want to move my SMB share contents to it", IMHO the best recipe for success in $place_of_work is to:
1) create a replication task to the new server
2) plan a maintenance window and notify all stakeholders of upcoming change
3) during maintenance window:
3a) make SMB share dataset RO on old server
3b) replicate any remaining changes,
3c) create new share on new server (with same share name as old one)
3d) (maybe change path of share on old server to a directory containing instructions on how to access new share on new server)
3e) push automated changes to all relevant SMB clients in environment to mount the new path and unmount old one (can be via GPO in Windows / AD environments)

This is a very rough outline, but gives an idea of how to do the migration. It's important to plan out and test for yourself each step along the way and write up a "forwards" and "backwards" plan in case something happens and you have to revert the cut-over.

My experience is that manually moving things around piecemeal between servers isn't a recipe for success. You inevitably end up with two copies of things and users writing to the wrong copy.

 

[winnielinnie]

It does work with a mouse.

Right click would select, once you have the mouse working.

Wicked cool. However, it doesn't cover the scope of what the OP is trying to do, which is direct TrueNAS server to TrueNAS server.

Apparently, QNAP NAS has a web-based file browser that also supports direct QNAP server to QNAP server, bypassing the client. (I don't know if that's accurate, but just assuming it based off of the OP's descriptions.)

 

[Ericloewe]

However, it doesn't cover the scope of what the OP is trying to do, which is direct TrueNAS server to TrueNAS server.

Sure it does, it natively supports SFTP and a bunch of other things.

 

[alpha754293]

You can install a VNC server and a graphical environment of your choice inside a jail.

To get at all your share datasets from within a jail - if it is only a small number - you can mount them inside the jail via the "mountpoints" feature.

That should work then.

That leaves the issue of migrating data from one system to a second one, because there is no remote filesystem mount capability in TrueNAS.

TrueNAS does not support mounting remote filesystems

That seems perhaps a tad bit surprising to me.

I just logged into the TrueNAS VM that I am using to test and I was able to go to the web management GUI, log in, go to shell, and then create a mountpoint directory for the remote mount
# mkdir -p /mnt/qnap1/share

and then mount the NFS filesystem from my QNAP NAS:
# mount -t nfs qnap1:/share /mnt/qnap1/share

And that appear to have worked correctly (i.e. no warnings, no error messages).

So, I'm not sure that I follow what you guys mean when you say that "there is no remote filesystem mount capability in TrueNAS".

My thought process is this (and please educate me if my way of thinking is in any way inaccurate):

If I can create the jail which gives me the graphical file manager, and if I am able to mount the NFS remote mount (presumably that's better/easier in TrueNAS to use NFS than to use SMB), and I set those as mountpoints for said jail, doesn't that mean that I am now able to move the files between the QNAP NAS and my TrueNAS server using said graphical file manager that runs inside said jail?

Either this works the way that I am thinking that this should work or that it won't work because there is some other piece of information that I am critically missing and/or not understanding how it works properly.

Really? You mean this page?

Sorry. My bad. The default for that went to "Brew". It didn't even dawn on me that there was a Unix tab and a Windows tab. I thought that it was a static image.

Because none of the whiners who have been asking for it have bothered to put in the least effort to figure out how to do it, apparently, preferring instead to complain about why nobody else has done the work for them.

Interesting.

1) People asking for feature requests = whiners.

2) I'm not sure why you would assume that the people who are asking for said feature would have to skills, knowledge, and expertise to be able to do this.

I mean, before I wrote my thread/topic on here, I googled it.

And if you try to google "truenas graphical file manager", your proposed solution doesn't even come up in the search results.

Therefore; I am not sure why you would necessarily think or assume that people know what you know (not only in terms of the solution path, but also the path to implementation)?

I am clearly dumber than you are (because I looked at your code and I would have NEVER, not in a million years, think of that solution path).

But by that statement, it also means that there are CLEARLY other people who are smarter than I am and therefore; when you go to google it, and this solution doesn't pop up in the search results, one of the possible, logical, obvious conclusions is that "even the people who are smarter than I am in regards to this, hasn't come up with a solution yet".

So....why would you task an idiot (myself) to do something smart like that on the assumption that I am both an idiot and yet, somehow also expected to be able to do something smart?

I don't quite understand that.

You assume that we are lazy a la "preferring instead to complain about why nobody else has done the work for them", but you left out the entire realm of possibility that maybe, I'm just too dumb to even come up with the solution, let alone an implementation path.

Think about it.

The instructions on the Unix tab for installing it only gave what it gave.

But you compare and contrast those instructions that filebrowser.org provided with the rest of your code - those two lines is just two lines out of a total of 10 lines, plus the lines that are in the text file itself.

I'm confused. Either you think that we are smarter than we actually are, or that you think we're just lazy.

It wouldn't have even DAWNED on me to create a jail for this and let alone create the text file that you put up.

I can assure you that you overestimate my abilities by a LONGGGG shot.

My gut feeling is that for a local system the particular sharing protocols with a modern client do sufficient server-side offloading that there's not a lot to be gained by doing things locally vs through a client for simple tasks like "move this directory to this other place on the same server" or "make a copy of this thing in this other share".

As I have noted, where it makes a difference is "Is the client managing the server, and the servers are excuting the tasks directly on the server itself?" or "Is the client somehow facilitating the transfer?"

In the latter model:
The data going from server1 -> client -> server2 actually passes through the client because the client is the one that's issuing the move/copy commands/requests.

The example of this would be say you have two TrueNAS servers and both have been configured with SMB.

Then on a Windows client, you mount the SMB shares as a different drive and assign it different drive letters.

Now you start moving/copying files from one system to other.

In this scenario, the Windows client, through Windows File Explorer, is actually involved in the data transfer in that it has to pass through the client from server1, in order for it to land on server2.

Conversely, in the former scenario/model (which is what I was trying to do), the idea is to take the actual transfers of data out of the equation from the client's perspective, where the client only tells server1 to send the data to server2 and the client sees NONE of that data traffic.

This is where the two models (can) make a HUGE difference, because as I mentioned, my Windows clients only has a single GbE NIC on them whereas my servers can have 10 GbE NICs on them (in addition to dual GbE NICs).

"I bought this new server and want to move my SMB share contents to it"

I don't disagree with you if you are doing a wholesale migration.

But that isn't the case here.

Therefore; given that it ISN'T a wholesale migration, how would you change or modify your plan of action/attack/migration to account for the fact that this isn't a wholesale migration?

(IF it was wholesale migration, I understand and agree that you can simply the entire process by just using ZFS replication. But this isn't the case here and therefore; as such, ZFS replication is not applicable/is a no-go here.)

Apparently, QNAP NAS has a web-based file browser that also supports direct QNAP server to QNAP server, bypassing the client. (I don't know if that's accurate, but just assuming it based off of the OP's descriptions.)

I can confirm that this is a true statement because I just mount the QNAP server #2 from QNAP server #1 as either a NFS mount and/or as a SMB mount. (i.e. if you want to mount it as both a NFS and a SMB mount, then you just create two mounts using their Hybrid Mount tool/app)

I don't have a 10 GbE switch, but two of my QNAP servers have dual 10 GbE ports, so I just have a short maybe like 6" cable that links the two systems together directly. Set the IP address up for each of the interfaces and then it can see each other and then I can send data over each other over 10 GbE without the need for a 10 GbE switch. I do this quite often actually.

So when I am logged into their web based management screen (QTS), I can click on File Station and move the data back and forth, using said graphical file manager and all my Windows client is doing is issuing the commands and telling the servers what to do. My Windows client does not participate in the actual acts of transferring the data. (It can't because my Windows client doesn't have a 10 GbE NIC in it.)

Sure it does, it natively supports SFTP and a bunch of other things.

Once again, I think that you might be missing important/critical pieces of the discussion, i.e. being able to perform an unstructured multi-select for the operation.

mc -x running via Cygwin Terminal is a potentially viable solution.

The other is filebrowser, as @danb35 mentioned, running from inside a jail which will provide said graphical file manager that you can run via a web browser, which also meets the statement of requirements.

SFTP works great if you don't have the unstructured multi-select. (i.e. if you have a structured multi-select, then you can use shell expressions for that and that'll work.) (mc without the '-x' works for the structured multi-select as well.)

And ZFS replication works if you just want to clone/migrate the whole thing.

Thank you all for your input and education. It is greatly appreciated.

 

[Patrick M. Hausen]

You can of course try NFS. But if you created the files on your QNAP via SMB, I wish you much fun with the filenames. NFS doesn't have a clue about character sets, while SMB does. The names are going to get messed up unless you use only US ASCII in file names.

That's why it was repeatedly recommended to go through a client that connects to both SMB servers.

Similarly for ACLs, btw. ...

 

[danb35]

People asking for feature requests = whiners.

No, but this particular request seems to attract them. Such requests are full of emotive language about how surprised or disappointed or (whatever) the requester is that the feature isn't there, how it places Free/TrueNAS at such a disadvantage compared to QNAP/Synology, why why why, rejection of any suggested alternate solutions, and to cap it all off, demanding of someone who explains exactly how to do it why he didn't do it sooner. Put simply, the "requests" tend to take the form of "do it for me" rather than "help me do it."

I'm not sure why you would assume that the people who are asking for said feature would have to skills, knowledge, and expertise to be able to do this.

On the contrary, I assume the opposite. I assume that people who are asking questions have some degree of technical skill. I assume that people who are administering file servers know a thing or two about how to do that. And while TrueNAS insulates you from the FreeBSD command line in most cases, I do tend to assume some working familiarity with that as well.

It wouldn't have even DAWNED on me to create a jail for this

Really? Because you're the first one to mention a jail in this thread. And a jail is really far from a perfect solution, because it can only operate on the data that you give it access to; they aren't ordinarily designed to have full access to your pool. But in case it's of any interest, here's more or less the thought process that got me to the instructions in my earlier post:

• Adding extra software to TrueNAS should be done in a jail, as a general principle. There are some exceptions (and filebrowser, as a Go binary, may be one of them), but a jail is the general rule.
• Google for something like "web-based file manager", and see that filebrowser was one of the first few hits. Check its GitHub repo, and see that it has a FreeBSD binary available--that's a good sign.
• Follow its instructions to install it. I know that a jail doesn't come with bash by default, and forgot (but was reminded) that it doesn't come with curl or nano either, so manually installed those.
  • Note that it's listening on 127.0.0.1, and get "connection refused" when trying to reach it from my desktop machine, which tells me I need to change the listening address--check the filebrowser docs on how to do that. Once done, note that it works.
• I want it to start when the jail starts, which means it needs an init script. I'm an idiot with those, so google for "filebrowser freebsd init script". One of the first page of hits is Supervised FreeBSD rc.d script for a Go daemon | redByte blog, which gives me the template for the script I used.
• From there, it was trial and error. The init script gave an error about the /var/run/filebrowser/ directory not existing, so I created it. /usr/local/etc/rc.d/ doesn't exist by default, so I created that. Ditto /usr/local/etc/filebrowser/. I wasn't able to log in, and I still don't know why the default admin user wasn't there, but it wasn't, so I again consulted the filebrowser docs for how to add a user.

So not much "special" or FreeNAS/TrueNAS/FreeBSD-specific knowledge here, other than knowing to use a jail, knowing I need an init script, and knowing where they go. The rest of it was following docs and tracking down errors.

So that gives you a web-GUI file browser/manager. And as a Go binary, it's a single, standalone binary, with no dependencies on anything else, so it could even be run on the base TrueNAS system--though I don't think I'd want to try to deal with an init script for non-default software there.

 

[Ericloewe]

Once again, I think that you might be missing important/critical pieces of the discussion, i.e. being able to perform an unstructured multi-select for the operation.

Didn't you just say that mc works fine for your selection needs (even if a bit on the clunky side)? So just have mc connect to another machine via SFTP or whatever you feel like using:

That option will set up an SFTP connection in the right panel. Needless to say, the left panel can do the same thing if you'd rather do things that way.

Unless I'm misunderstanding something, this would solve your immediate issues at the cost of being clunky but with the bonus of requiring zero setup or close to it.

 

[alpha754293]

NFS doesn't have a clue about character sets, while SMB does

So far, *knock on wood*, I haven't encountered any issues, even with non-US ACSII characters.

That's why it was repeatedly recommended to go through a client that connects to both SMB servers.

I've been told/read that NFS *may* have performance benefits that SMB doesn't (cf. https://ferhatakgun.com/network-share-performance-differences-between-nfs-smb/) but I have never systematically nor extensively tested it.

Most of the time, as long as the data gets to its destination in a relatively reasonable amount of time, I'm somewhat ambivalent in regards to the performance of it.

The only time that I would really pay a lot more attention to the performance is if I'm moving around a LOT of really tiny files, but that also typically involves my Windows client in the middle of it, which, I have also found that if I take said Windows client out of the middle of it, server-to-server transfers - it's generally not much of an issue.

Put simply, the "requests" tend to take the form of "do it for me" rather than "help me do it."

Playing devil's advocate, let's look at the question from the point-of-view of the requester.

There are quite a number of people that rave about TrueNAS (and it can be argued that there might be about as many people raving about Unraid as well) and how it is an alternative solution to more "paid" options such as QNAP/Synology, etc.

Whether this is the fault of the "market" (i.e. people buy "prebuilt" NAS boxes) and it works however it works (e.g. QNAP) and then they have a hardware failure or watch YouTube videos of other people talking about how said pre-built boxes are crap because finding replacement parts for if and/or when parts fail makes said pre-built boxes crappy, and then they're introduced to this whole idea of "hey! You can build your own NAS server with this thing called FreeNAS/TrueNAS!"

And of course, as a n00b, you wouldn't know any better nor any different either.

(I am, of course, speaking from my own personal experience of how I ended up with TrueNAS, but I didn't have the hardware failure (yet) on my QNAP units. I actually started using TrueNAS because my old Buffalo Linkstation 441D was crap (couldn't get more than 20 MB/s speeds with four 3 TB drives in RAID5). So I ditched that and that was the first time that I deployed TrueNAS thinking that my old dual Xeon server has got to be able to do better than said crappy Buffalo Linkstation. And said Buffalo Linkstation did NOT have a graphical file manager, but it was so slow anyways that even if it had one, I would've ditched it regardless.)

So, you step into the TrueNAS ecosystem after reading/watching people talk about it (and/or rave about it) and so you go "okay, I'm going to try it."

But as you get more and more into it, this is where and when you start to find out/realise the pros and cons of each solution, and this type of a "con" (i.e. if you were line up the two options, say QNAP vs. TrueNAS and you compare it feature by feature, that's really when you will find out that there are things that TrueNAS can do that QNAP can't, and there are things that QNAP can do that TrueNAS can't.)

Getting back to your point though, re: "do it for me" vs. "help me do it" - again, playing devil's advocate, isn't this why people use other people's stuff pretty much ALL. THE. TIME.?

I mean, this is why you have people who are experts in their respective fields because there are other non-experts.

My friend and I have this "inside joke" between us where he says that I am a grossly unqualified sysadmin and I ALWAYS fully agree with him that I AM a grossly unqualified sysadmin.

People have marvelled at what I have and have been able to do. Some think or assume that I work in IT, professionally. But I don't. I'm a mechanical engineer by trade and education. I only do this IT stuff because I HAVE to in order to facilitate the engineering side of things.

Thus, as a non-IT professional, my goal is to get the system up and running as quickly as I can so that I can spend more time doing engineering work vs. spending my time doing IT work. (If I wanted to do IT work, I would have elected to go into IT professionally, which is what my dad wanted, but I literally walked away from that. And according to my sister, she said that it took my dad 6 months to get over the fact that his son wasn't going to be following in his footsteps. My dad used to be a computer operations associate for a bank, and I think that my first visit to a "real" server room was probably like when I was 9.)

There's also the adage of "why reinvent the wheel if someone else has already invented it?"

(I have a OneNote file that I keep with CentOS deployment notes for my micro HPC cluster so that if I had to redeploy a node (something goes wrong, and I have to delete everything and redeploy the node), I can usually do that now in about 2 hours after LOTS of googling. To this day, there is still a HPC application that from my initial testing, I have NOT been able to deploy in CentOS. A la "help me figure it out", yes, I could've done that. OR what I ended up doing is popping in another boot/OS drive instead, installed Ubuntu on it, and then installed that other HPC app instead because it works on Ubuntu rather than in CentOS. I can understand that different personalities and different people MIGHT be really interested in getting something to work on something and that's would be what they find interesting/fun/challenging.

But if you're working on production, and each hour of downtime that the system isn't crunching numbers is costing you money, and deploying Ubuntu is a faster solution so that you can get the system back up and running quickly, yes, you don't learn how to do it in CentOS, but that isn't necessarily the focus either.

As I mentioned, to me, the IT stuff is just a tool/means to the engineering end. And as such, you pick the best tool for the job.)

This is a really big reason as to why I deploy "canned" solutions vs. compiling everything that I need from scratch. (I don't program/code myself either, so if something goes wrong, the debug/error messages often means little to nothing to me, and if googling it doesn't yield a solution that someone else has already written about, I don't dwell on it. I move on.

So what if I can't run OpenFOAM on CentOS? I can run OpenFOAM in Ubuntu. Good enough. Pack it, slap a label on it, and ship it and move on.)

(More to this point, people who are NOT experts in any given area are ALWAYS at the mercy of experts in said area.)

Like you had mentioned filebrowser initially. But it isn't until a number of replies/comments further where we talked about deploying it in a jail and that's when the solution struck in regards to this.

As I mentioned, I tried googling for a solution first and I also tried searching for "file manager" on this forum as well, with some posts dating back I think to 2011. (cf. https://www.truenas.com/community/threads/file-manager-in-the-gui.65/)

I also literally just ran a search on here for "filebrowser" and if you cross reference that with the search for "file manager", there's literally 7 pages of results for "file manager", but only about maybe a page of results for "filebrowser". And if you run a search on this forum for "filebrowser "file manager"", this is the only thread that pops up.

So...

a) You don't know what you don't know.

b) Even when you try to search for a solution, prior to this thread, if you were to search for "filebrowser "file manager"" would have not yielded any results on this forum. (i.e. this is one of the threads that according to the search engine, links the two together) (i.e. before this thread, I wouldn't have even known to look at filebrowser for/as a graphical file manager).

c) Even after you had mentioned it, it isn't post 19 where the first instance of the word "jail" appears.

d) Based on your jailed solution in post 23, if this is one of the answers to this "commonly requested feature", then a part of me thinks that it still took 23 posts to get before your expert knowledge is passed down to someone else who is not an expert (me) in terms of how to deploy it.

Like I literally just googled one of the terms that you used in your rc.d script, "filebrowser_command", and this thread is the only result that shows up.

The point that I am trying to make is that this is expert knowledge that you have and are sharing.

Again, I can't ask the "help me figure it out" question if I don't even know the question that I should be asking in trying to figure out how to deploy it.

This gets back to (a) - you don't know what you don't know.

And as I also mentioned, I read through (albeit, perhaps poorly) the installation instructions from filebrowser.org and again, you needed your specific expert knowledge to be able to tie what filebrowser.org tells you and what you know in order to produce the code that you've provided in regards to how to deploy filebrowser in a TrueNAS jail.

I assume that people who are asking questions have some degree of technical skill. I assume that people who are administering file servers know a thing or two about how to do that. And while TrueNAS insulates you from the FreeBSD command line in most cases, I do tend to assume some working familiarity with that as well.

But it depends on what specific technical skill you are assuming they are in possession of.

(i.e. coding? shell scripting?)

Cuz I can guarantee you that I possess neither of those skills if you are assuming that those who are administering a server will be in possession of those said such technical skills.

Again, there can be quite a number of people who administer servers because they HAVE to, and not necessarily because they want to. This is certainly true in my case.

I know what people tell me/what I read and/or what people point me to.

And it has been my experience that documentation is often times written from the perspective of people who are in possession of some said knowledge, rather than it being written with the assumption that it's your first day of university where you DON'T know what you know now, but you're still required to quote "get the job done".

re: FreeBSD
I hate to say this, but nope.

I have no clue how FreeBSD works.

(My background is in Solaris before Linux. And for Linux, it's mostly CentOS with a TINY sprinkle of Ubuntu thrown in there for good measure. That's it. I've never seriously dabbled with FreeBSD (as a standalone OS nor as the basis for something else (e.g. TrueNAS). It's been QUITE the journey. FreeBSD is quite a lot different when compared to Solaris and CentOS. Like idiot me STILL doesn't how to create a new user using the web management GUI and also create said user's home directory. (It says that it needs to be in the /mnt directory, but it doesn't give you an option there to create a new folder from said /mnt directory for the home directory of the new user. So, for me, my dumbass way of doing that is to do it via command line as root. And then chown it after the user's created. Never figured that out.))

So no, I have no idea how FreeBSD works.

Really? Because you're the first one to mention a jail in this thread.

On-the-fly answer.

From reading the responses, it doesn't sound like that iXSystems was going to change the main web management GUI/interface, so if I was hoping for something like the QNAP QTS where they have a File Station right then and there, it quickly became relatively obvious that that wasn't going to happen.

So, what's the next best thing? Sounded to me like pretty much everything else that you want or need it to run, all runs inside jails.

So, smash the two ideas together and that's how I came up with the idea of "can't they (iXSystems) run it in a jail?"

It still took 19 posts for me to get there.

Google for something like "web-based file manager"

Yeah....the search term that I used was "truenas file manager" and that didn't get me where it needed, hence this thread.

It wasn't for a lack of trying though.

This is also the first time that the entire concept of an entirely web based file manager came up because in QNAP, it's a PART of their QTS operating system/web management interface, so I didn't really see nor think of it as being something that can be separated out from the OS.

I know that a jail doesn't come with bash by default

I didn't even know that.

Never used TrueNAS/FreeBSD jails before.

And a jail is really far from a perfect solution, because it can only operate on the data that you give it access to; they aren't ordinarily designed to have full access to your pool.

Yeah. But this goes back to my original question though, whereby imagine if you had a solution like filebrowser, but because of the way it is implemented, it CAN have full access to the pool and whether you can move only user files and/or root files, would depend on your log in to the TrueNAS web management GUI.

This idea/MO is entirely based on my experience with QNAP's QTS software. Not saying that it HAS to be done this way, but this would be one approach if you don't want to give it root acess to everything, all the time.

So not much "special" or FreeNAS/TrueNAS/FreeBSD-specific knowledge here, other than knowing to use a jail, knowing I need an init script, and knowing where they go. The rest of it was following docs and tracking down errors.

Believe it or not, again, based on my having tried to google for a solution before coming here to ask my dumb question -- you might not think that it is all that "special" but the fact that there hasn't been or there isn't a google search result that provides this answer/solution probably IS your "special sauce" that you just created.

Even on here, one of the oldest threads for a file manager dates back to 2011.

And as others have mentioned, it is also one of the most commonly requested features.

And yet, 23 posts into this and you literally have a working answer/solution to this question that people has been asking for, for quite literally, the past decade.

It might not seem "special" to you, but the lack of a solution to this prior to this thread, suggests that it actually IS something "special" because no one else has written (about it) before like you have here.

(Like if you google it now, this is the thread that pops up in said search results and it is the ONLY search result on Google. So that tells you how "special" true truly is.)

Unless I'm misunderstanding something, this would solve your immediate issues at the cost of being clunky but with the bonus of requiring zero setup or close to it.

Yes, it would work.

(I didn't even know that TrueNAS shipped with mc. I didn't even know that mc existed until this thread.)

I forget though - can sftp run data transfers in the background so that you can have it going whilst queuing up the next set of transfers?

It's been a really long time since I had to use (s)ftp "hard core", so I don't remember if there's an option to give it data transfer commands and have it running in the background whilst you browse to set up for the next set of data transfers.

And if there's a transfer that failed, will it only report out the error at the task level or would it report out the errors at the individual file level?

My concern is that if you are trying to move like half a million files, and there are errors, the log is going to be super large because it's logging half a million lines of failed data transfers.

Thanks.

 

[Patrick M. Hausen]

My concern is that if you are trying to move like half a million files, and there are errors, the log is going to be super large because it's logging half a million lines of failed data transfers.

If I was moving half a million files I would first and foremost make sure the copy is ok and complete. So either zfs send|receive or rsync. Turning a full copy into a partial copy can be done afterwards by deleting the things that are not needed. Can be done from the client, with multi-select, and the operation itself doesn't take much time.

 

[alpha754293]

So either zfs send|receive or rsync. Turning a full copy into a partial copy can be done afterwards by deleting the things that are not needed. Can be done from the client, with multi-select, and the operation itself doesn't take much time.

I'm a little bit confused by the last statement.

How do you get zfs send|receive with multi-select, using a client issuing commands to the servers, accomplish this?

I'm not sure that I follow the thought process/logic in terms of how this would be done.

Thanks.

 

[Patrick M. Hausen]

Use ZFS or rsync to copy everything, set up SMB share, then delete what is not needed ...

 

[alpha754293]

Use ZFS or rsync to copy everything...then delete what is not needed ...

Not possible. Destination doesn't have enough free space to be able to copy everything.

I'm still confused as to why the methodology is changing from using mc -x to using ZFS/rsync when it has already been determined that that's not a possible/feasible solution because it fails to meet the statement of requirements.

(N.B. This is half a million files out of a total of about 5.5 million files. Destination server can't absorb 5.5 million files. Insufficient free space.)

 

[Patrick M. Hausen]

Destination doesn't have enough free space to be able to copy everything.

I must have missed that part, sorry.

The reason why I am suggesting using a complete copy is that I don't trust drag & drop thingies. Like at all. How can I tell if the "graphical file manager" of choice really copied

• all hidden files
• all files with weird names
• all owners and groups
• all attributes
• all ACLs
• ...

correctly? When an rsync or ZFS operation finishes I know for certain that all data that was supposed to be copied was copied successfully. I don't trust anything but use of the command line for operations like that.
I can't recall how many times something like the Mac Finder or the Windows Explorer went to work and after a couple of hours of copying reported "oops, there was an errror ..." - without any possibility to tell what was copied and what wasn't and where precisely the error happened.

But you of course do you. I don't think further discussion will lead anywhere.