Graphical file manager application/plugin?

[alpha754293]

But you of course do you. I don't think further discussion will lead anywhere.

Don't mistake my commentary as disagreement in regards to your points.

I totally understand and get all of your points and I agree with them.

Unfortunately though, when the intended destination doesn't have enough free space, you have to get really creative in regards to how to go about trying to accomplish the same task/goal.

And also unlike a professional IT department which also comes with a professional IT budget, as a home lab user, I can't just willy-nilly go out and spend like $5000 on more hard drives to address the "insufficient space" problem.

So? What are you left with?

Playing musical chairs with the data where you have to leverage the free space that you DO have, so the data gets distributed across multiple servers/systems as TWO systems are being rebuilt and then I will reconfigure the TrueNAS server, bring it back up online, confirm the deployment methodology/strategy, and then push all the data that used to live on it back onto to it.

THIS is why the graphical, unstructured, multi-select was such an important and critical requirement for me (due in part because I am having to distribute the data that's being evacuated on to multiple destinations before eventually sucking it all back in together).

In regards to your question of how do you check that the operations have been completed successfully?

Luckily, I don't have very many hidden files and the ones that I do have are all system generated thumbnail database files, so if I lose those, it really isn't a problem for me.

And also being a home lab user, ACL is also not an issue because it's just me that's reading/writing data to the servers. (Single user accounts makes ACL also a non-issue.)

But again, I agree with you, that in a professional IT setting, what I am doing here wouldn't fly.

But also in a professional IT setting, I would also have a professional IT budget behind it as well, so an issue like this (running out of space and/or having to rebuild a server rarely, if ever, happens like this as well).

In a professional IT setting, the ZFS replication for building new servers and migrating data is more likely what's going to be done because for some/most companies that has a professional IT department with said professional IT budget, spending another $30k on another server is worth the $30k vs. the time and/or the problems that may arise with spending less on a less stable/less reliable solution to the same task/problem.

At home, in a home lab setting, that's entirely different.

I think that it is important, for IT professionals, to keep the idea that home lab != professional IT department in mind and a LOT of that, unfortunately, is tied to money/budgets.

(Because if I had $4k to spend on the QNAP 12-bay 2U NAS unit which uses their new QuTS OS (which has ZFS on Linux) and everything else that I have been talking about here, plus another $5k in drives, I would've been all set and good to go with a ZFS replication/data migration strategy instead. But that's just not the case here. I had to get the wife to sign off on spending $1k on eight (8) refurbished HGST 10 TB SAS 12 Gbps 7200 rpm HDDs because the SAS drives were cheaper than the SATA drives (at the same capacity, believe it or not) and it just so happens that all of my HW RAID HBAs are all LSI SAS RAID HBAs, so the system/server was able to and ready to accept SAS drives.) Like I said, with basically little to no budget appropriated for IT maintenance, the solutions that are required to solve the same problems look drastically different compared to professional IT solutions. I think that sometimes, IT professionals maybe forget that a little bit or that it's a little bit "out of sight, out of mind".)

Thank you for your help.

 

[Phil1295]

Just to add my experience:
- A web-gui based file manager like I used in ReadyNAS OS would be in fact a nice addition
- For my media and file SMB/NFS libs, I don't like the copy/move between datasets using the client file manager because 1) we loose checksum copy, 2) it takes useless bandwidth on the network even if I do have a 10 Gb connection
- For this last part, I use replication tasks and shell with tmux session for most operations

However, I also use these two methods and they work amazingly good:
- KDE with Dolphin file browser installed on a jail with TigerVNC set as a service on jail start
- x11 SSH forwarding, which also works perfectly with all apps I tested. On a local LAN it is as fast as VNC, but obviously not for WAN to LAN or over high latency connections

I use VNC with KDE for apps like video remux and to manage data and media libraries through datasets faster than cli and without the limitations I mentioned when using SMB through the client

I posted a guide for how to properly setup both a KDE with VNC as a service and X11:

GUIDE: Setup secure GUI control on a jail using either VNC or X11

It is not easy to find guides for FreeBSD. In this guide, I explain with a near step by step how to: - install a KDE5 or GENOME 3 GUI in a jail - remote control the jail using either VNC or the X11 forwarding - install VNC as a service started on jail boot - properly setup a secure SSH...

www.truenas.com

 

[truecharts]

Just to add my experience:
- A web-gui based file manager like I used in ReadyNAS OS would be in fact a nice addition
- For my media and file SMB/NFS libs, I don't like the copy/move between datasets using the client file manager because 1) we loose checksum copy, 2) it takes useless bandwidth on the network even if I do have a 10 Gb connection
- For this last part, I use replication tasks and shell with tmux session for most operations

However, I also use these two methods and they work amazingly good:
- KDE with Dolphin file browser installed on a jail with TigerVNC set as a service on jail start
- x11 SSH forwarding, which also works perfectly with all apps I tested. On a local LAN it is as fast as VNC, but obviously not for WAN to LAN or over high latency connections

I use VNC with KDE for apps like video remux and to manage data and media libraries through datasets faster than cli and without the limitations I mentioned when using SMB through the client

I posted a guide for how to properly setup both a KDE with VNC as a service and X11:

GUIDE: Setup secure GUI control on a jail using either VNC or X11

It is not easy to find guides for FreeBSD. In this guide, I explain with a near step by step how to: - install a KDE5 or GENOME 3 GUI in a jail - remote control the jail using either VNC or the X11 forwarding - install VNC as a service started on jail boot - properly setup a secure SSH...

www.truenas.com

For SCALE we already have multiple file manager Apps available :)

 

[Phil1295]

For SCALE we already have multiple file manager Apps available :)

Until now, I never had crashes under FreeBSD servers, unlike my Ubuntu servers which crash sometimes more than Windows PCs...
As long as the FreeBSD version is supported, I won't move to a Linux based system.

Personally, I used a Netgear NAS debian based for years and still has one as backup machine target. They have a WebGUI browser that I rarely used. The reality is that for long operations, a browser can crash and is not really meant to be waiting for some processes to be completed.

Using VNC has the huge advantage of not relying on the client session. You can close it and your tasks will continue to run on the server. WebBased GUI browser, and X11 solutions are client dependent. The browser being the worst because it relies on the browser, in addition to the client.

Maybe just having a built in VNC solution over SSH is far better than spending time on a webGUI file browser. I understand that most users do not have the skills to go into configuring VNC or X11 in jails...

 

[Phil1295]

Here's a VNC screenshot using Dolphin, with the whole KDE desktop, at 4K resolution, managed remotely in a jail, with two datasets side by side.
It is just better and faster than any web based GUI file manager

 

[winnielinnie]

Here's a VNC screenshot using Dolphin, with the whole KDE desktop, at 4K resolution, managed remotely in a jail, with two datasets side by side.
It is just better and faster than any web based GUI file manager

Very cool, but...

• Only feasible to KDE users (technically desktop Linux/Unix users even without KDE, but requires extra packages and configuration)
• Requires setting up a VNC
• Requires setting up a jail with the mount points to all the datasets

On top of all of that, how practical would it be to use for multiple datasets across one or more jails? What about direct "server-to-server" (i.e, TrueNAS-to-TrueNAS), akin to what the OP wrote about in regards to QNAP's built-in GUI file browser that supports QNAP-to-QNAP?

I think the main gist of wanting a built-in GUI file browser is that it provides a familiar user-friendly interface (like Windows, Mac, or Linux file browsers); and is integrated with the TrueNAS GUI; and doesn't require the additional setup of any jails, VNC, and such.

 

[alpha754293]

The reality is that for long operations, a browser can crash and is not really meant to be waiting for some processes to be completed.

So...I think that it really depends on how this features is implemented.

For example, for QNAP NAS systems, their File Station basically runs all copy/move operations/tasks/jobs as background tasks/jobs that are issued to the server.

Whilst I don't know EXACTLY how their web GUI interacts with the underlying hardware (and software) itself, I will say that I've had NO problems with issuing multi-TB copy/move operations that run overnight and I don't need a live browser session for said operation to continue running in said background.

Suffice to say that even if my browser crashes and/or freezes up, as long as the copy/move operation/job/task is registered by the system, it will execute/act upon it.

So, as a web based GUI file management solution, it works decently well.

On top of all of that, how practical would it be to use for multiple datasets across one or more jails? What about direct "server-to-server" (i.e, TrueNAS-to-TrueNAS), akin to what the OP wrote about in regards to QNAP's built-in GUI file browser that supports QNAP-to-QNAP?

I agree with this question.

When you are trying to move a LOT of data (I was evacuating about 12 TB over the course of four days), because of the bimodal distribution in regards to the file size distribution, when it was moving a LOT of the smaller files, it made sense for me to try and parallelise that task as much as possible because a single transfer thread would only be using a very tiny amount of the line speed/bandwidth capability. Therefore; with parallelising the transfers, I would be able to make better/greater use of the bandwidth that's available.

(The data has been evacuated. I tried to consolidate the server all onto the TrueNAS server. VMs starting having issue under the heavy traffic load as I am putting all of the data "back" onto said TrueNAS server. As such, I ended up having to redeploy my smaller QNAP unit again where it will host the VM hard disk files, so in other words, the consolidation failed. (VMs would report that it lost connection to its hard drive file that was sitting on said TrueNAS server (whilst I was in the middle of pulling all of the data back together and sucking it all back up and into said TrueNAS server). By hosting the VM hard drive files on my smaller QNAP unit now, the VMs are up and running without any problems and the TrueNAS can still continue to suck all of the data back in from the distributed/shot gun effect that occurred when I was evacuating the data.) As such, my two bigger QNAP servers are sending about 16 threads worth of data each to the TrueNAS server.)

I don't know how well would a jail perform with 32-threads of network transfers either coming into the jail (to populate the underlying server) and/or leaving the jail (in a TrueNAS-to-TrueNAS selective migration).

 

[Phil1295]

On top of all of that, how practical would it be to use for multiple datasets across one or more jails? What about direct "server-to-server" (i.e, TrueNAS-to-TrueNAS), akin to what the OP wrote about in regards to QNAP's built-in GUI file browser that supports QNAP-to-QNAP?

Sure, you need a jail with mount points for the needed datasets and, as I mentioned, the skills to setup the Xorg environment. This is not affordable to most users I understand, but I wrote a step by step guide linked above. Since TrueNAS is free opensource, I personally, as a previous developer, prefer that the community focuses on root NAS features instead of GUI "plus". They already managed to make a FreeBSD unfriendly CLI OS affordable as a NAS GUI to the less techy users.

A GUI browser with advanced features like managing snapshots, remote server and server side file operations is just a huge project on its own for a NAS device. You need a php server that translates all this to shell commands + keep a GUI task progress detection. It would be great to have one. However, sometimes, open community looses the track by spending time and resources on features not part of the project, and thus, accelerating its own death.

(VMs would report that it lost connection to its hard drive file that was sitting on said TrueNAS server...)

FreeBSD Bhyve implementation is far from being optimal... and speed is just not its strength. Probably, Linux is a far better server to host VMs.
But, for heavy VMs, there are better solutions than TrueNAS, which is mainly a NAS and does it perfectly.

I don't know how well would a jail perform with 32-threads of network transfers either coming into the jail (to populate the underlying server) and/or leaving the jail (in a TrueNAS-to-TrueNAS selective migration).

Jails, in my environment, do perform the same in network / disk performance as the main TrueNAS host, across local 10 Gb connections.

As of your initial question, like said by others, the best migration / backup options between TrueNAS servers are replication tasks. They are just more powerful, faster and preserve all advanced file properties. The second choice is some scripts to perform rsync jobs between a TrueNAS host and a non zfs compatible host. I use both, the first between TrueNAS servers, and the second with Netgear BTRFS debian NAS. For Windows machines, I use backup software with SMB, that's the currently best approach for windows systems.

This supposes that you use your NAS as a NAS so that your data transfers can be structured to automate scripted tasks.

Sure, having a webGUI would be a nice addition for many users for casual Ctrl-C and manual select of sets of files, but I am not sure it would be the optimal answer to your transfers between servers.

 

[danb35]

Only feasible to KDE users (technically desktop Linux/Unix users even without KDE, but requires extra packages and configuration)

Huh? KDE is being installed in the TrueNAS jail, you can then use a VNC client under any OS to connect to it.

You're right that it's a lot of stuff to put in a jail, and that you'd need to mount all the relevant storage to the jail, but I don't understand this objection.

 

[winnielinnie]

Huh? KDE is being installed in the TrueNAS jail, you can then use a VNC client under any OS to connect to it.

You're right that it's a lot of stuff to put in a jail, and that you'd need to mount all the relevant storage to the jail, but I don't understand this objection.

Strikethrough that one. And the other thing I implied (Dolphin being familiar to KDE users) isn't so much a problem considering it follows a lot of the same principles and layouts as even Windows' Explorer. (Edited original post.)

 

[Ericloewe]

For masochists, I'm sure the GNOME setup is pretty simple and analogous...

 

[Patrick M. Hausen]

Jails, in my environment, do perform the same in network / disk performance as the main TrueNAS host, across local 10 Gb connections.

Of course. It's all symmetrical multiprocessing running on the same kernel. Unless you explicitly set (CPU, memory, ...) resource limits there is NO performance penalty imposed by the jail itself. The bridged networking (if you use it) hurts, but that is going to improve greatly with TN 13.

 

[Phil1295]

For masochists, I'm sure the GNOME setup is pretty simple and analogous...

LoL, I tried Gnome, and there is this section [OPTIONAL] Install gnome3 in my xorg-X11 guide linked above.
And I agree someone must be masochist because the Gnome BSD port is lagging behind Ubuntu releases and full of bugs. The most annoying being the black screen with multi-auth

 

[alpha754293]

They already managed to make a FreeBSD unfriendly CLI OS affordable as a NAS GUI to the less techy users.

Agreed.

Prior to this, the closest that I ever got to FreeBSD was via MacOS.

But of course, Apple makes FreeBSD (by way of MacOS) so simple that even my 1.5 year old is starting to learn how to manipulate our Mac Mini that runs the TV in our living room. And she doesn't/can't even read yet.

That's about the closest that I have ever gotten to FreeBSD prior to using TrueNAS.

A GUI browser with advanced features like managing snapshots, remote server and server side file operations is just a huge project on its own for a NAS device. You need a php server that translates all this to shell commands + keep a GUI task progress detection. It would be great to have one. However, sometimes, open community looses the track by spending time and resources on features not part of the project, and thus, accelerating its own death.

Yeah, I'm not really sure where people are coming up with the idea or think that that's what I'm asking for.

Snapshot management can already be handled, graphically, using the built-in web based, GUI that you already currently log into to manage the TrueNAS system.

So...in essence, outside of somehow, the idea coming up that people want to do it from within this discussion about a graphical file manager, is a little beyond my understanding/comprehension because I view tasks like taking snapshots as something that you manage OUTSIDE of the file manager level.

(Unless, of course, you only and specifically want to take snapshot of individual files. But I also think and assume that if that's what you want to do, there are other better tools for that.)

Between mc -x and @danb35's filebrowser.org solutions, those are two that, I think works. (As far as I can tell so far.)

And then there's also the KDE/Dolphin and your proposed solution as well.

This supposes that you use your NAS as a NAS so that your data transfers can be structured to automate scripted tasks.

This was the biggest requirement and confounding factor to the "defacto standards" that you mentioned above.

This was also why I deliberately and purposely explicitly mentioned unstructured data transfers because I understand that if the transfers were structured in any way, shape, or form; then I should have been able to script it, with rsync, at minimum.

But the reality was that because I had to evacuate the data onto which ever system had sufficient free space, it was a purely unstructured/random "shotgun" effect.

(And even then, one of the QNAP NAS units that I ended up pushing data onto was, itself, running out of space, so I had to partially evacuate a little bit of data off of that to make room for the incoming data stream from evacuating my smaller QNAP units.)

But, for heavy VMs, there are better solutions than TrueNAS, which is mainly a NAS and does it perfectly.

Yeah...I can't tell if the reason why my VM clients would be reporting a loss of connection to their hard drive file, which was being hosted on my TrueNAS server (that finally got redeployed) was because of the older Xeon hardware that I was running, TrueNAS, or a bit of both.

Hard to say.

Ironically though, since I had moved said VM clients' hard drive files over from the TrueNAS server and back onto my smaller QNAP NAS unit/server, the VM clients have been running without a hitch as I am also repopulating said smaller QNAP NAS unit with other, additional data as well that it will now become the host of.

And the VM clients didn't complain about losing connection/connectivity to its hard drive file. So, *shrug* - still can't tell the source/root cause of why the VM clients were reporting an issue.

Bottom line - I got the systems reconfigured to work now. I wanted to and wished that I was able to decommission the smaller QNAP NAS unit/server, but alas, that just turned out to not be the case, so it is what it is.

Thanks.

 

[Phil1295]

Of course. It's all symmetrical multiprocessing running on the same kernel. Unless you explicitly set (CPU, memory, ...) resource limits there is NO performance penalty imposed by the jail itself. The bridged networking (if you use it) hurts, but that is going to improve greatly with TN 13.

I use a few bridged jails on dedicates VLANs
All is fine as long as I do not try an iperf3 server on the jail. The Network stack on all bridges crashes and a TrueNAS restart is needed
So yes, the bridged network just doesn't perform as expected on high demand. It can still reach a 8 Gbps when it doesn't crash

 

[Patrick M. Hausen]

Does the TrueNAS host have an IP (v4/v6) address in that bridged network? If yes, you did put that on the bridge interface and not on the physical or VLAN that is a bridge member?

 

[Phil1295]

Does the TrueNAS host have an IP (v4/v6) address in that bridged network? If yes, you did put that on the bridge interface and not on the physical or VLAN that is a bridge member?

No, TrueNAS host is bound to a different physical interface on a different VLAN

The default iperf3 command from client runs fine most of the time

Code:

iperf3 -u -c 10.0.50.55 -b 10G -l 24k -w 512K -i 1 -R

If I increase the socket buffer size, it will crash all the bridges on ixl1 interface, even those not on the same VLAN. Interface ixl0 is preserved

Code:

iperf3 -u -c 10.0.50.55 -b 10G -l 24k -w 1M -i 1 -R

By the way, setting kern.ipc.maxsockbuf=16777216 in /boot/loader.conf inside jail doesn't seem to apply and iperf3 cannot be started with high buffers

Here's a sample bridged VLAN in interfaces:
VLAN 50:
Is set to physical interface ixl1, no IP, only VLAN tag

Code:

- Type: VLAN
- Name: vlan50
- Description: VLAN 50
- DHCP: unchecked
- Autoconfigure IPv6: unchecked
- Parent Interface: ixl1
- Vlan Tag: 50
- Priority Code Point: leave empty
- Disable Hardware Offloading: unchecked
- MTU: 1500
- Options: up
- IP Address: empty

Bridge 50:
members: vlan50

Code:

- Type: Bridge
- Name: bridge50
- Description: VLAN 50 Bridge
- DHCP: unchecked
- Autoconfigure IPv6: unchecked
- Bridge Members: vlan50: VLAN 50
- Disable Hardware Offloading: unchecked
- MTU: 1500
- Options: up
- IP Address: empty

TrueNAS host:
VLAN 90 on the switch
Listening on physical interface ixl0, static IP 10.0.90.30

Jail on VLAN 50:
interfaces: vnet0:bridge50

Code:

#
- DHCP Autoconfigure IPv4: unchecked
- NAT: unchecked
- VNET: checked
- Berkeley Packet Filter: unchecked
- vnet_default_interface: none
- IPv4 Interface: vnet0
- IPv4 Address: 10.0.50.55
- IPv4 Netmask: 24
- IPv4 Default Router: 10.0.50.2
- Autoconfigure IPv6: unchecked
- IPv6 Interface: empty
- IPv6 Address: empty
- IPv6 Prefix: empty
- IPv6 Default Router: none

# Network Properties:
- interfaces: vnet0:bridge50
- host_domainname: WORKGROUP
- host_hostname: streaming
- resolver: search intranet;nameserver 10.0.50.2
- exec_fib: 0
- ip4.saddrsel: checked
- ip6.saddrsel: checked
- ip4: New
- ip6: New

 

[Patrick M. Hausen]

By the way, setting kern.ipc.maxsockbuf=16777216 in /boot/loader.conf inside jail doesn't seem to apply and iperf3 cannot be started with high buffers

That's a kernel setting. Jails run on the same kernel as the host. You cannot change these parameters from inside a jail.
Try creating a tunable for the host in the UI.

Bridging is improved in FreeBSD 13. You might want to try the TrueNAS BETA.

 

[Phil1295]

That's a kernel setting. Jails run on the same kernel as the host. You cannot change these parameters from inside a jail.
Try creating a tunable for the host in the UI.

Bridging is improved in FreeBSD 13. You might want to try the TrueNAS BETA.

Thank you a lot, I completely forgot about the kernel tunables to be set in main host

After editing the kernel tunables, I can properly run the iper3 server in jail and the physical interface is no longer crashing
Bridged jail:

Code:

[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   889 MBytes  7.45 Gbits/sec  88124
[  5]   1.00-2.00   sec   895 MBytes  7.51 Gbits/sec  77937
[  5]   2.00-3.00   sec   895 MBytes  7.51 Gbits/sec  76748
[  5]   3.00-4.00   sec   891 MBytes  7.48 Gbits/sec  76549
[  5]   4.00-5.00   sec   894 MBytes  7.50 Gbits/sec  81232
[  5]   5.00-6.00   sec   891 MBytes  7.47 Gbits/sec  88172
[  5]   6.00-7.00   sec   892 MBytes  7.48 Gbits/sec  87675
[  5]   7.00-8.00   sec   891 MBytes  7.47 Gbits/sec  87692
[  5]   8.00-9.00   sec   886 MBytes  7.43 Gbits/sec  87296
[  5]   9.00-10.00  sec   892 MBytes  7.48 Gbits/sec  87496
[  5]  10.00-10.01  sec  8.60 MBytes  6.50 Gbits/sec  999
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.01  sec  8.71 GBytes  7.48 Gbits/sec  0.000 ms  0/839920 (0%)  sender

The result is really good I find and on large bandwidth connections, despite the 20% average loss in bandwidth
TrueNAS host:

Code:

[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  1.12 GBytes  9.61 Gbits/sec  48882
[  5]   1.00-2.00   sec  1.12 GBytes  9.61 Gbits/sec  48859
[  5]   2.00-3.00   sec  1.12 GBytes  9.61 Gbits/sec  48885
[  5]   3.00-4.00   sec  1.12 GBytes  9.58 Gbits/sec  48751
[  5]   4.00-5.00   sec  1.12 GBytes  9.61 Gbits/sec  48856
[  5]   5.00-6.00   sec  1.12 GBytes  9.61 Gbits/sec  48885
[  5]   6.00-7.00   sec  1.12 GBytes  9.61 Gbits/sec  48884
[  5]   7.00-8.00   sec  1.12 GBytes  9.61 Gbits/sec  48884
[  5]   8.00-9.00   sec  1.12 GBytes  9.61 Gbits/sec  48888
[  5]   9.00-10.00  sec  1.12 GBytes  9.61 Gbits/sec  48882
[  5]  10.00-10.03  sec  36.4 MBytes  9.60 Gbits/sec  1553
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.03  sec  11.2 GBytes  9.61 Gbits/sec  0.000 ms  0/490209 (0%)  sender

 

[ChrisRJ]

When I look at the length of some posts and the overall amount of time spent on the subject, I get the feeling that the actual goal could have long been achieved in multiple ways by now.

Yes, this is not really helpful. But as much as I love a good discussion, a certain level of pragmatism does not hurt. And I couldn't resist ...

Have a great weekend, everyone!