Transfer from OLD NAS: Rsync, overnight, with crontab

[JesseJ80]

I have just recently setup my first TruNAS server, I'll give Hardware and setup details at the bottom.

I need to move data from an old NAS (Dlink DIR-323) to the new TrueNAS server, there is no direct way to do this from the UI so from the webshell I mounted the 2 SMB1 volumes into new directories on TrueNAS, and wrote a script to do the transfer with rsync and to email me each night with a status report and a final email when complete, then tried to add a cron task to do the transfer over night. I can manually start the script and it will transfer files just fine, but it doesn't start automatically at the appointed time. The reason I am trying to do it this way is that if I use another computer to tranfer the files, it cuts the already abysmal speed in half bringing it on a Windows machine, then back out to the TrunNAS server. The Dlink DNS-323 was amazing when it came out in 2006, and has served me faithfully for 18 years now, but its abysmally slow by todays standards.

That's the first issue. The second being that last night and today, having given up on scheduling the task, I tried to start the job manually and let it run, but as soon as the user session on the webui goes to sleep and logs out, it stops the transfer, even if the shell session is detached.

It seems to me that some basic functionality is missing that I cant really understand being missed this long.

1. An obvious way to pull files from another location that isn't limited to "module"/"SSH" with rsync capable devices and locations. For Example: shared files/folders on windows/mac, older SMB1 shares.

2. Ability to add cron tasks in the webui that allow the use of hyphens (-) so that a task can run from a start time to and end time. For Example
Code:
0 23-7 * * * /path/to/file.sh


3. I should be able to control whether a session logs out or stays active, this should be a basic setting. Especially since this also has VM support, and could be used as a primary hypervisor for home labs and even small businesses.


Now that I have complained, what are the work arounds for these issues currently. How do I actually move the files without occupying another device and slowing down the transfer that will already take approximately 2.5 days? Am I missing something obvious?


Setup:

Server - Supermicro X9DRH based server: 2x E5-2690 v2, 256GB 1800Mhz RAM, 8 x 2.15GB 2.5" SAS HDD w/LSI 2208 Controller(dont bug me, I made it work)
Hypervisor - Proxmox
VMs - PFSense - Firewall, router, ad blocking for the whole house.
- Ubuntu Sever - Docker Lab and Game server
- TrueNAS Scale - has each drive from the LSI 2208 passed through with SN.

Thanks in advance.

 

[JesseJ80]

 

[sretalla]

2. Ability to add cron tasks in the webui that allow the use of hyphens (-) so that a task can run from a start time to and end time. For Example
Code:
0 23-7 * * * /path/to/file.sh

That won't do what you think it will do... even if it was syntax that cron would understand.

Launching a script happens at a moment in time, not over the period of 8 hours. What the script then does is up to the script.

If you want an rsync task to run, perhaps using an rsync task from the GUI is more useful, specifying either the --stop-at or --stop-after auxiliary parameters.

or you can modify your script to use one of those parameters and continue using cron with a start time only (as it's designed).

 

[JesseJ80]

From everything I have ready, that's impression I got from being able to define a "range" for a process to run. If that is mistaken please point me in the right direction to learn more.

I'm losing my mind!!

I don't even know how to describe my frustrations with this whole process! For Something SOOOO simple as moving files from one NAS to another, I should not be met with continual walls.

After more research last night, you are right, a "range" for cron is not a rang of time for a process to run, it is a range of time to define an interval to "launch" a given command multiple times. which is completely useless, and could be accomplished by adding loops in a script. running a process for a given amount of time would be far more useful I think.. Which means for something as simple as running a rsync for 8 hours over night, I would have to write 2 scripts and put 2 entries in the crontab.

So I gave up on that and just decided to do a cp over night and check on it in the morning. Well that didn't work because as soon as my computer went to sleep, the cp command which was running in the CLI on the actual server, accessed through the Proxmox console, errored out.

So, I manually tried to run the rsync from the same cli and I keep getting permission errors, operation not permitted errors, files/attrs were not transferred errors, then it stops. I've made all the files and directories full 777 permissions, I've tried to not copy meta data by removing -a which just says there are files, then doesn't copy anything.

As I said before this is an OLD NAS, I can't ssh into it, the only access I have is as an SMB1 share, which means adding an rsync task in the webui is impossible, because it requires ssh or module to be defined.



GAHHHHHHH!!!!!!!!!


What am I missing!?!? How am I supposed to move stuff onto it?

 

[jgreco]

From everything I have ready, that's impression I got from being able to define a "range" for a process to run. If that is mistaken please point me in the right direction to learn more.

What you have above is broken. A crontab entry can launch a cron entry at intervals that are as long as once a year, or as little as once per minute.

Minutes: 0
Hours: 23-7

My best guess is that you meant this to mean "hours between 23:00 and 07:00", but that's not how computers interpret numbers. That's how some humans sometimes interpret numbers. To cause a crontab to execute on "hours between 23:00 and 07:00", what you need is

Minutes: 0
Hours: 0-7,23

which covers 00:00, 01:00, 02:00, 03:00, 04:00, 05:00, 06:00, 07:00, and 23:00.

But also note that the job will actually be spawned at each of these times. It isn't a time specification that only allows the cron job to run during "23:00-07:00" which seems to be what you are thinking. If you have a six hour cron job and launch it at 07:00, it will still be running until about 13:00.

and could be accomplished by adding loops in a script.

But the idea is to have the system take care of figuring out the start time of the job. Lots of systems, including
UNIX and Windows, work this way.

running a process for a given amount of time would be far more useful I think..

That sounds useless to me. Most cron tasks do not have a fixed duration, and many of them need to run "until completed". Many tasks such as backups, or rsync jobs, or replications, would be nearly useless if they were interrupted mid-job.

Which means for something as simple as running a rsync for 8 hours over night, I would have to write 2 scripts and put 2 entries in the crontab.

Why? If you WANT it to die after 8 hours, why not just wrap it in timeout(1)? What's the purpose of the second script? Cleanup? Just do that from within the first script, after the timeout-wrapped command(s).

Well that didn't work because as soon as my computer went to sleep, the cp command which was running in the CLI on the actual server, accessed through the Proxmox console, errored out.

Why is your computer going to sleep? This is very confusing. This feels like a self-inflicted injury. If your problem is caused by your computer going to sleep, choose a power plan that does not include sleep mode. You can easily configure both PC's and laptops to only sleep if explicitly told to sleep, and people working with ssh or virtual terminals really NEED to do that.

As I said before this is an OLD NAS, I can't ssh into it, the only access I have is as an SMB1 share, which means adding an rsync task in the webui is impossible, because it requires ssh or module to be defined.
GAHHHHHHH!!!!!!!!!
What am I missing!?!? How am I supposed to move stuff onto it?

You're a sysadmin. You think about the variables of the problem. You only have SMB1 access? Then by all means, set up one of your PC's with sleep disabled and use robocopy or xcopy or whatever your preference is to copy your stuff using SMB, and just let it run for a week or whatever is necessary.

 

[sretalla]

tmux and/or read my suggestion about using the switches with rsync.

What @jgreco suggested is also good logic.

 

[JesseJ80]

Yes I already figured out that the way the documentation for cron is written has no comprehension of the difference between a interval of time and a range of time.

For example:

• 00 8-16 * * * /driveway/car/start.sh
  • means go start the car every hour between 8am and 4pm (on an interval-set). Not start the car at 8am and let it run till 4 pm (for a Range).

That sounds useless to me. Most cron tasks do not have a fixed duration, and many of them need to run "until completed". Many tasks such as backups, or rsync jobs, or replications, would be nearly useless if they were interrupted mid-job.

It is really simple, I want to move the data to the new Truenas server, I ONLY want to move data overnight, I DON'T want to have to babysit a file transfer for a week. I DON'T want to use a 3rd device to move the data, cutting speed down by 50-75%, because of the limitations of the old NAS.

Why? If you WANT it to die after 8 hours, why not just wrap it in timeout(1)? What's the purpose of the second script? Cleanup? Just do that from within the first script, after the timeout-wrapped command(s).

Because while trying to find ways to stop a running task at a certain time, nothing has ever shown anything about using timeout. Especially since every solution came back to use cron and a range.

Why is your computer going to sleep? This is very confusing. This feels like a self-inflicted injury. If your problem is caused by your computer going to sleep, choose a power plan that does not include sleep mode. You can easily configure both PC's and laptops to only sleep if explicitly told to sleep, and people working with ssh or virtual terminals really NEED to do that.

My computer going to sleep should have ZERO impact on a process running on another machine altogether.

You're a sysadmin. You think about the variables of the problem. You only have SMB1 access? Then by all means, set up one of your PC's with sleep disabled and use robocopy or xcopy or whatever your preference is to copy your stuff using SMB, and just let it run for a week or whatever is necessary.

My household needs access to the data during the day: media shares, project files etc, and, if I can move the files with another machine, why can't I just as easily move them with TrueNAS?

 

[JesseJ80]

tmux and/or read my suggestion about using the switches with rsync.

tmux still times out when it is not the active browser window or browser tab, thats why I went right to the console through Proxmox.
Again, cant use the webui for this task, the old NAS has ZERO support for rsync modules or ssh. That's Why I mounted the volumes into TrueNAS and am trying to do things this way.. there seems to be an unexpressed assumption that data will only ever be moved from or via another computer, or with rsinc to or from a reasonably new device.

What @jgreco suggested is also good logic.

Yes, I am looking into that, but I still need to deal with the errors that are driving me bonkers.

 

[JesseJ80]

I did find another post that @lietu provided a script to allow changing the timeout-logout . that appears to have worked. Honestly its a basic setting that should be able to be controlled directly from the WebUI. I can't imagine the thought process behind locking it away and changing it frequently.

 

[jgreco]

way the documentation for cron is written has no comprehension of the difference between a interval of time and a range of time.

Sure it does. "Interval of time" is not relevant to cron. This is only a thinking issue on your part. Something that is irrelevant is not contemplated in the documentation for cron. Cron has been around for a hell of a long time, so I don't think this is a cron problem either. There are a number of versions of cron, and I do not recall anything like what you're talking about in ANY of them.

It is really simple, I want to move the data to the new Truenas server, I ONLY want to move data overnight, I DON'T want to have to babysit a file transfer for a week. I DON'T want to use a 3rd device to move the data, cutting speed down by 50-75%, because of the limitations of the old NAS.

Okay, that's nice. So put a VM up on the new NAS and do your moves there. This really just comes down to the fact that it is unlikely anyone is going to have an exact solution to your very particular situational requirements, so then you are left to do what any good sysadmin does when confronted with an unusual situation.

Because while trying to find ways to stop a running task at a certain time, nothing has ever shown anything about using timeout.

It's a UNIX system and if you're dealing with cron, then you're already running arbitrary UNIX commands like rsync. You want a weird capability. UNIX allows you that weird capability, as well as a number of others, such as pv for rate limiting, limit for resource limiting, or nice to control CPU utilization. It is really your responsibility to know basic scripting, and timeout is a common utility to FreeBSD, Linux, macOS, and even Windows.

Especially since every solution came back to use cron and a range.

No solution came back to "use cron and a range." Cron does not support this sort of limit. You probably just misunderstood. That's fine, we can fix misunderstandings.

 

[garm]

I wouldn't trust the copy if it continuously got interrupted, especially if its run with robocopy or xcopy. Moving the data while random family member still have access also sounds like a nightmare.. just bite the bullet, take the old NAS "offline" and start rsync from a VM or other suitable client and let it run to completion, then take a snapshot of the data and run it again to check that you got everything

 

[JesseJ80]

I wouldn't trust the copy if it continuously got interrupted, especially if its run with robocopy or xcopy. Moving the data while random family member still have access also sounds like a nightmare.. just bite the bullet, take the old NAS "offline" and start rsync from a VM or other suitable client and let it run to completion, then take a snapshot of the data and run it again to check that you got everything

That is exactly why I started down this goat trail. I thought using rsync would be one of the safest and easiest ways to move data, and would also be able to verify that it had moved correctly, with only basic automation to make it a set-it and forget-it thing.


Its all Moot at this point, I am pretty sure I had solved all my problems, but its still not working. If I had any idea it would be this difficult to move large amounts of files to the new NAS last week I would have just set another machine to move the files, and it would be done already. So, I give up doing this logically and efficiently with basic automation. I'll manually move a few folders at a time and babysit it over then next couple weeks in the evenings.

Thanks for trying to help. Yes I'm a bit aggravated by all this. The whole thing feels like I've been looking for something that I thought would be a automatic consideration for a NAS setup, and it seems to not even be a side note.

 

[jgreco]

I thought would be a automatic consideration for a NAS setup, and it seems to not even be a side note.

Sure, it's a consideration. That's why you're allowed to import disks under the Storage menu. But once you start talking about another NAS unit, then that's more complicated. You're expected to do it with something rational like rsync. If your old NAS doesn't support that, it isn't really clear what you expect here.