LFTP causing Invalid File Handle?

[briarknit]

I have a couple of cron jobs that are set up to sync my remote server's files down to my local freenas server throughout the hour. I have a total of 4 jobs that run. One job for movies, one for tv shows, and one for everything else. I also have a fourth one that syncs files up to the remote server in case theres anything that needs to go up.

My issue is on one particular folder that is a part of my dataset. I have 4 WD Red 3TBs in RAIDZ1 (yes I know RAIDZ1 is not recommended, that is not what this post is about), and I have a 750GB dataset in that pool that holds Movies and TV shows (in a Movie and TV folder, respectively). My issue is that the Movies folder always works fine, but after the LFTP script that syncs files to the TV folder runs I notice that afterwards the TV folder will throw an error on my Windows machine saying "The file handle is invalid."

I thought at first maybe one of my drives was DOA since these drives are less than a week old, so I ran SMART tests and even a scrub on the pool and both came back clean. Whats strange is that if I force-refresh the permissions on that dataset the problem goes away and I am able to browse and use files in that directory again until the next time LFTP runs.

What about LFTP could be causing this? Each sync script runs 5 parallel downloads maxed a total of 2.5Mbps each (so 10 Mbps among all 4 scripts). Could it be that the scripts are running too close together and are causing too much load to be put on the server? Not sure how that could cause the issue I am having but I am running out of ideas here.

My next test will be to disable all of my cron jobs that run those scripts and see if the problem occurs again. From there I can possibly narrow it down to a particular script if that is really the issue, but I was wondering if anyone here had ever experienced anything like this before?

 

[nightshade00013]

From what I can tell from this https://blogs.msdn.microsoft.com/oldnewthing/20141031-00/?p=43723 though it is not for the exact same problem is that the LFTP is in some way corrupting the files when making the copy.


It could be a permission problem in your case however It depends on which group the LFTP program is running under and what it is trying to set the permissions as when it is done.

What I have setup for my twin FreeNAS boxes is done using Syncthing, each one runs it and one is setup to use trashcan versioning incase of any issues. I did however have to change a setting to ignore file permissions on each end or it would have issues.

 

[briarknit]

From what I can tell from this https://blogs.msdn.microsoft.com/oldnewthing/20141031-00/?p=43723 though it is not for the exact same problem is that the LFTP is in some way corrupting the files when making the copy.


It could be a permission problem in your case however It depends on which group the LFTP program is running under and what it is trying to set the permissions as when it is done.

What I have setup for my twin FreeNAS boxes is done using Syncthing, each one runs it and one is setup to use trashcan versioning incase of any issues. I did however have to change a setting to ignore file permissions on each end or it would have issues.

Thats what I was thinking, since technically the first cron job was running 4 times an hour, the second twice and hour, and the 3rd and 4th once an hour. Each task had LFTP set to use 5 threads to download files. So at any point LFTP could've been trying to spin up 20+threads, all trying to write to the same disk at the same time.

I think what I am going to try is instead of having a separate script for everything I will have a single script to LFTP all files from my server (this worked great when I only had one cron job and one script) and another script that simply goes through the primary download directory and does a simple mv -rfv on the files and moves them to their appropriate folder (folders labeled with "movie" get moved to the movie folder for plex, likewise for TV). Hopefully this will solve this issue. If not I will try to limit LFTP to a single thread and go from there.

If all else fails I can do what you suggested by using syncthing between the two servers instead of LFTP.