S3 Cloud sync stopped working

[Michaelxpo]

hello, i was wondering if someone could help me with a problem I have.
I have 2 machines that have 3 cloud sync tasks set to run on a schedule. on both machines, 2 tasks pass and a specific task fails with the error code below.
this worked before, and the tasks that fail are both to the same bucket, but I compared everything to the other buckets that pass and it's the same.
what could be the reason for this?

Code:

Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 349, in run
    await self.future
  File "/usr/local/lib/python3.7/site-packages/middlewared/job.py", line 386, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 961, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/cloud_sync.py", line 1007, in sync
    await rclone(self.middleware, job, cloud_sync)
  File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/cloud_sync.py", line 229, in rclone
    raise ValueError(message)
ValueError: rclone failed with exit code 1

thanks!

 

[Michaelxpo]

hey, just an update, this now affects 2 out of my 3 buckets. 1 was working fine yesterday. can someone please help? this is a critical backup, thanks!

 

[Michaelxpo]

Another update, it suddenly worked for 1 day on all buckets with nothing changed, and the next day one failed, and the day after that another failed again. nothing changed in the config. this seems like a bug.

 

[Samuel Tai]

Are you using the free tier of S3? Did you check your bucket to make sure you're still under? Have you tried regenerating your access keys?

 

[Michaelxpo]

hey, thanks for the reply. 1. it's a paid bucket. 2. the bucket still exists. 3. the current access keys work for a third bucket so I doubt it's the keys. but if nothing helps I'll try that as well.

 

[Michaelxpo]

still acting weird. without me changing anything now these buckets succeed, and I'm sure they'll fail again soon. the error message is not clear enough to understand what is causing this.

 

[Michaelxpo]

After weeks of working normally, this stopped working. it happened after I lost internet connection for one night when the sync was supposed to happen. ever since then, one bucket refuses to sync, while the other 2 finish successfully. I'm on 13.3 U4.

 

[Michaelxpo]

so i downloaded the logs, and found the errors, there were io errors, FreeNAS needs to show the errors of the logs, not what I showed in the OP.