SOLVED Cloud sync with Backblaze B2 limit upload speed

[Mr. Slumber]

Following the mindset better save than sorry I maintain a local onsite backup of my files on a different server and thanks to FreeNAS 11.2 an encrypted offsite cloud backup. Done that for some years with Backblaze and never had a problem.

There are lots of howtos on how to achieve this with FreeNAS e.g. Backing up my FreeNAS to Backblaze B2

Did that, works great but steals too much upload bandwidth so I'm searching for a way to limit the cloud sync upload speed in FreeNAS.

Thank you for your help in advance! :)

 

[KazuyaDarklight]

From what I've seen, a bandwidth throttling setting is currently flagged as a feature request that will be implemented in 11.3 *fingers crossed*. In the meantime, your only options are to A) configure your backup via shell, as the rclone process that actually handles the work does have a bandwidth setting available, its just not in the webui, or B) if you have or want to buy an advanced enough router, you can throttle your FreeNAS boxes upload speed there. That's the one we are doing at my work. We have any outbound from the FreeNAS IP throttle during working hours and then it has free reign at night.

 

[Mr. Slumber]

have or want to buy an advanced enough router

pfSense VM and very happy with it :)

you can throttle your FreeNAS boxes upload speed there

OK, will have to look in this in depth because not sure on how to set this up at the moment

Thanks for your post!

 

[dlavigne]

From what I've seen, a bandwidth throttling setting is currently flagged as a feature request that will be implemented in 11.3 *fingers crossed*. In the meantime, your only options are to A) configure your backup via shell, as the rclone process that actually handles the work does have a bandwidth setting available, its just not in the webui, or B) if you have or want to buy an advanced enough router, you can throttle your FreeNAS boxes upload speed there. That's the one we are doing at my work. We have any outbound from the FreeNAS IP throttle during working hours and then it has free reign at night.

Definitely coming in 11.3: https://jira.ixsystems.com/browse/NAS-100390.

 

[Mr. Slumber]

Definitely coming in 11.3

Cool, thanks! :)

 

[Mr. Slumber]

Definitely coming in 11.3: https://jira.ixsystems.com/browse/NAS-100390

Any update on this topic? Can't find it in 11.3-U2 still...

Thanks for a response. :)

 

[Mr. Slumber]

Ok did some reading, edited my cloud sync task and it works. Thanks for implementing this :)

Details can be found here and here.

 

[MattFYI]

I upgraded to 11.3-U3.2 but do not see this feature when editing a cloud sync task - any ideas?

 

[Mr. Slumber]

Yes. Please take a look here: Tasks / Cloud Sync Tasks / Edit

There you can enter (at the very bottom part) e.g. the following settings = 100kB/s max upload speed all the time everyday

 

[MattFYI]

thanks - i did not have the rsync service enabled so was not seeing this feature on the cloud sync task

 

[MattFYI]

doesn't seem like the weekday option works, Mon-06:00,512 17:00,off returns this error = [EINVAL] cloud_sync_update.bwlimit.0.cloud_sync_bwlimit.time: invalid literal for int() with base 10: 'Mon-06'
any ideas?

 

[Samuel Tai]

Unfortunately, 11.3 only implements a single bandwidth limit per cloud sync task in /usr/local/lib/python3.7/site-packages/middlewared/plugins/cloud_sync.py, line 156. You'll have to use "0:00,512KB" 24x7.

 

[MattFYI]

ok thanks - i ended up throttling with the firewall

 

[elorimer]

I had the reverse problem. My Backblaze upload was only going at 11mb/s, when my upload capacity is around 35mb/s.

"transfers" is a setting next to the bandwidth setting. I entered 32 there (the default is 4), and my upload went to the full 35 mb/s. This reduces my upload time to 1 day from 3 days. I don't know where between 32 and 4 the speed tops out to my connection speed. This brings my 4-core CPU to around 30%.

Edit: 12 gets me to the same speed, with CPU still at 30%.

 

[RichardPimple]

Unfortunately, 11.3 only implements a single bandwidth limit per cloud sync task in /usr/local/lib/python3.7/site-packages/middlewared/plugins/cloud_sync.py, line 156. You'll have to use "0:00,512KB" 24x7.

Actually you can specify multiple limits during the 24hr period - I have jobs running during the day with bandwidth limits and full throttle overnight, like this:

In 11.3 GUI you enter each one separately pressing 'enter' between them, in this format: 08:00,0.625M and 21:00,off.
The M after the number denotes MBps, so 0.625M is 5Mbps (5/8 = 0.625)

 

[RichR]

Anyone know if there are plans to implement this fully, as from the link in the 11.3 GUI? https://rclone.org/docs/#bwlimit-bandwidth-spec

What I'd like to do is:
Mon-07:00,10M Mon-19:00,50M
Tue-07:00,10M Tue-19:00,50M
Wed-07:00,10M Wed-19:00,50M
Thu-07:00,10M Thu-19:00,50M
Fri-07:00,10M Fri-19:00,off