Slow Cloud Sync Task pull from Onedrive for business

[jevan23]

Hi All,

I'm running TrueNAS-SCALE-22.12.3.3 on the system in my signature. Performing a cloudsync pull of about 500GB from a Onedrive for business account to the TNS system over a 300Mbps internet connection, and I'm about 4 hours in and have only transferred about 80GB of data. The sync configuration has bandwidth limited to 25MB and the sync starts out strong, but then the transfer speed drops to bytes/sec, with the odd shortlived burst back up to 25ish MBps.

I've tried different NIC cards (all intel), different routers(pfsense & asus consumer), even different mobo/Cpu combos for the Truenas system (Asus Z690 Pro w/ i5-13500), but get the same result. The addition of a LOG VDEV to the pool seemed to help keep the transfer speed up a little longer after starting the task, but then it drops again. CPU is sitting at 1% and half the memory is free.

The dataset being written to has the following settings: Sync=standard, LZ4 compression, Atime off, ZFS Dedup off, and the pool is encrypted. The data is all smaller files.

Can anyone point me in the right direction to get these speeds up, or what could be the bottleneck? Transferring about 3GB/hour will take days..

Started transfer at 16:12, ran flawlessly for about half an hour then..

this.

 

[jevan23]

Does anyone else regularly backup more than 100GB of files from Onderive to TrueNAS? If you get solid transfer speeds can you share your setup and configuration?

 

[Webo]

I'm having the same problem syncing my personal OneDrive to my TrueNAS-SCALE-22.12.3.3.I have a 250/250 internet connection and it starts good and maxing the line as I want to get it down fast (i have set it to "High bandwidth (16)" under the cloud sync settings) but then goes down to very slow speeds for some time and then up again and down again. I have my Synology NAS set to sync my OneDrive space and it run full speed untill it's done.

I have tested to stop the sync and and run it again but it just starts with good spped and then goes down again to almost nothing and then up and down.

 

[Webo]

I'm having the same problem syncing my personal OneDrive to my TrueNAS-SCALE-22.12.3.3.I have a 250/250 internet connection and it starts good and maxing the line as I want to get it down fast (i have set it to "High bandwidth (16)" under the cloud sync settings) but then goes down to very slow speeds for some time and then up again and down again. I have my Synology NAS set to sync my OneDrive space and it run full speed untill it's done.

I have tested to stop the sync and and run it again but it just starts with good spped and then goes down again to almost nothing and then up and down.
View attachment 69786
View attachment 69783

View attachment 69784
View attachment 69785

This is how it look last few hours

 

[Webo]

This is how it look last few hours
View attachment 69794

And disk and CPU look like this

So there is no problem with resources or so and no problem with internet beeing used for other things either

 

[jevan23]

I'm having the same problem syncing my personal OneDrive to my TrueNAS-SCALE-22.12.3.3.I have a 250/250 internet connection and it starts good and maxing the line as I want to get it down fast (i have set it to "High bandwidth (16)" under the cloud sync settings) but then goes down to very slow speeds for some time and then up again and down again. I have my Synology NAS set to sync my OneDrive space and it run full speed untill it's done.

I have tested to stop the sync and and run it again but it just starts with good spped and then goes down again to almost nothing and then up and down.

Can you share your Truenas system specs and dataset configuration details?

 

[Webo]

This is how it looks right now and during the day

 

[Webo]

I had some strange time offsets. Dashboard said 2,5 hours ahead of time, reporting was right time and TrueNAS shell has 2 hours ahead of time. Searched for it and tried to solve it with the solutions I found but no luck so I thought I might as well change the boot drive anyway even if the speed was good. WOrked doe 1,5 hours to fit in a 2,5" SSD in the 3.5" hotswap cage and the plate I had made the connection ports sit to high so could not insert the disk but managed to get it working without a plate to fasten the SSD drive in as I screw it in the frame of the cage and then had som fiddling to get it to go into the connectors. I had saved the config before I turn it of and just did a reinstall och TrueNAS Scale again and when in I restored the config and it rebooted. After that my time is correct in Dashboard, Reporting and TrueNAS shell and so far it's running fast.

I'll see if it continiues all night as I'll get to bed and get 2,5 hours sleep before work starts
I'll update in bout 3-4 hours with info on hos it's going with the speed.

 

[Webo]

It went ok and it has now synced all files

So save config and reinstall and import config seems to have solved all my problems (3 different times showing in TrueNAS Scale depending where I look) and also the Cloud Sync. So maybe you can try the same and see if that solve your problem also?

 

[MrGuvernment]

I recall back when I was trying to backup my OneDrive to my QNAP, reading that OneDrive throttles connections and download speeds on anything other than their client. of course I can not find that info now though.

[Edit] Some notes from others like Veeam

M365 Throttling

Best practice from the field for Veeam Backup for Microsoft 365.

bp.veeam.com

Wonder if the TrueNAS app has a config file that can be tuned similar to this?

Performance tuning and Throttling configuration in M365 OneDrive plugin of Backup Exec

www.veritas.com

 

[sfatula]

I recall back when I was trying to backup my OneDrive to my QNAP, reading that OneDrive throttles connections and download speeds on anything other than their client. of course I can not find that info now though.

Yes, I agree. Just because your (op) internet connection is not saturated and is full speed does not mean the other end (Onedrive) is always full speed.

 

[jevan23]

Didn't even occur to me that Microsoft would throttle a 25MB limited transfer... But MrGuvernment's links and some reading on microsofts site about throttling limits, yeah that could very well be the issue.

 

[MrGuvernment]

Ya, who knows how they may be doing it, but seems they are definitely doing something. Maybe when I am back from the long weekend, I can get a clean Win install, connect my onedrive and get some large files on it for download, and then monitor the down speeds via OneDrive, and then try via my QNAP and maybe TrueNAS just to see...