TrueNAS Google Drive Cloud Sync Task takes very long

[Midblep]

Recently I have started using TrueNAS as a local NAS on an old computer with a bunch of hard drives in it. I synchronize some data from my PC to TrueNAS with rclone as I'm on LInux, and then want to synchronize that data to my Google Drive to have an off-site backup just in case. I just PUSH SYNC, as the NAS should be the sole controller of the data.

I have about ~100 - 200 GB of data stored, but the syncs take very long (multiple days to a week), and the network traffic is 0 bytes/s most of the time. I am properly authenticated and this issue does not resolve after some time like a ratelimited Google Drive speed would suggest. The task pictured has started yesterday as I tried to see if restarting would fix it. I own a Gigabit internet connection with no datacaps from my ISP. I can download other things at speed fine, and uploading files to TrueNAS is fast.

Is there anything I am missing or I am able to do to make this sync faster? Thanks a ton for any help. I am new to TrueNAS since a few weeks so if I need to post any more info or logs please let me know.

Version: TrueNAS-13.0-RELEASE

 

[elvisimprsntr]

Welcome!

1. Who is your ISP and what is your up link bandwidth?
2. Have you tried disabling Take Snapshot?
3. What happens when you click on Dry Run?
4. Can you provide details of your setup? (see forum rules)

Forum Rules

Welcome to the TrueNAS Community! Updated 11/28/2023 First and foremost I want to thank you for joining the TrueNAS community! My name is JoshDW19 and I'm the Community Manager here at TrueNAS. The TrueNAS Community is a friendly place to discuss open source software, technology, and products...

www.truenas.com

 

[Midblep]

1. Who is your ISP and what is your up link bandwidth?
Caiway in The Netherlands. It's gigabit like reported in the post, both up and down.
2. Have you tried disabling Take Snapshot?
No, I did not think this would affect anything. I will try now and let you know.
3. What happens when you click on Dry Run?
Very fast, completes pretty much immediately with success.
4. Can you provide details of your setup? (see forum rules)
Sure:
- Intel(R) Core(TM) i3-7100 CPU @ 3.90GHz (doesnt ever utilize above 20%)
- 8GB DDR4 RAM
- 2x 500GB HDD in RAID1 and 1x 300GB (no redundancy, not used for anything important)
- No controllers or special network cards

This is just a home device, I imagine it would be able to do a higher uplink to Google if I can reach it locally with such high speed and transfer large quantities of data in short periods to it? I hope these details are sufficient.

 

[Midblep]

Disabling 'Take Snapshots' has not seemed to change anything about the speed of the Task. It still often hangs at 0B/s and progresses very slowly.

 

[elvisimprsntr]

RAID1

1. You mean RAIDZ1 and not RAID1 using an on-board RAID controller?
2. 8 GB is a bit light.
3. Make and model of mobo?
4. Make and model of drives?
5. Looks like you are using a Realtek NIC. Those are finicky at best. Make and model.

 

[Midblep]

1. You mean RAIDZ1 and not RAID1 using an on-board RAID controller?
2. 8 GB is a bit light.
3. Make and model of mobo?
4. Make and model of drives?
5. Looks like you are using a Realtek NIC. Those are finicky at best. Make and model.

1. It tells me MIRROR, I was not aware about a difference between RAID when using a controller or not. I indeed have no such thing, it's a secondary repurposed computer.
2. It seemed to handle it just fine considering I was uploading at gigabit speed to it. Would uploading be different on the memory? Or is that more of a general advise?
3. Gigabyte GA-H110M-S2H
4. Two drives in question are WD-WMASY3061838 and Z2APM81P
5. My motherboard says it has a Realtek RTL8111H as network controller inside it.

Thank you for your help.

 

[Midblep]

1. It tells me MIRROR, I was not aware about a difference between RAID when using a controller or not. I indeed have no such thing, it's a secondary repurposed computer. 2. It seemed to handle it just fine considering I was uploading at gigabit speed to it. Would uploading be different on the memory? Or is that more of a general advise? 3. Gigabyte GA-H110M-S2H 4. Two drives in question are WD-WMASY3061838 and Z2APM81P 5. My motherboard says it has a Realtek RTL8111H as network controller inside it. Thank you for your help.

More specifically for the drives:
- WD 500AAKS
- Seagate ST500DM002
I inspected them physically now since I realized TrueNAS showed me the serial number only.

 

[nate.hancock]

I am experiencing the same issues on TrueNAS-13.0-U1.1.

I found that when the file size are small, the transfer occur quickly... but as soon as I hit a file larger then 150MB in size, my transfer speeds drop way low.
I found that GCP has some settings to optimize large file uploads by breaking up the large files into smaller chunks and processing them in parallel. Its called parallel_composite_upload. It seems that TrueNAS 12 supported this feature in the past on the sync tasks but now its not listed as an option in the GUI in Truenas 13.

@Midblep did you ever discover the cause of the slow speeds you were seeing?

 

[HermanoGrande]

I experienced the same problem.

For me the solution was to specify the "Transfers" and "Bandwith Limit" options.
Transfers is set to "1000"
Bandwith Limit is set to "500Mb" (which is half the speed of my internet connection)

TrueNas now reports transfer speeds up to ~50MiB/s, which is what to expect with a 500Mb/s Bandwith Limit

 

[Midblep]

Upon first check this seems to work, I'll hit it with a big load in a few hours and report back.
But that is crazy to think, and was perhaps a super obvious thing I should've checked way earlier.
Thanks so much for reporting your solution here!

 

[Midblep]

I experienced the same problem.

For me the solution was to specify the "Transfers" and "Bandwith Limit" options.
Transfers is set to "1000"
Bandwith Limit is set to "500Mb" (which is half the speed of my internet connection)

TrueNas now reports transfer speeds up to ~50MiB/s, which is what to expect with a 500Mb/s Bandwith Limit

I can now confirm this works under sustained load too, what an amazing fix! Thank you so much again, this solved a lot for me.

 

[nate.hancock]

I experienced the same problem.

For me the solution was to specify the "Transfers" and "Bandwith Limit" options.
Transfers is set to "1000"
Bandwith Limit is set to "500Mb" (which is half the speed of my internet connection)

TrueNas now reports transfer speeds up to ~50MiB/s, which is what to expect with a 500Mb/s Bandwith Limit

I also tested this by setting my transfer to "1000" and bandwidth limit to 1000MB (gigabit fiber ISP). My transfer speeds are hitting 75Mbytes/s on the cloudsync tasks that used to drop way down to under 1Mbyte/s before. Thanks @HermanoGrande!

 

[CheeryFlame]

I experienced the same problem.

For me the solution was to specify the "Transfers" and "Bandwith Limit" options.
Transfers is set to "1000"
Bandwith Limit is set to "500Mb" (which is half the speed of my internet connection)

TrueNas now reports transfer speeds up to ~50MiB/s, which is what to expect with a 500Mb/s Bandwith Limit

I know you only have 1 forum post but it's the most valuable forum post I've read on the forums, you saved my life and probably others too as well, thank you!