Large disk utilisation on Truenas 13.0.U2

[Sherzod]

Good afternon,

I have an interesting situation. I have Cloud Sync Task to S3 configured on Truenas, all backups are downloaded on weekends to S3 bandwidth of 2Gb/s. Disk utilisation is up to 80% and the backups are downloaded in one job. (Previously there were 6 such jobs and disk utilization reached 100%). Two 10Gb/s network cards are combined into Lcap. Average size of the files is 10-20 gb, I don't understand why there is such a load on the disk. I was expecting less load on disk. What could this be related to?

Secured two graphs: disk utilization, traffic on the interface.

Truenas version: TrueNAS-13.0-U2
Server model: Dell R540 - Silver 4214
RaidZ: RAIDZ2
Compression type: lz4
RAM: 64GB
Raid card model: DELL PERC H740P

I tested results of reading from disk with dd, I leave results below.

Code:

# dd if=random_read_write.fio of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 0.458905 secs (2339789333 bytes/sec)

 

[Sherzod]

Tried downloading a backup copy using scp at 1.5Gb/s giving a 30% load on the disk. the question is why Cloud Sync Task is giving a high load of almost 50% of the disk?

 

[ChrisRJ]

I have an interesting situation. I have Cloud Sync Task to S3 configured on Truenas, all backups are downloaded on weekends to S3 bandwidth of 2Gb/s.

Not sure it matters, but to me it is not clear in which direction the data flow. Are you downloading to TrueNAS or uploading from TrueNAS?

 

[morganL]

Your system is reading from ZFS and writing to S3 at about 200MB/s

ZFS is reading from disks and driving the disk activity - its doing a lot of small disk reads because you a a 10 drive Z2 configuration.

Each ZFS read requires 8 small disk reads.

 

[Sherzod]

Not sure it matters, but to me it is not clear in which direction the data flow. Are you downloading to TrueNAS or uploading from TrueNAS?

Тrunas uploads a backup to S3

 

[Sherzod]

Your system is reading from ZFS and writing to S3 at about 200MB/s

ZFS is reading from disks and driving the disk activity - its doing a lot of small disk reads because you a a 10 drive Z2 configuration.

Each ZFS read requires 8 small disk reads.

Thank you for your reply.
I am trying to understand.

I have tested with scp uploading data to another server with Truenas. Same backups. Scp gives 40-50% less load on disk.
The scp copy was done from the same dataset and the same backups.

Why does cloud sync give more load on disk?

 

[ChrisRJ]

Why does cloud sync give more load on disk?

Under the covers this uses rclone. Perhaps the documentation there has some pointers ...

 

[morganL]

Thank you for your reply.
I am trying to understand.

I have tested with scp uploading data to another server with Truenas. Same backups. Scp gives 40-50% less load on disk.
The scp copy was done from the same dataset and the same backups.

Why does cloud sync give more load on disk?

I'd guess its due to the size of the Reads... rclone might read in smaller chunks. Are these larger files?
It would be useful to check average disk I/O size for each mode of transfer.

 

[Sherzod]

I'd guess its due to the size of the Reads... rclone might read in smaller chunks. Are these larger files?
It would be useful to check average disk I/O size for each mode of transfer.

Good afternoon!
Thanks for the tips on rclone. Indeed it was the problem, I tweaked the parameters and tested, the results improved.

Now I have a new question: how can cloud sync be configured to run rclone with these parameters (upload_cutoff = 4Gi, chunk_size = 1Gi, upload_concurrency = 2) ?

I found such a way: https://www.xenlens.com/freenas-rclone-bwlimit-parameter/
but I don't really like it because it suggests changing the script, and as I understand, these changes will be erased after each TrueNAS update.

Can you suggest a more effective way to solve this problem?

 

[Sherzod]

Under the covers this uses rclone. Perhaps the documentation there has some pointers ...

Thanks for the tips on rclone