Backblaze B2 Cloud Sync bucket way larger than dataset

[Liquid3564]

Hello all,

Recently I have started using TrueNAS for the first time. My TrueNAS server has been running wonderfully for about a month now. I have decided to make offsite backups to Backblaze B2.
A little background information.

I have 2 servers. A general Linux machine on which I host all my services (mostly containers) and my TrueNAS server.
- Linux machine:
Rocky Linux 9.2
Ryzen 3700x
ssd for boot and ssd for container volumes and ssd for game servers

- TrueNAS machine:
TrueNAS Core: TrueNAS-13.0-U5.1
Xeon e5-2683 v4
128gb ram
4x seagate ironwolf 4tb RaidZ1*
1x ssd for jails

*Yes I know that this isn't an optimal setup. I am waiting for more mounting hardware for the case. Then I'll get 2 more drives and make it a RaidZ2 with 6 drives. That's why I also need the backup to be complete so I can restore the data from the cloud backup.

Both connected on a 1Gb network.
The pool and dataset is as follows:

The docker-volumes and pterodactyl-games datasets are copies of the folders on my linux machine. On the linux machine they are on seperate ssd's. I use Syncthing to synchronize the folders on the linux machine and the TrueNAS server.

I have 5 cloud sync tasks running. Three of which work wonderful as expected:

The two tasks that are giving me some issues are the docker-volumes and pterodactyl-games tasks.

Now after I finally had all of the data synced using Syncthing I started the cloud sync backup. It has been a couple days and as you can see the Backblaze folders are way larger in size (I also accounted for compression ratio):

The settings for the cloud sync task are (basically identical for the pterodactyl-games task):

Now I am wondering why the bucket is so large. I am guessing that it might be connected to Syncthing constantly syncing more and more files that then get uploaded to backblaze. This would be weird because I had Syncthing on pause for the first day of the cloud sync task and the bucket still already went to over 100gb for each task.

Does anybody know what I am doing wrong and is willing to maybe give me some tips?

Thanks in advance!

 

[Liquid3564]

There's an update. The cloud sync tasks have now finished with succes.

But as you can see the bucket is still way larger than the datasets.

Can anybody explain what's going on and how I can fix this.

Thank you.

 

[jgreco]

Welcome to the forums.

Sorry to hear you're having trouble. Please take a few moments to review the Forum Rules, conveniently linked at the top of every page in red, and pay particular attention to the section on how to formulate a useful problem report, especially including a detailed description of your pool. It seems likely that something is using space, but what you really need to do is go in and look to see what's using the space. It's also possible that there are snapshots, etc., that are consuming space.

 

[Liquid3564]

Welcome to the forums.

Sorry to hear you're having trouble. Please take a few moments to review the Forum Rules, conveniently linked at the top of every page in red, and pay particular attention to the section on how to formulate a useful problem report, especially including a detailed description of your pool. It seems likely that something is using space, but what you really need to do is go in and look to see what's using the space. It's also possible that there are snapshots, etc., that are consuming space.

Thank you for the reply jgreco. I am sorry for not complying with forum rules. I scanned through the rules and thought I added everything, I was wrong. I added my hardware to my signature.

Now for the dataset settings (both problem datasets have the same settings):

I have not set up any snapshot tasks, nor does the Cloud Sync task create a snapshot. Actually I don't have any snapshots currently.

Also doing a

Code:

ls -l [path]/docker-volumes

shows the exact same folders and files as on the Linux machine that I am syncing from (the same for the games folder).
Doing a

Code:

du -sh [path]/docker-volumes/*

does not show anything weird either (again same for the games folder).

Again, thank you for the reply!

 

[Liquid3564]

Oh and I am using Syncthing version 1.22.0_6 on release 13.1-RELEASE-p7 from the official iXsystems collection.

 

[metadaddy]

Hi @Liquid3564 - by default, Backblaze B2 buckets have versioning enabled. It looks like B2 is keeping previous versions of files that are being synced. To have B2 only keep the most recent version of each file, log in to the Backblaze web console, click 'Buckets' in the menu on the left to get to the list of buckets, then click 'Lifecycle Settings' for the bucket in question. Select 'Keep only the last version of the file', and click 'Update Bucket'.

Note - although it says that changes will take effect in 10 minutes, that's just the update to the bucket settings. The old files get deleted on a nightly job, so it will take up to 24 hours to see that bucket's size fall.

 

[Liquid3564]

Thanks for the reply @metadaddy
I haven't said it in my previous posts - I'm. sorry - but I have already set the life cycle setting to latest only when I created the bucket a couple of weeks ago.
Furthermore this shouldn't be the reason for my problem (I think) since it was a fresh sync. A new backup to an empty folder without any of the files changing on the TrueNAS dataset since Syncthing was paused. So there weren't any "older files" to be kept in the bucket.

Thanks

 

[BB-8]

Hello All,

I apologize for reviving an older thread, but I'm facing a similar issue and was hoping to find a solution. I have about 140 GB of data on TrueNAS, but the same data occupies around 410 GB on a Backblaze B2 bucket using SYNC transfer mode. I've also configured File Lifecycle to keep only the most recent version.

I have daily snapshots enabled on the TrueNAS instance. Could these snapshots contribute to the increased storage size, possibly being concealed within the Truenas dashboard's storage usage display?

Any insights or solutions would be greatly appreciated.

 

[NugentS]

May I suggest:
1. Create New Dataset with no snapshots
2. Copy known data into that dataset
3. Backup to B2 and compare. I suspect you will find the sizes the same (approx)

I am using B2 for a small backup of ( Windows says 1.86GB, Size on Disk of 3.53GB)
On backblaze its 3.9GB - but it does keep previous files for 14 days