Archive / delete old data automatically

[HansWege]

I'm total Linux newbie. I'm using TrueNAS CORE as my small NAS with my 1 IPCAM recordings on it that are saved via FTP. I've dedicated 300 GB of space for my camera recordings. These recordings are not critical, and only will only need to look at them if motion capture won't trigger and something happened. Camera generates around 60 GB of data per day (around 2.5 GB per hour).

I'm looking for a thing that would archive the oldest folder with best compression and delete the folder itself. If space is low (let's say <90%) delete the oldest archive.

Foder structure looks like this main\CameraFTP\Cam\2023\06\23 as in: main share \ CameraFTP \ camera name \ current year \ current month \ current day. Filenames inside the folder cameraname_00_20230623143452.mp4, as in cameraname _00_ year month day hh mm ss

This seems complicated to achieve for me, knowing almost nothing about linux, it took my half a day to setup TrueNAS, setup FTP and setup SMB share. Now I'm concerned what will happen when the partition fills up.

 

[HoneyBadger]

Hello @HansWege

This is something that's best handled by a NVR (Network Video Recording) software suite (eg: Zoneminder, Blue Iris) that will automatically delete your oldest recording. Video files also don't traditionally benefit in a significant manner from lossless compression once they've already been compressed using a lossy codec, so the "archival" step isn't likely to be beneficial.

What you're asking for can be done with shell scripts and scheduled to run periodically with a cron job; however, what you're asking for is an inherently destructive action (data deleting) so it's unlikely that anyone will offer a sample or explicit code snippet because they don't want to be liable for an incorrect or inappropriate deletion.

You can set up alerting within TrueNAS to send messages once your pool reaches a given utilization and manually remove files as well.

 

[winnielinnie]

Video files also don't traditionally benefit in a significant manner from lossless compression once they've already been compressed using a lossy codec, so the "archival" step isn't likely to be beneficial.

Somewhat related, even though it doesn't seem intuitive at first: leave LZ4 inline compression enabled on the dataset.

Even though the video streams will not compress, you still gain the benefit of "squishing the extraneous slack" at the end of a record. (Squishing it to essentially "nothing".) This will allow you to use 1 MiB recordsize (for a dataset mainly dealing with large video files), without the overhead at the end files that don't perfectly consume that last bit of a 1 MiB record.

ZFS also uses "early abort" for LZ4. (Hopefully this will be available with ZSTD in the near future.)

This doesn't address your "compress and archive" question. But often there's the misconception of outright disabling inline compression for datasets with multimedia files.

 

[pschatz100]

To the OP @HansWege,

I want to point out that TrueNAS is built on FreeBSD, not Linux. While there are many similarities between FreeBSD and Linux, there are differences as well... so be careful when searching for solutions on the internet. You need to be certain you are trying solutions that will be compatible with FreeBSD.

As you are not familiar with FreeBSD, I would suggest you follow @HoneyBadger advice to either install an application such as Zoneminder or set up some alerts within TrueNAS to remind you when to delete some files. While it is technically possible to automate the process with scripts, if you don't know what you are doing then you run the risk of causing significant damage to your system.

Good luck.