Full system backup including ix-applications in the event of boot drive failure

[dbrannon79]

Hello everyone,

I'm sure this has probably been discussed before, but I don't seem to be able to find the info I'm looking for. Originally I wanted to have a second truenas system that mirrored the same as my main system so that in the event my main server went down for whatever reason I could just use the second. This really didn't work out like I had wanted with some of the apps I have running like adguard home. Currently I have a second truenas server housing my backup pool that keeps all my data safely backed up on a schedule, but with my main server running apps that control aspects of my home like adguard which my home router is directed through has a LOT of DNS lists among other things and other apps running like Plex, I want a viable way to make a full backup of my main servers boot drive and the ix-applications (within the pool) so that it all can be restored without having to re-install each app one by one and setting each one up (taking hours on end) some of which I know I have forgotten how I set them up long ago!

I don't mind if this has to be done in multiple backups like one for the boot drive and another for the apps. just as long as once both are restored, everything just works.

I'm not sure if there is a way the boot drive can be backed up using truenas other than the configuration, maybe using clonezilla with a drive or usb and just keeping it in a cabinet for safe keeping. I just need some suggestions and help. maybe pointing me to an online or video guide would help.

both my main and backup servers are running TrueNAS-SCALE-22.12.2 which I have neglected to update worrying about things breaking!

Thanks in advance for the help.

 

[Jailer]

There is no need to clone the boot drive or perform any other type of backup on it. Keep a current copy of your config file and if your boot drive ever fails just replace it and install a fresh copy of TrueNAS on it. Once it's installed upload your saved config file and everything will be back up and running just as it was before. It's designed this way.

 

[dbrannon79]

Just an update, I created a created a second replication task solely for the ix-applications data set hoping this will safely backup the apps and allow me to restore them if ever I need to. on my first replication task I had errors in the beginning if I didn't exclude this data set.

Will this work in conjunction with keeping a copy of the config files on both servers?

 

[sfatula]

I'd be very careful with snapshots of your ix applications dataset. Keep an eye on the number of snapshots on your system, you may not like what is coming! Each pool snapshot can mean hundreds of them as there are numerous datasets within that pool, and you are doing every hour!? If you do insist on snapshoting them, I'd recommend heavyscript to keep the number in check and better automate the process. Otherwise, you'll find a lot of trouble coming.

 

[dbrannon79]

I'd be very careful with snapshots of your ix applications dataset. Keep an eye on the number of snapshots on your system, you may not like what is coming! Each pool snapshot can mean hundreds of them as there are numerous datasets within that pool, and you are doing every hour!? If you do insist on snapshoting them, I'd recommend heavyscript to keep the number in check and better automate the process. Otherwise, you'll find a lot of trouble coming.

I looked at what snapshots it has taken over time and see there are a total of 3,860 cataloged. I have both of the of the snapshot tasks only keeping them for 2 days and taking them every hour along with the replication tasks making a backup each hour. I most likely don't need the ix-applications to be backing up or doing snapshots that frequently though. Maybe for that task I should set it for once per day instead. I will keep watch on it for a couple more days and see if the snapshots double or if the system is actually deleting them over the 2 day time.

I am curious, what is this Heavyscript you mentioned.

I know I will need to upgrade my backup server with some extra space. it has a total of 7.8tb with 4ea 3tb drives in a raidz1. my main server is setup in z2 with 8 of the same size drives. Only two options to upgrade are to pull and replace each drive with larger (1u server only has 4 bays) or find 4u rack server that can house at least 8 or more drives.
at one time I did attempt using a desktop PC case but had numerous issues with the SAS drives and break-out cables dropping drives. using a server with a black plane solved that problem.

 

[sfatula]

How much space those will take are dependent on a variety of things. If you are using Truecharts and PVC storage, you will exhaust your space somewhat depending on how much the data changes. There are people with tens of thousands of snapshots and certain commands/screens stop responding then. With 10-15 apps, all using PVC, this can happen pretty quickly. Many threads in the forums about such users.

Heavyscript is used by many in these forums. Simple script to do many useful things. It can be found here:

GitHub - Heavybullets8/heavy_script: Bash script for managing Truenas SCALE applications, automatically update applications, backup applications datasets, open a shell for containers.

Bash script for managing Truenas SCALE applications, automatically update applications, backup applications datasets, open a shell for containers. - Heavybullets8/heavy_script

github.com

It does some cleanup tasks and various useful things along with backing up app pool. Sometimes, you will find (otherwise) your snapshots can't be deleted and then you can spend days and days of effort, maybe weeks, trying to clean them up on your own.

I have 938 snapshots, but I am not using Truecharts. I believe I keep about 7 snapshots, 1/day.

 

[dbrannon79]

Well I just looked today and see I am up to over 4k in snapshots and with using the search bar I can see 2500 or so are all of ix-application snapshots! I see what you mean. I just went through and deleted all of the ix-app snapshots up to the last hour they were made, then changed both the snapshot task and the replication task to just do ix-applications once per day and keep snapshots for 2 days.

deleting all of the ix-applications up to the last hour they were taken left me with only 54 ix-app snapshots. two days worth will be 108 + there are an additional 1488 of them for all the other datasets (2 days worth) so with that, I should only see a max of 1596 of them at any time I look since both tasks are only keeping them for 2 days. I will see how this goes moving forward.

on a side note: when I setup the replication task for just the ix-applications, I had wanted for it to back them up on my other server in a different dataset so that I could launch a couple of apps in the backup server. one of them I use is zerotier so I can access my data remotely when I have my laptop and traveling. but unfortunately when I was in the GUI on my backup server I attempted to create a new dataset it would not let me telling me something about the system being read only! not sure if this is due to something I set in the replication tasks or something odd going on with the backup server it's self.

I do have an alert with the boot drive on the backup server about it being degraded. it's an ssd so I am assuming that SSD is failing and might be in read only mode now. I've had this alert for several months and figured I would need to just replace the boot drive, install truenas and upload the backup config to get rid if this. Again not sure if this is the reason I could not create another dataset in the backup pool.

 

[dbrannon79]

This is the error I get, even though I select "off" for read only mode, it still will not let me past this to create a new dataset in the pool

 

[dbrannon79]

I might have found the issue on the backup server, my main dataset is set in read only mode. but I tried to change it to show off. it accepts the changes but when I go back in to edit the main data set it still shows in read only mode! not sure what's happening here. my backups are working find writing to this dataset or what I call the pool. (not sure the correct terminology here) I sure hope I don't need to re-do the entire backup of my main server over this.

the replication task is set to make the files or datasets that my main server pushes to the backup read only but I did not think it would hard lock everything down to the pool or top dataset as read only. not that I want to change any of the data in it, I just want to make it use another dataset to backup ix-applications so that if I want to install an app or two in the backup server I can. I'm sure I am asking too much for these machines to do.

a couple of apps I would like to run on the backup server would be the zerotier (access data and the GUI outside my network if needed) and also setup adguard home so that if my main server fails for some time, I can easily go into my network router and change the DNS server to use another instance of adguard home while I get or work on the main server. as it is if the main server gets powered off I will loose internet throughout the entire house due to how strict I have adguard setup and the router. I don't want any internet traffic going around the filters I have set in adguard. I have been told that if I have a backup or second DNS ip set in the router, if there is a resolve issue or adguard blocks something, the router can easily divert around it using the second DNS IP on the fly. if I have both servers running instances of adguard, I can then add it's IP to the second DNS setting in the router.

 

[sfatula]

I am not sure what this menas:

"I just want to make it use another dataset to backup ix-applications so that if I want to install an app or two in the backup server I can."

How is the backup connected to apps on backup server?

 

[dbrannon79]

it's not. on the backup server I don't have a pool set for apps mainly because the main server is using the same dataset /ix-applications to store the backup. the entire pool is essentially a mirror of what's on the main server.

what I wanted to do is on the backup server, create a dataset or child dataset in the pool named /ix-app_backup for the main server to push /ix-applications to. this would allow the backup server to use it's own /ix-applications dataset for it's own apps

the way it is now, if I go to apps while in the GUI of the backup server it prompts me to choose a pool to use, if I allow it to use the only pool it has, it will pull up the data that was pushed over from the main server since truenas has in it's setting to use /ix-applications for this. If I allow it, I feel like that will open up an entire new can of worms!

hope this makes better sense.

 

[dbrannon79]

while in the backup server GUI, when I go to datasets and try to edit the main dataset (top one listed or what I call the pool) and try to change it from read only = on to read only =off. I get this message in the logs.

Nov 22 14:43:34 TrueNAS-Backup middlewared[2547091]: special_small_blocks: property requires a special device in the pool

 

[dbrannon79]

Update: I tried something different, though not sure if this will cause problems. I did some searching and ran across a similar issue posted on a thread about the replication process setting the other system's dataset to read only. so here is what I did...

opened the shell and used the command "zfs set readonly=off Storage" (Storage is the name of the pool) this allowed me to create the child dataset "Storage/ix-app_backup" Next I went into my main servers GUI and changed to ix-applications replication task where it was pushing "Main/ix-aplications to "Storage/ix-applications" to "Main/ix-aplications to "Storage/ix-app_backup"

now I need to clear out the data that is in "Storage/ix-applications" and set apps to use the pool for apps. I hope this will work.

 

[sfatula]

I don't know why not. I don't use the Scale replication screens, found a lot of limitations so I use other software. Someone else would have you guide you on the Scale replication takss.