Backup config file every night automatically!

[ahnoroot]

Anyone know where the new DB is stored in Corral?

I second this.. anyone got a location for the new DB in Corral?

 

[ian351c]

I don't think the database files exist in the "backup" format on Corral. It looks like a config backup is a mongodb dump file. I was able to get a backup from a bash shell command line thusly:

Code:

cli -e 'system config download path="/tmp/test2.db"'

Now to test if it works in the Calendar...

ETA: I created a little shell script that looks like this. Does the trick nicely.

Code:

#!/bin/sh
/usr/local/bin/cli -e 'system config download path="/blahblah/temp.db"'
/bin/mv /blahblah/temp.db /blahblah/`date +%Y%m%d`.db

I could probably find a way to do that in one line, but it works...

 

[ezra]

Any update for 11.1 on this matter? Doesn't work running it on: FreeNAS-11.1-RELEASE

Code:

[root@freenas ~]# /usr/local/bin/cli -e 'system config download path="/tmp/temp.db"'												
bash: /usr/local/bin/cli: No such file or directory																				 
[root@freenas ~]# cli -e 'system config download path="/tmp/temp.db"'															   
bash: cli: command not found																										
[root@freenas ~]#

Thanks, and merry Christmas

 

[Glorious1]

I think that might be the Corral version. Try this (change the destination path to suit of course):

Code:

cp /data/freenas-v1.db /mnt/Ark/Jim/configbak/`date +%Y%m%d`_`cut -d' ' -f1 /etc/version|cut -d'-' -f2`.db

 

[ezra]

I think that might be the Corral version. Try this (change the destination path to suit of course):

Code:

cp /data/freenas-v1.db /mnt/Ark/Jim/configbak/`date +%Y%m%d`_`cut -d' ' -f1 /etc/version|cut -d'-' -f2`.db

Thanks, this does seem to work, although (in a test VM) when i try to restore the config to test it (i just entered some email server details) it gives: something went wrong in the WEBUI, i can't find any error messages...

Any idea?

 

[Wolfeman0101]

Just tried Sonarr and it seems like the sqlite3 issue is still there because Sonarr won't start.

 

[ezra]

I've tried these backup scripts and they work on 11.1
https://github.com/Spearfoot/FreeNAS-scripts

 

[diedrichg]

?Any idea how to fix:

Code:

bash: -c: line 0: syntax error near unexpected token `('
bash: -c: line 0: `rm /mnt/dvgmar/FreeNAS_Config_Backups/2017-10-05-FreeNAS-11.0-U4 (54848d13b).db'

Here is the rm code

Code:

# remove files older than 90 days ending in ".db"							  
$(find "$DEST_DIR" -type f -mtime +90d -name "*.db" -exec bash -c			  
'rm {}' \; )

 

[JoeB]

I created a script with a cp command in it, but the created file when run via cron has a question mark appended to the filename.

I cant work out why?
Script:

Code:

#!/bin/sh
cp /data/freenas-v1.db "/mnt/vol1/backup/FreeNAS/FreeNAS_Config/aaaa.db"

Cron command:

/mnt/vol1/backup/FreeNAS/FreeNAS_Config/backup_freenas_config.sh


Output:

Code:

root@JOE-FREENAS:/mnt/vol1/backup/FreeNAS/FreeNAS_Config # ls -l
total 290
-rw-r-----  1 root   admin  406528 Jan 11 17:40 20180111_174033_FreeNAS-11.1-RELEASE (dc7d195f4).db
-rw-r-----  1 root   admin  406528 Jan 11 20:36 aaaa.db?
-rwxrwxr-x  1 admin  admin	  88 Jan 11 20:35 backup_freenas_config.sh

Is this a FreeNAS bug? or a coding error?

 

[JoeB]

Anyone?

 

[fracai]

Are you sure there isn't a character there in the file? Maybe something hidden? I'd suggest posting the output of

Code:

xxd /mnt/vol1/backup/FreeNAS/FreeNAS_Config/backup_freenas_config.sh

but xxd isn't available in FreeNAS.
How about

Code:

base64 /mnt/vol1/backup/FreeNAS/FreeNAS_Config/backup_freenas_config.sh

Then we can see exactly what's there in the file.

 

[iudico]

It's important to note that you should never interact with a live database externally without taking a lock on it or you'll end up with a corrupted backup.[1] It's easy to achieve this by using the SQLite shell from the CLI instead of a call to cp, like so:

/usr/local/bin/sqlite3 /data/freenas-v1.db ".backup main '/path/to/backups/config_backup.db'"

The line above works with command substitution and cron. Note that main refers to the DB you'd like to back up and is optional if there is only one.

The .backup command uses the SQLite Backup API[2] ensuring proper locking is provided and that the backup is performed incrementally via a shared read lock so that other processes are not made to wait for the operation to complete. You can verify this for yourself by examining the shell source code.[3]

I've coupled this with taltman's solution for all the reasons they've outlined because it makes absolute sense to use the mechanisms provided and the effort to do so is minimal, both in terms of user effort and system resources used.

 

[ian351c]

Also note that there is a simpler way (for me at least) to perform a safe backup of the database. Use the freenas-debug command to dump the config database.

Code:

root@freenas:~ # freenas-debug -B > name_of_backup_file

In addition, it should be possible to backup the config with either the Websocket (v1.0) API or REST (v2.0) API. Unfortunately, the Websocket API doesn't seem to support config backup (just factory reset) and the REST API throws a 500 when accessing the config backup URL. So neither of those appear ready for prime time.

ETA: It appears that freenas-debug is a shell script which calls other shell scripts which (when using the dump database option) basically does what @iudico has documented in his command. Here is what my backup script looks like (I've set it up as a Cron Job in the FreeNAS GUI):

Code:

#!/bin/bash

{ #begin anonymous function
  /usr/local/bin/freenas-debug -B
} > /path_to_backup_folder/`date +%Y%m%d`.db

I'm not much of a coder, so I ended up googling my way to the "curly braces" solution for getting the output redirected the way I wanted it. Something about calling shell scripts within shell scripts within cron makes the redirection really hard to figure out.

 

[iudico]

Also note that there is a simpler way (for me at least) to perform a safe backup of the database. Use the freenas-debug command to dump the config database.

Code:

root@freenas:~ # freenas-debug -B > name_of_backup_file

Nice. Thanks for this. I didn't know it existed. I traced the dump command back and it does indeed call through to the sqlite command shell so I assume it uses the locking mechanisms provided, although I couldn't find any documentation to that effect.

In addition, it should be possible to backup the config with either the Websocket (v1.0) API or REST (v2.0) API. Unfortunately, the Websocket API doesn't seem to support config backup (just factory reset) and the REST API throws a 500 when accessing the config backup URL. So neither of those appear ready for prime time.

You can use the API to perform a config backup. See config.save in the Websocket API. However, I would currently recommend against it because it uses the naive file copy method that I advised against. I've opened an issue but it's being considered low priority so I'll try to get a PR in at some point to fix it as I'd like to make use of it.

 

[ian351c]

I saw the config.save method in the "native" docs (http://my_freenas_box.local/api/docs/) but it's not reflected in the API documentation at http://api.freenas.org and I wasn't able to find an endpoint by going to http://my_freenas_box.local/api/v1.0/ and reading the XML returned. So I assumed that the local docs were generic (there's quite a bit of generic docs from the various packages used for the Websocket and REST APIs installed on our FreeNAS boxes).

Were you able to call the config.save API method? I'm curious as to how, even if we shouldn't use it for backups.

 

[iudico]

I saw the config.save method in the "native" docs (http://my_freenas_box.local/api/docs/) but it's not reflected in the API documentation at http://api.freenas.org and I wasn't able to find an endpoint by going to http://my_freenas_box.local/api/v1.0/ and reading the XML returned. So I assumed that the local docs were generic (there's quite a bit of generic docs from the various packages used for the Websocket and REST APIs installed on our FreeNAS boxes).

Were you able to call the config.save API method? I'm curious as to how, even if we shouldn't use it for backups.

The RESTful API has v1.0 and v2.0, while the Websocket API would appear to mirror v2.0; the docs for the former reside at api.freenas.org while those for the latter reside at localhost/api/docs, and the (RESTful) data for each can be found at localhost/api/ with the appropriate v1.0 or v2.0 path appended. According to both the RESTful and Websocket API docs, v2.0 has the config.save endpoint and it's reflected in the returned data also, so it should work, but I've not yet tested it with a websocket client.

As to the how, I've looked in to that and the python installation included with FreeNAS includes ws4py, so it should be a simple matter to access the API using that.

Interesting what you say about the generic docs. I'd appreciate some clarification from the mods and/or devs on that point as well as the correctness of my statements above.

 

[Ryan Allen]

Hello all! I've been messing around with this for some time now and Finally got it to work..
NOW.. i would like to see if i can get the config file to auto backup WITH the secret seed.
Anyone know what I need to add to the script to make that magic work?

PS.. thanks guys!

 

[iudico]

Hello all! I've been messing around with this for some time now and Finally got it to work..
NOW.. i would like to see if i can get the config file to auto backup WITH the secret seed.
Anyone know what I need to add to the script to make that magic work?

PS.. thanks guys!

You can audit the code to see what the API is doing in the background and simply do the same thing in your script. There are no hidden or privileged system calls being used so it should be fairly simple.

 

[InGenetic]

I've created a cron job that will automatically backup your config every night(or whenever you want).

Here's how I did it: (the name of my zpool is tank, yours may vary so you will need to adjust the following accordingly)

I created a file on my zpool at /mnt/tank/bkpconfig.sh. It has one line:

cp /data/freenas-v1.db /mnt/tank/`date +%Y%m%d`.db

Edit:
-----------------------------------------------
If you want the config file to append the version of FreeNAS you are using, you can use this command:

cp /data/freenas-v1.db /mnt/tank/.scripts/ConfigBackups/`date +%Y%m%d`_`cat /etc/version | cut -d'-' -f2`_`cat /etc/version | cut -d'-' -f4`.db

You'll get filenames that are "20150125_9.3_201501241715" where the last characters tell you the build date/time you were using.
-----------------------------------------------

Another edit for 9.3.0 (but not 9.3.1 users) users (see https://bugs.freenas.org/issues/9091):

If you want to do these commands as tasks that run from cron directly, you'll want to use the following commands, correcting for your locations as appropriate:

cp /data/freenas-v1.db "/mnt/tank/`date \+%Y\%m\%d`.db"

- or -

cp /data/freenas-v1.db /mnt/tank/`date \+%Y\%m\%d`_`cat /etc/version | cut -d'-' -f2`_`cat /etc/version | cut -d'-' -f4`.db

-----------------------------------------------

Note that the ` is the top left key on the keyboard, not the typical apostrophe symbol. You can change the location of the db to anything you want, but you should keep it on your zpool or another location that is NOT on your USB key. Also you may not want to put the backup config anywhere that is shared. You wouldn't want anyone to have access to the database or delete them on accident.

Then log into the FreeNAS UI.

Expand "System", then expand Cron Jobs and then click "Add Cron Job".

For User set it to root.

For command use sh /mnt/tank/bkpconfig.sh or whatever location your bkpconfig.sh file is located at.

For short description enter something that makes sense to you. I put "Daily Backup of Config".

I chose to do the backup at 23:55 every day of the week. I figured that would allow me to backup the file as an "end of the day" routine while also doing the backup at an odd time so that other nightly tasks won't interfere.

For minute I chose the "Each selected minute" tab and chose 55.

For hour I chose the "Each selected hour" tab and then chose 23.

For Day of month I did "Every N day of the month" tab and set it to 1.

For Month I checked every month(default)

For Day of week I checked every day(default)

I left "Redirect Stdout" checked, "Redirect Stderr" unchecked and "Enabled" checked(all defaults).

Click "OK" and you should see the job be created.

If you have emails for root setup and your cron job fails you will get an email of the error.

My config file is only 207kb. I'd expect that regardless of your configuration your file isn't going to be big enough to worry about the lost space storing the config files. Plenty of people have been posting in the forums lately that didn't have a backup at all or had an old backup. With this you will have many copies of the config. This may be handy if the database gets corrupted since you have a backup history you can use.

Keep in mind that while the config file is safe on your zpool or elsewhere, you will need to copy the database file to your desktop to upload the config file. I just auto-imported my zpool and created a quick FTP server using anonymous access and was able to do a recovery in about 5 minutes. If you are really motivated you could create an FTP script in windows/linux/whatever that would auto-download your config files every time you logged into windows.

hi, may i ask about this :

cp /data/freenas-v1.db /mnt/tank/`date \+%Y\%m\%d`_`cat /etc/version | cut -d'-' -f2`_`cat /etc/version | cut -d'-' -f4`.db

is this mean it will save config with "Export Password Secret Seed" if we doing save config manual ?

please advice.

Regards,

 

[Glorious1]

is this mean it will save config with "Export Password Secret Seed" if we doing save config manual ?

No, if you save the configuration with such a command like this,
it does not also save the password secret seed.