Hi folks,
this post is more like "I am thinking out loud" and wanting to hear your professional opinion and looking for some guidance.
This is what I am having and doing:
FreeNAS-9.10.2-U4, with one vdev of 8x4TB in RAID-Z2 (my "tank"). In addition, I have 2x4TB in mirror, called backup.
I also have configured a jail, and installed BORGBACKUP and B2 CLI.
As you can probably already tell, I am using BORG to backup my "tank" to "backup" for a local backup. Once the backup finishes, I sync the backup vdev to B2 Cloud Storage.
I find BORG dedup super efficient (3.77 TB become 1.35TB ...). This works quiet nicely, and BORG is very easy to use!
Now this is what I still want (and need) to do:
I also need an offsite backup, that is easily accessible to me, like taking a hard drive to my workplace, and having a hard drive in my NAS and rotating them weekly or so.
I have 2x2TB 2.5" hard drive in a removable enclose at my disposal. I could rsync the backup vdev to the 2TB HDD.
I was thinking of formatting the 2TB with ext2, but it generally not recommended. As well as UFS is not a good choice.
Than I though of using ZFS on single disc, however, it can create problems when corruption is occurring on the 2TB and I cannot import that single disc vdev anymore.
Now my final though is, to have a 3 disc mirror like 4TB:4TB;2TB. I know, that I will loose half of the capacity on the 4TB discs, but that is completely acceptable to me at this point, since all my data combined is ~1TB at the moment. It will take years for me to reach 2TB backup data.
What I would do, is remove the 2TB disc (disc_1) , (need suggestions from you how I can do this in a script), and resilve to the other 2TB disc (disc_2).
This is what I like about this idea:
- Resilver would be fast, since it would just copy data.
- I have the option to do scrubs and data corruption can be repaired (that would not be possible with a single disc vdev)
My biggest question at the moment is, how can I automate this in a script, aka, how can I automate the removal of a disk from a vdev and replace it with another disc.
These are the steps I would like to use for the backup procedure:
1: Backup data using BORG from tank to backup (note backup vdev would be 2TB large and have 3 discs in mirror)
2: Use B2 CLI to sync the backup vdev to cloud
3: run scrub (maybe only weekly)
4: remove (read umount) offsite disc_1 from vdev, so that I can physically remove the offsite disc_1 from the slot and insert the offsite disc_2.
5: Resilver offsite disc_2
I basically have automated step 1-3, but unsure on howto tackle the zfs vdev portion of this backup strategy. (essentially I would break a good vdev in order to be able to remove the disc ...)
One further note: I do not want to have to write commands in CLI before I can physically remove the disc to swap it with the second one. This has be automatic.
I can use the script to check if the disc has changed.
I will always know, what the status of my backup is (if it failed or completed, and if the offsite disc is still mounted) via status emails that I use in the script. So, before I remove the offsite disc from the NAS to replace it, I can check my emails and know, if the disc is still mounted or not.
This got a bit long and I hope its clear. If you need clarification etc. please ask.
Please do not suggest other software to backup. I have tried using snapshots for backup, and they are nice and work good, for reasons that would make this post to long and would be "off-topic", I decided not to use snapshots for backup (but I do use snapshots regularly and love them till death do us part, they have saved my butt in a big way more than once ...!).
I also have looked into using rsync to sync to a offsite location in town, but I do not have good options, so rotating disc is best for me right now.
Please have your comments, ideas and code coming ...
Thanks in advance, and cheers ...
Eddi
EDIT: Adjusted some numbers to be more accurate.
this post is more like "I am thinking out loud" and wanting to hear your professional opinion and looking for some guidance.
This is what I am having and doing:
FreeNAS-9.10.2-U4, with one vdev of 8x4TB in RAID-Z2 (my "tank"). In addition, I have 2x4TB in mirror, called backup.
I also have configured a jail, and installed BORGBACKUP and B2 CLI.
As you can probably already tell, I am using BORG to backup my "tank" to "backup" for a local backup. Once the backup finishes, I sync the backup vdev to B2 Cloud Storage.
I find BORG dedup super efficient (3.77 TB become 1.35TB ...). This works quiet nicely, and BORG is very easy to use!
Now this is what I still want (and need) to do:
I also need an offsite backup, that is easily accessible to me, like taking a hard drive to my workplace, and having a hard drive in my NAS and rotating them weekly or so.
I have 2x2TB 2.5" hard drive in a removable enclose at my disposal. I could rsync the backup vdev to the 2TB HDD.
I was thinking of formatting the 2TB with ext2, but it generally not recommended. As well as UFS is not a good choice.
Than I though of using ZFS on single disc, however, it can create problems when corruption is occurring on the 2TB and I cannot import that single disc vdev anymore.
Now my final though is, to have a 3 disc mirror like 4TB:4TB;2TB. I know, that I will loose half of the capacity on the 4TB discs, but that is completely acceptable to me at this point, since all my data combined is ~1TB at the moment. It will take years for me to reach 2TB backup data.
What I would do, is remove the 2TB disc (disc_1) , (need suggestions from you how I can do this in a script), and resilve to the other 2TB disc (disc_2).
This is what I like about this idea:
- Resilver would be fast, since it would just copy data.
- I have the option to do scrubs and data corruption can be repaired (that would not be possible with a single disc vdev)
My biggest question at the moment is, how can I automate this in a script, aka, how can I automate the removal of a disk from a vdev and replace it with another disc.
These are the steps I would like to use for the backup procedure:
1: Backup data using BORG from tank to backup (note backup vdev would be 2TB large and have 3 discs in mirror)
2: Use B2 CLI to sync the backup vdev to cloud
3: run scrub (maybe only weekly)
4: remove (read umount) offsite disc_1 from vdev, so that I can physically remove the offsite disc_1 from the slot and insert the offsite disc_2.
5: Resilver offsite disc_2
I basically have automated step 1-3, but unsure on howto tackle the zfs vdev portion of this backup strategy. (essentially I would break a good vdev in order to be able to remove the disc ...)
One further note: I do not want to have to write commands in CLI before I can physically remove the disc to swap it with the second one. This has be automatic.
I can use the script to check if the disc has changed.
I will always know, what the status of my backup is (if it failed or completed, and if the offsite disc is still mounted) via status emails that I use in the script. So, before I remove the offsite disc from the NAS to replace it, I can check my emails and know, if the disc is still mounted or not.
This got a bit long and I hope its clear. If you need clarification etc. please ask.
Please do not suggest other software to backup. I have tried using snapshots for backup, and they are nice and work good, for reasons that would make this post to long and would be "off-topic", I decided not to use snapshots for backup (but I do use snapshots regularly and love them till death do us part, they have saved my butt in a big way more than once ...!).
I also have looked into using rsync to sync to a offsite location in town, but I do not have good options, so rotating disc is best for me right now.
Please have your comments, ideas and code coming ...
Thanks in advance, and cheers ...
Eddi
EDIT: Adjusted some numbers to be more accurate.
Last edited:
