Repurposing HDDs like tapes for offline storage - anyone have experience with this?

[Doogie]

I plan to upgrade a server containing forty 4 TB drives and replace these with larger drives. I'd like to repurpose the 4 TB drives for offline storage, using them essentially like tapes. I am looking for robust, easy to use (and hopefully <$500) software which I can use over a 1 Gb LAN to backup a >100 TB TrueNAS server to these externalized disks. This backup is for "last resort" scenarios if the primary and several secondary servers were to fail; I would be OK if I had some of the HDDs fail and lose partial data, but importantly, I could NOT have one or a few failed HDDs result in the loss of ALL data. I would envision using an eSATA or high speed USB dock, and plug in each bare drive as prompted. Ideally, I'd like the data stored on the HDDs to be in an industry standard format with directory structures retained in case the software failed and all I needed to do was mount the HDD and copy the data over to a new location (similar to unRAID).

Backup and recovery speeds are not hugely important. I'd prefer simple software instead of enterprise-grade level software with a cumbersome, complex feature set. All I want to do is run a program that incrementally copies the contents of my TrueNAS server to an external USB/eSATA drive and prompts me for the next drive when full, and similarly, restores the contents of the data set in similar fashion, skipping over any corrupted HDDs encountered to restore a partial data set. Linux is preferred over Windows, but reliability/ease of use is paramount.

Any suggestions are appreciated. THANKS much.

 

[ChrisRJ]

Back when I was using external USB hard disks for backup (roughly 2010), I had a Windows program called SyncBack (https://www.2brightsparks.com/syncback/syncback-hub.html). The free version was quite limited so I went for the lower commercial version at the time (later upgraded to PRO). I had tested various free alternatives, but none would work reliably with tens of thousands of files (a lot of source code). No idea how good this is today, but at the time I was happy with it.

 

[Arwen]

I have used "dar" in the past to create optical disc images for backup. It's somewhat like TAR in that it stores files in a container. But, in DAR's case, the containers are able to be automatically sliced up, like the old UU Split program. DAR includes options to run commands between creation of slices, so that the current disk can be "ejected" and the next disk inserted. (Which in my case was to "burn" the slice to optical media, and wait until I was ready for the next burn.)

There is also this:

Dar format is quite robust against corruption: Only the file where the corruption took place in the archive will not be possible to restore.

Don't know if DAR is available for FreeBSD.

 

[Constantin]

I had a Windows program called SyncBack.

I agree that it's a great program, very good in terms of filters, etc. Not sure it supports slicing of data data sets into 4TB chunks (IIRC, it was a file-level copier) nor whether it supports some sort of distributed parity system to help save corrupted files. Having copies in a OS-native format can be very helpful for restoration purposes, especially if the data fits somewhat neatly into 4TB chunks.

However, figuring out where something should be (i.e. what disk, etc.) is something I would give a lot of thought to. Some programs automate the task of indexing all that.

 

[no_connection]

Remember to periodically read the data so the HDD re-magnetize any parts that are getting weak. Especially if you are doing partial backups where the initial data might be untouched for a loooong time.

 

[cobrakiller58]

Remember to periodically read the data so the HDD re-magnetize any parts that are getting weak. Especially if you are doing partial backups where the initial data might be untouched for a loooong time.

I believe that requires rewriting simply reading the data only confirms it is intact at that moment.

 

[winnielinnie]

I believe that requires rewriting simply reading the data only confirms it is intact at that moment.

Two methods of doing such are to dd the drive to "itself", which sounds terribly scary if something were to go wrong (or you accidentally typo a mistake within the command), or to use badblocks in non-destructive read-write mode. This will take significantly longer than a straight dd pass, but it will also check to see if blocks can be read correctly; rather than simply refreshing the data.

Then you can label the date this was done, as well as the command used, and the results, as a reminder to yourself.

Obviously, an accessible copy of this data should live elsewhere, regardless.

Example command of dd:
dd if=/dev/sdx bs=128K status=progress of=/dev/sdx

Example command of non-destructive badblocks read-write pass:
badblocks -n -s -v -b 4096 -c 65536 /dev/sdx

Example of label printer stickers on the refreshed drive:

DATE: 2021-02-11

COMMAND: badblocks -n -s -v -b 4096 -c 65536

RESULTS: No errors

These can be very dangerous commands, so just be careful as to not accidentally erase everything.

As far as solid-state drives go, I'm not entirely sure if this is helpful. From some reading, simply prodiving power to them occassionally is all that is required.