Two things that annoy me

[Ericloewe]

Huh, if you need proof that the forum structure needs work is that I missed this airing of grievances. I have so much to air!

Like the idiots who decided to use LTO tapes for backups in 2007 (not idiots, just weirdos), then decided to write a new Table of Contents file after each real file they'd write to tape (I guess...?), then decided that a specific ustar variant supported by GNU tar but not supported by BSD tar was what they'd use (Oh, these Linux folk...) and finally, the pièce de résistance:

They wrote each file in an individual tar archive with zero compression.

Yes, they had multiple files to write at once. From two to like 10 per satellite image, depending on the satellite. One big image plus tiny metadata images, but oops, there go another 128 kB of the tape for that 10 kB XML file. Plus another 128 kB for a new copy of the ToC. Plus EOF markers. That's 25x write amplification! And extra tape wear! And worse manageability!

And getting the damned drives to even work... They'd been sitting for 10 years and the heads were stuck like they were glued in place, so there I sat for hours carefully lubricating the head bearings (six of them times two drives!), boring local high schoolers on a work experience thing with Abe Simpsonesque stories about the distant year 1999 at the same time they make feel old ("You guys know about VHS?" "Oh yeah, my grandfather has one!" - emphasis mine). I get one drive done, we stick it back into the monster tape library (HP MSL6000 series double decker with two LTO-3 tape drives installed and room for two more, parallel SCSI version - not the fancy Fibre Channel stuff - for those following along at home), it feeds the tape and makes noise... but the drive errors out again. I repeat the process for the second drive, but break a little ferrite stick attached to the head, which goes into a coil and must be used to read the head's position on its vertical axis.
Somewhat defeated, a few days later, we went back and set up the drive on the bench, outside of the tape library. The year being 2022, we don't have an assortment of parallel SCSI cables with different connectors lying around to directly wire up the trusty Broadcom/LSI (I guess Broadcom/LSI still write the Linux driver for these things?) Fusion-MPT Ultra320 SCSI HBA to the drive's internal-style connector, so we precariously balance the drive's sled (it's just a standard internal full-height 5.25" drive inside a library-specific sled) so that we can attach external power via the AMP Mate-N-Lok power connector (don't call them Molex, folks) while using the sled to adapt the drive to the SCSI cable we had on-hand. The expectation was that we'd see what was going wrong with the (not-broken-by-Eric-the-klutz) drive.
Well the drive worked flawlessly. I suspect the tape library's sole functional PSU is less functional than thought and that was keeping the drives from working - and probably why it was abandoned 10 years ago, until we actually needed to read the old tapes. The drive heads would then have seized up simply by virtue of sitting around for a decade. Sure, the room is air conditioned, but salty air in the middle of the ocean still has nasty effect in such a room.

So now I get to deal with the work of someone who apparently never learned in school (Or at work. Or in their free time. Or through a shocked look on their co-workers' faces. Or in any of a million ways.) that tar is best used to archive multiple files in one archive.

What does this have to do with TrueNAS? I have to complain about TrueNAS? Well, I guess TrueNAS could include GNU tar in addition to BSD tar, since it has more features.

 

[Ericloewe]

Also, I always viewed tape as this mystical thing nobody could interact with. Sure, you need an application that speaks SCSI tape commands to do fancy things like "rewind without pressing the eject button" or "seek to the end of the tape" or "count the number of files on the tape without reading it all". But if you're just dumping a ton of data in large tar archives, just keep writing to the tape, it's just a block device that rewinds after closing if you use the wrong pseudo-file. With auto-rewind: /dev/saN (FreeBSD) or /dev/stN (Linux). Without auto-rewind, so you can actually read multiple files: /dev/nsaN (FreeBSD) or /dev/nstN (Linux). Use the mt application to do the above-mentioned fancy things or even fancier things.

 

[danb35]

About 15 years ago, before I started using what was at the time FreeNAS, and is now NAS4Free XigmaNAS, my home file server was Mitel SME Server, which I ran on a used Dell server with both hardware RAID and a tape drive--it's been long enough ago that I don't remember what kind of tape. But the machine had it, and I was able to back up to it (and even restore from it), and did so daily. I don't really like that we don't have a viable backup medium any more, other than just more spinners.

 

[jgreco]

The year being 2022, we don't have an assortment of parallel SCSI cables with different connectors lying around to directly wire up the trusty Broadcom/LSI

(puzzled look) Why don't you?

 

[Ericloewe]

Hey, tape still works, if you want to put up with it and pay more per TB in most cases.

 

[Ericloewe]

(puzzled look) Why don't you?

Because the facility was originally built circa 2007, when parallel SCSI was already a weird choice for a tape library attached to a server that had SAS bays on the front panel. I'd never actually touched a parallel SCSI device before this whole experience, so my personal collection doesn't extend to the many weird and wonderful types of SCSI cables and connectors. Although NVMe brings much of the same experience back, with U.2; U.3; SFF-8643; SFF-8087 because I'm sure someone, somehwere has tried that; Oculink; the two newer generations of stuff LSI has been pushing only to replace them again in like a year; ...

 

[Arwen]

...
I don't really like that we don't have a viable backup medium any more, other than just more spinners.

Well, for static items, we could use M-Discs, (the 25GB ones). Now all we need is a optical disc library or auto-loader / writer. Give it a pile of blanks, some software and your source data, then off it goes.


I've got a 25GB M-Disc blank that has been sitting next to the CD/DVD/Blu-Ray/M-Disc writer for probably a year. I have been meaning to create a zVol with a partial backup of my home computers, (but all the important things), as a ZFS pool. And since that backup is likely to use less than 12GB, I was thinking about enabling "copies=2". Considered "copies=3", but had to rule it out because I wanted to encrypt the entire M-Disc pool. Then burn that zVol to the 25GB M-Disc blank.

The advantage of using ZFS on the M-Disc is that I would be able to "scrub" the pool, as well as any read would let me know if a checksum fails. Normal CD/DVD/Blurays with other file systems might tell me if it was unable to read a block on a file access. But, ZFS can "scrub" the entire pool and let you know if their were any problems without having to go file by file.

 

[winnielinnie]

And since that backup is likely to use less than 12GB, I was thinking about enabling "copies=2". Considered "copies=3", but had to rule it out because I wanted to encrypt the entire M-Disc pool. Then burn that zVol to the 25GB M-Disc blank.

You could also use dvdisaster to "augment" an ISO file before burning it to the disc. It will create a Reed-Solomon based ECC and "bake it into" the ISO itself. How large the ECC data will be depends on the "remaining space" of a standard disc size. So if the ISO is 20 GB, then using the option "multithreaded RS03" will create a new augmented ISO file that is 25GB in size (20GB + 5GB ECC). It is this ISO your burn to the disc.


The neat thing is that it can recover the original ISO no matter where the damage is on the disc.


I gave this a try on a 100GB BDRXL disc. After creating 90GB worth of a data .iso file, I then used dvdisaster to "augment" this file which grew to 100GB in size (+10 GB of ECC). I burned this new ISO and then later verified the disc. No corruption, 100% verified. Then I took a straight-edge knife, scratching this disc in random places. Lo' and behold, dvdisaster showed a multitude of errors scattered throughout the media! I told dvdisaster to extract the ISO from the BDRXL disc. When I scanned the ISO on my computer, dvdisaster showed me the corruption, but told me "recovery is possible". I told it to try to recover the original ISO. It worked!

* There is a minimum size for the ECC (I think around 12%? I might have used more like "88GB"). If you try to augment an ISO that will have less than 20% ECC, the program will warn you, but allow you to continue. To be safe and not waste time, calculate for 20% ECC, which means for a 100GB disc the original ISO should not exceed 80GB.) Don't forgot the common mistakes we always make when different software uses different metrics (e.g, "GB" vs "GiB).


In my opinion the pros:

• It doesn't just work "in theory"; I confirmed it actually works in practice
• Doesn't just let you know about corruption; also can repair it
• Standard data ISOs as they are meant to be burned to disc (no need to figure out how to make a "read-only" version of ZFS)
• "Augments" the ISO file before you burn it, so that the ECC data is embedded into the ISO (and burned disc)
• Allows to you opt for a separate ECC file, if you don't want to augment the ISO (you can save this .ecc file somewhere safe)
• It works with CD-Rs, DVD-Rs, Blu-ray discs, and M-Discs
• Has a GUI (GTK) version and command-line version

Some caveats:

• The recovery process can take a long time
• Requires a working computer with enough free space to store the ISOs during the augmenting / recovery processes
• Nothing is done on the disc itself, only the ISO (before burning and after extracting; similar to how we use dd or ddrescue)
• The augmenting process happens to the originally created ISO "in place" (I recommend making a temporary copy of the original ISO)

 

[Davvo]

@winnielinnie that would be at least 10 disks per TB: 10TB would be 100 disks. Loooot of work if you have to restore withba backup.

 

[winnielinnie]

@winnielinnie that would be at least 10 disks per TB: 10TB would be 100 disks. Loooot of work if you have to restore withba backup.

Agreed, but that's why we reserve it for the "important" stuff. Irreplaceable photos, videos, documents, and other nostalgic or sentimental data. (This can easily fit on a 25, 50, or 100 GB M-Disc.) I'm not using it to make long-term reliable backups of movies, TV shows, or Linux distros.

As @Arwen implied:

I've got a 25GB M-Disc blank that has been sitting next to the CD/DVD/Blu-Ray/M-Disc writer for probably a year. I have been meaning to create a zVol with a partial backup of my home computers, (but all the important things), as a ZFS pool.

EDIT: As a "real world" example, I burned 25GB and 50GB M-Discs (with ECC) for a neighbor to permanently store all of their home videos from their Hi8 and VHS tapes.

 

[NugentS]

I'd love to have a backup that wasn't HDD's
Of course I suspect I wouldn't love to pay for it.

 

[Arwen]

@winnielinnie - Great idea.

Since my initial backup will be much less than 12GBs, I will still see if I can use ZFS & "copies=2". ZFS gets me the ability to compress & encrypt the files without hassle. And I can have separate datasets for different home clients. Plus, if created right, I get both ZFS error detection and correction, as well as the Reed-Solomon ECC

I looked at optical disc stack burners, but they all seem to come with their own optical drive. And not enough easy information to clearly determine if they supported M-Disc burning. If I live long enough, perhaps I will investigate one of those.

 

[winnielinnie]

I will still see if I can use ZFS & "copies=2". ZFS gets me the ability to compress & encrypt the files without hassle. And I can have separate datasets for different home clients.

All well and good, but at the end of the day, will it work?

I'm not saying it won't work, nor is it a non-viable option. (I actually think it's feasible!) But an actual test of this concept (which will require sacrificing a disc or two) will carry more weight than "theoretically" it works.

I might test this myself on less expensive media (since each M-Disc is more expensive than an equivalent Bluray disc, which I want to reserve only for legit archives.)

Whether you try this before me, only if the very final step succeeds, will I consider it a legitimate long-term (future generations) archive method.

1. Create an encrypted dataset with the options "copies=2" and compression set to ZSTD (ZSTD-19 might be overkill and pointless, especially for multimedia, so ZSTD-3 seems like the ideal choice).
2. Fill this dataset with files.
3. Create a snapshot and then "send" this snapshot to a file, such as mydataset20230115.zfs? (Never tried this.)
4. Burn this file to an M-Disc (or BD-R).
5. Verify the integrity of the disc and data (most software has this option after burning)
6. (Optional) From the disc, copy the file (mydataset20230115.zfs) to a local drive.
7. Use "zfs" to "recv" this file (mydataset20230115.zfs) into a ZFS pool. (Never tried this. Can you actually "recv" a file that was previously created from a "send" of a snapshot?)
8. Unlock the freshly received dataset and access the files within.
9. Repeat steps 6 through 8, except this time before step 6, make some small scratches with a knife in random areas of the disc.

Can you still "recv" this file as a dataset/snapshot?

Can you still unlock it?

Can you still access all your files?

What does a "scrub" reveal?

Will ZFS automatically "repair" the corrupted files based on "copies=2"?

What if damage (from scratching, or aging, or normal wear) happens to corrupt both "copies" of a record? (Think about the circular "shape" of a disc. It's very possible for two different areas of a filesystem to be damaged from a single scratch, and this could in fact mean both "copies" of a stored ZFS record.)

What if damage (from scratching, or aging, or normal wear) happens to corrupt the ZFS metadata itself? Without redundancy (normally provided from a vdev), you may in fact not be able to "recv" this dataset in the first place. I would assume the inner-most rings of a disc is where much of this metadata lives?

 

[winnielinnie]

I looked at optical disc stack burners, but they all seem to come with their own optical drive. And not enough easy information to clearly determine if they supported M-Disc burning

If they support writing to BD-R, BD-R DL and BDXL, then it means their lasers can burn 25GB, 50GB, and 100GB M-Discs as well.

 

[Arwen]

My thought for creating a M-Disc was;

• Create a temporary zVol without compression
• Make a single partition in zVol
• Create a compressed, encrypted pool inside the zVol
• Create child datasets for my various home clients
• Backup the home clients to the temporary backup pool and specific datasets
• Export backup pool
• Burn it
• Destroy temporary zVol

This was intended to allow direct file access on the optical disc, (though reader OS needs to support ZFS). Recovering a single file would be a simple copy. Not the complex method you wrote. Of course, if I add in encryption in addition to trying to import the pool as R/O, that does add 2 additional potential gotchas.