File sizes on FreeNAS/ZFS vs other FSs

[bajaking]

Quick, dumb question (feel free to scold me accordingly):

I know there is "real time data compression" on ZFS, and I know my FreeNAS 9 is using the default lz4 in all the default places. What does this mean for file sizes at rest?

For a ham-fisted luddite like me, I like to use du -sh as a sanity check when transferring files. If the source and destination are the same size, I feel ok to move on to the next transfer task. If they're not, I'm scared, and then go to writing & checking md5sums, which I'd normally only do for really mission-critical data.

So is there a rule of thumb for comparing file sizes on non-ZFS (ext3 or ntfs) with ZFS, like divide by a consistent compression ratio? Or do I always have to do checksums if I want to make sure my files transfer correctly and completely? (and will that even work with files that may be compressed at rest?).

 

[danb35]

There is no consistent compression ratio, as it depends entirely on the type of data stored. My datasets range from 1x to 4.93x compression.

 

[jgreco]

You can use the results of "ls -l" instead of "du -sh". du will report disk blocks used, which is after compression. The number in ls -l represents the reported file size.

You can also look into mtree which is capable of generating very nice filesystem verification trees, including the option to verify size, and/or various checksums.

Checksums naturally work on files that are "compressed at rest", because the act of reading the file to compute the checksum means they're not at rest.

 

[bajaking]

Thanks, folks. Perfectly answered. mtree looks awesome, a nice gift for those of us new to the BSD world.

 

[jgreco]

mtree is probably a little bit daunting because it has a bunch of abilities, including the ability to set permissions and other maintenance things. It is the tool used to define the permissions that should exist throughout the OS (see the stuff in FreeBSD: /etc/mtree). It can also be used to fingerprint all your system files, to help detect trojans, etc. It's a really cool tool but somewhat limited in usefulness. But this stuff, it is good at.