rsync and locked volume = data loss?

[Uhtred]

My rsync task runs on my Backup volume each night to another freenas server over SSH. The Backup volume is encrypted.

A few days ago the source server had a panic, I think because of a large CIFS transfer (9.2.1). This rebooted the server and locked the Backup volume. It took me a couple of days to notice this.

When I did I unlocked the volume in question and upgraded to 9.2.1.1 and the day after than noticed a huge amount of activity.

Upon investigation I found rsysnc replicating my entire Backup volume again to the remote freenas server.

Did rsync replicate when the volume was locked and wipe the remote copy? Should it have done that? I would have thought the mount path would be unavailable and it would simply fail?

 

[cyberjock]

Shouldn't have. I'd have expected it to fail too.

Hardware specs please?

Right now, it sounds like you had a kernel panic because of bad non-ECC RAM, and now that stuff is being corrupted in RAM, it's corrupting every file in your backup as a bonus.

 

[Uhtred]

Thanks for the reply. I certainly hope silent data corruption isn't at play, that is one of my worst nightmares!

I don't think it is though because there is other data on that volume which is fine and other rsync tasks. Add to that crash plan runs through the same data on the source freenas server and it all checks out. If the data was corrupt at source or destination I would expect the data to be overwritten rather than simply vanish like it did.

I've had a chance to look at my syslog server and found an event which indicates rsync did run while the volume was locked...

Feb2103:00:42CALLISTOFeb2103:00:00/usr/sbin/cron[40967]: (root) CMD (PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/root/bin" lockf-s-t0-k '/mnt/Backup/Jupiter' rsync-r-t-z --delete-p-e 'ssh-p22-oBatchMode=yes-oStrictHostKeyChecking=yes' '/mnt/Backup/Jupiter' root@lalala@lala.com:'/mnt/Backup/' 2>&1 |/usr/bin/logger-trsync)

This shows when the volume was locked on the source freenas server

This shows the destination server (rsync task runs at 3am)

It then occured to me that I could recreate this quite easily so I locked my Backup drive and ran rsync but no data was lost and instead I got this error...

2/23/14
3:32:49.000 PM

Feb 23 15:32:49 CALISTO Feb 23 15:32:00 rsync: lockf: cannot open /mnt/Backup/Jupiter: No such file or directory

So, I wasn't able to recreate the problem and I'm still at a bit of a loss (I have upgraded to 9.2.1.1 since then).

Any thoughts?

 

[cyberjock]

And yet you didn't post your hardware specs... sigh.

Good luck to you sir!

 

[Uhtred]

I can't remember the destination specs exactly, Intel(R) Core(TM)2 CPU 6300 @ 1.86GHz with a gigabyte motherboard and 4gb non ecc ram.

Source is an HP N40L again with non ecc RAM.

Both drives are single 1tb using ZFS.

Neither drive has detected problems but i've yet to run a scrub.

 

[cyberjock]

First thing I'd do is a RAM test on both of those machines. non-ECC is so destructive if it fails you can't overstate the consequences.