zvol snapshots appear to be corrupted after snapshot replication

[byroncollege]

Hi this one is a bit odd,
I have a number of zvol's shared via iSCSI to windows server, NTFS formated. They work perfectly and have periodic snapshots taken of them.
Those snapshots can be mounted and shared as iSCSI volumes for backup and restore purposes.

However, when a replication task fires and replicates the zvol snapshot to another disk, the snapshot becomes corrupted both the original and the clone on the disk. (This is a localhost replication to another harddrive). The live zvol is not affected and keeps working as is and I can even take a manual snapshot of it and then mount that snapshot - works perfectly. It is only after a replication (which completes successfully) that both source and destination snapshots are corrupted.

When I say corrupted, they are no longer recognised as a valid disk and cannot be initialised in Windows or Linux. See output from dmseg below when trying to add the corrupted snapshot via iscsiadm

Code:

 [2478535.448635] scsi host2: iSCSI Initiator over TCP/IP
[2478535.713753] scsi 2:0:0:0: Direct-Access	 FreeNAS  iSCSI Disk	   0123 PQ: 0 ANSI: 6
[2478535.715205] sd 2:0:0:0: Attached scsi generic sg2 type 0
[2478535.720688] sd 2:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[2478535.720700] sd 2:0:0:0: [sdb] Sense Key : Not Ready [current]
[2478535.720706] sd 2:0:0:0: [sdb] Add. Sense: Logical unit not ready, manual intervention required
[2478535.723768] sd 2:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[2478535.723778] sd 2:0:0:0: [sdb] Sense Key : Not Ready [current]
[2478535.723784] sd 2:0:0:0: [sdb] Add. Sense: Logical unit not ready, manual intervention required
[2478535.728369] sd 2:0:0:0: [sdb] Write Protect is off
[2478535.728380] sd 2:0:0:0: [sdb] Mode Sense: 73 00 10 08
[2478535.729392] sd 2:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[2478535.736759] sd 2:0:0:0: [sdb] Read Capacity(16) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[2478535.736770] sd 2:0:0:0: [sdb] Sense Key : Not Ready [current]
[2478535.736777] sd 2:0:0:0: [sdb] Add. Sense: Logical unit not ready, manual intervention required
[2478535.738788] sd 2:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[2478535.738798] sd 2:0:0:0: [sdb] Sense Key : Not Ready [current]
[2478535.738803] sd 2:0:0:0: [sdb] Add. Sense: Logical unit not ready, manual intervention required
[2478535.745140] sd 2:0:0:0: [sdb] Attached SCSI disk
[2478701.302171] sd 2:0:0:0: [sdb] Synchronizing SCSI cache
[2478701.311744] sd 2:0:0:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[2478701.311759] sd 2:0:0:0: [sdb] Sense Key : Illegal Request [current]
[2478701.311770] sd 2:0:0:0: [sdb] Add. Sense: Logical unit not supported

As opposed to when the snapshot hasn't had a replication attempt on it

Code:

[2478836.547833] scsi host3: iSCSI Initiator over TCP/IP
[2478836.812547] scsi 3:0:0:0: Direct-Access	 FreeNAS  iSCSI Disk	   0123 PQ: 0 ANSI: 6
[2478836.814481] sd 3:0:0:0: Attached scsi generic sg2 type 0
[2478836.816613] sd 3:0:0:0: [sdb] 52428800 4096-byte logical blocks: (215 GB/200 GiB)
[2478836.816621] sd 3:0:0:0: [sdb] 16384-byte physical blocks
[2478836.825563] sd 3:0:0:0: [sdb] Write Protect is on
[2478836.825574] sd 3:0:0:0: [sdb] Mode Sense: 73 00 90 08
[2478836.828302] sd 3:0:0:0: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
[2478836.937899]  sdb: sdb1 sdb2
[2478836.949469] sd 3:0:0:0: [sdb] Attached SCSI disk

I have this setup (iSCSI, zvol snapshots and replication to a local disk) working on a number of hosts (identical hardware - see below), and it is just the one host where this seems to be happening.

HP N40L with 16GB Ram ECC
FreeNAS 9.10 Stable

 

[depasseg]

Has this always been like this? Or did it work at one time? Has anything changed?

Replication really has nothing to do with the source snapshot data, so it's strange that it gets affected. But I just saw an issue with a recent 9.10 update and iSCSI, so I'm wondering if this is related.

 

[byroncollege]

No it has just started a little while ago. Yes it is very strange I don't know why it affects the source as well. It is like it is corrupting the partition record or something?

 

[depasseg]

What exact version of FreeNAS was this? If it's 9.10.1-U1, you might want to upgrade to U2. Otherwise you might need to file a bug.